v10: Trading OS — 5-subsystem architecture + Signal Intelligence + Dashboard Command Center

Architecture:
- Restructure into 5 subsystems: data/, features/, market/, signals/, execution/, apps/
- Unified ts_code conversion in core/codes.py (idempotent, kills 4 duplicate copies)
- analytics_conn() + kline_glob() — zero hardcoded DB/Parquet paths
- Fixed double-suffix bug (.SZ.SZ) in backfill pipeline root cause

Signal Intelligence (the moat):
- 14 signal types: EMA52, Vegas, Chan, ORB, Gap, NR7, Inside Bar
- 640K+ historical signal instances across 8 backfilled types
- Multi-signal Expectancy Engine with breadth-similarity matching
- Signal backfill CLI: ashare-dp backfill signals

Market Intelligence:
- 8 engines: State, Leadership, Opportunity, Flow, Sentiment, Memory, Knowledge Graph, Recommendations
- Real limit-up/down sentiment via akshare (108 ZT, 19 DT, 52 broken board)
- Knowledge Graph: 8 themes × 30+ concepts with keyword matching
- Money-flow stock recommendations with entry/stop/target trade plans

Dashboard Command Center:
- Decision-first layout: COMMAND → WHERE → WHY → RISK → EXPECTANCY
- Multi-signal Expectancy comparison table (8 types ranked by WR)
- Theme Map visualization with rotation detection
- Intraday Replay infrastructure (30min state snapshots)
- RECOMMENDATIONS card with actionable trade plans

Trading Memory:
- trade_log table + POST/GET/PUT API for trade recording
- Performance stats aggregation

Code Quality:
- 0 hardcoded DB paths, 0 REPLACE hacks, 0 dead ts_code copies
- EMA52 screening deduplicated (CLI + scheduler share one function)
- read_parquet_sql() helper for 28 duplicate patterns
- 6 bugs fixed from code review (NR7 window, theme matching, column indices, etc.)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
jackyu66git
2026-07-06 12:11:42 +08:00
co-authored by Claude
parent ac5d538a04
commit cc95bbb638
83 changed files with 6328 additions and 728 deletions
+1
View File
@@ -33,3 +33,4 @@ logs/
# Virtual env # Virtual env
.venv/ .venv/
venv/ venv/
.gstack/
+72 -74
View File
@@ -1,94 +1,92 @@
# CLAUDE.md - A-Share Data Platform # CLAUDE.md A-Share Data Platform (Trading OS)
## Project Overview ## Project Overview
A-share (Chinese stock market) data platform providing 9 K-line frequencies via Parquet + DuckDB storage with REST and WebSocket APIs. Greenfield project, currently at v0.1.0. A-share trading operating system built on Parquet + DuckDB with REST/WebSocket APIs. Architecture follows a 5-subsystem design: Data Platform → Market Intelligence → Signal Intelligence → Execution Intelligence → Presentation. Dashboard at `/dashboard` renders a Trading Command Center.
## Tech Stack ## Tech Stack
- **Data source**: `akshare` (wraps East Money and Sina APIs) - **Data**: `akshare` (East Money / Sina), Parquet (Zstd), DuckDB (analytics)
- **Storage**: Parquet (Zstd compression, Hive-partitioned `year=YYYY/month=MM/day=DD/`) - **API**: FastAPI + uvicorn, single `/api/v1/dashboard/state` endpoint + K-line/stock/calendar REST
- **Query engine**: DuckDB (embedded OLAP, `read_parquet` with `hive_partitioning=true, union_by_name=true`) - **CLI**: Typer (`ashare-dp backfill daily|minute|industry|signals ...`)
- **Web**: FastAPI + uvicorn - **Scheduler**: APScheduler (EOD 15:05, EMA52 15:10)
- **CLI**: Typer - **Config**: pydantic-settings (`.env`)
- **Scheduler**: APScheduler 3.x (AsyncIOScheduler)
- **Config**: pydantic-settings (reads `.env`)
- **Logging**: loguru
## Project Structure ## Project Structure (v10 — 5 Subsystems)
``` ```
src/ashare_dp/ src/ashare_dp/
├── config.py # Pydantic Settings (all config via env vars) ├── config.py # Settings
├── core/ ├── core/ # Shared kernel
│ ├── models.py # Freq enum with storage_dir property, freq groupings │ ├── models.py # Freq enum, INDEX_CODES, INDEX_SINA_SYMBOLS
│ ├── calendar.py # Trading calendar, market state, Beijing TZ │ ├── codes.py # ★ Single ts_code conversion module (IDEMPOTENT)
── exceptions.py # AShareDPError hierarchy ── calendar.py # Trading calendar, market state, Beijing TZ
├── data/ │ └── exceptions.py
│ ├── akshare_client.py # AKShare wrapper: dual backend, retry, rate limit ├── domain/ # Ontology — shared contracts
│ ├── backfill.py # Historical backfill (daily/weekly/monthly + minute) │ ├── state.py # MarketState (7-dim continuous vector)
│ ├── eod.py # End-of-day batch pull (all stocks, all freqs) │ ├── context.py # TradingContext, Playbook, Expectancy, Opportunity
── realtime.py # Background async spot poller → WebSocket broadcast ── events.py # RiskEvent
├── storage/ │ ├── features.py # FeatureDefinition
│ ├── database.py # DuckDB singleton (get_db) │ ├── leadership.py # LeaderState enum
── schema.py # DDL: stock_info, trading_calendar ── signal.py # SignalType, SignalInstance
│ ├── repository.py # KLineRepository: write_klines, read_klines, get_latest ├── data/ # ═══ DATA PLATFORM ═══
── partitioning.py # Hive partition path builder ── sources/ # akshare_client, index, industry
├── api/ │ ├── pipelines/ # backfill, eod, realtime
── app.py # FastAPI factory + lifespan + embedded docs HTML ── store/ # database (get_db, analytics_conn, kline_glob), repository, partitioning, schema
│ ├── deps.py # FastAPI DI (get_repo) ├── features/ # Feature Store (6 registered features, all use analytics_conn + kline_glob)
│ ├── routers/ # stocks, kline, realtime, calendar routers ├── market/ # ═══ MARKET INTELLIGENCE ═══
── websocket/ ── state.py # infer_market_state
├── manager.py # ConnectionManager: subscribe/broadcast with async lock ├── leadership.py # assess_leaders (lifecycle per industry)
└── handlers.py # WS message dispatch (/ws/realtime) ├── opportunity.py # rank_opportunities
├── scheduler/ │ ├── flow.py # compute_flow (money flow graph)
│ ├── scheduler.py # APScheduler setup (EOD 15:05, health check 08:00) │ ├── sentiment.py # Phase 2 placeholder
│ └── jobs.py # Job implementations │ └── memory.py # StateStore (state_snapshot table)
── cli/ ── signals/ # ═══ SIGNAL INTELLIGENCE ═══ (the moat)
├── main.py # Typer root: ashare-dp {backfill, serve, query, version} ├── detectors.py # EMA52 cross detection + shared screening logic
├── backfill_cmd.py ├── store.py # signal_instance CRUD (to be extracted from detectors)
── serve_cmd.py ── expectancy.py # get_expectancy(state) — single entry, fallback chain internal
└── query_cmd.py ├── execution/ # ═══ EXECUTION INTELLIGENCE ═══
│ ├── playbook.py # build_playbook (State → strategies/bias/holding)
│ ├── risk.py # RiskRule engine + evaluate_risks
│ └── brief.py # build_brief + brief_to_api_dict (single serialization point)
└── apps/ # ═══ PRESENTATION ═══
├── api/ # app.py, routers/, websocket/, dashboard/
├── cli/ # main.py + backfill/serve/query/eod/screening commands
└── scheduler/ # scheduler.py, jobs.py
``` ```
## Critical Design Decisions
### Freq.storage_dir property
macOS APFS is case-insensitive, so `kline_1m` and `kline_1M` collide. The `Freq.M1` (monthly) uses `storage_dir = "1mon"` to disambiguate. Always use `freq.storage_dir` for filesystem paths, never `freq.value`.
### Batched writes to avoid file overwrites
`write_klines` groups all stocks for a day into a single `data.parquet` file. The backfill and EOD pipelines collect all DataFrames first, then write once per day/freq. Never write per-stock-per-day files.
### Dual backend with auto-fallback
`AKShareClient._resolve_backend()` probes East Money reachability once and caches the result. If unreachable (geo-blocked outside China), falls back to Sina. `get_stock_list()` and `get_trading_calendar()` are Sina-only.
### SQL parameterization
`read_klines()` and `get_latest()` use DuckDB parameterized queries (`$1`, `$2`) for user-supplied values (ts_code, dates). Do NOT use f-string interpolation for user input.
### 2h derivation
2-hour K-lines are derived on-the-fly from 1h data via `_read_2h()` using pandas resampling. No Parquet storage for 2h.
## Key Conventions ## Key Conventions
- Stock codes: `ts_code` format is `"000001.SZ"` (6-digit code + exchange suffix). Internal API calls use 6-digit numeric strings. ### ts_code conversion (CRITICAL)
- Exchange mapping: codes starting with `6`/`9` → SH, `4`/`8`/`92` → BJ, rest → SZ **Always use `from ashare_dp.core.codes import to_ts_code`** — the single idempotent implementation. Never write local `_code_to_ts_code()` copies. `to_ts_code()` is safe to call on any format: bare codes, already-formatted ts_codes, Sina symbols, even legacy corrupted `.SZ.SZ` values.
- Backend-specific column normalization: Sina returns English columns (`date`, `open`, etc.), EM returns Chinese. `_normalize_hist_df` and `_normalize_min_df` handle both.
- Proxy: env vars cleared + `urllib.request.getproxies` monkey-patched at module import time in `akshare_client.py` ### Database connections
- All async state in `ConnectionManager` is protected by `asyncio.Lock` - **DuckDB tables** (stock_info, trading_calendar, signal_instance): use `get_db()` context manager from `data.store.database`
- **Analytics queries** (features, engines): use `analytics_conn()` for raw DuckDB connection — the single sanctioned way. Never hardcode `"data/duckdb/ashare.db"`
- **Parquet globs**: use `kline_glob()` or `partition_glob(freq)` — never hardcode paths
### Architecture boundaries
- `data/` knows nothing about trading
- `features/` computes features, never classifies regimes
- `market/` infers state, knows nothing about signals
- `signals/` queries historical expectancy, knows nothing about execution
- `execution/` maps state to strategies, assembles TradingBrief
- `apps/` only renders, never reasons
### MarketState is a continuous vector (not enum)
7 dimensions: trend, fear, liquidity, rotation, participation, volatility, breadth. Each 0.01.0. Display labels derived downstream only.
### API Response
Single endpoint produces all dashboard data: `GET /api/v1/dashboard/state`. Response versioned (`"version": "1.0"`). Serialization in `execution/brief.py::brief_to_api_dict()` — the single serialization point. Router only orchestrates engine calls.
## Running ## Running
```bash ```bash
pip install -e ".[dev]" pip install -e ".[dev]"
ashare-dp backfill init # First time: create tables, load stock list ashare-dp backfill init # Schema + stock list + trading calendar
ashare-dp backfill daily # Backfill all daily/weekly/monthly history ashare-dp backfill daily # Daily/weekly/monthly + indices
ashare-dp serve start # Start API + scheduler + realtime poller ashare-dp backfill industry # Industry classifications
ashare-dp query stats # Check data status ashare-dp backfill signals # EMA52 signal detection + store
``` ashare-dp serve start # API + scheduler + realtime
open http://localhost:8000/dashboard
## Tests
```bash
pytest
ruff check src/
``` ```
+1 -1
View File
@@ -28,7 +28,7 @@ dev = [
] ]
[project.scripts] [project.scripts]
ashare-dp = "ashare_dp.cli.main:app" ashare-dp = "ashare_dp.apps.cli.main:app"
[build-system] [build-system]
requires = ["setuptools>=75.0"] requires = ["setuptools>=75.0"]
+1
View File
@@ -0,0 +1 @@
"""PRESENTATION — API, CLI, scheduler. Renders, never reasons."""
@@ -10,12 +10,14 @@ from fastapi import FastAPI, HTTPException, Query
from fastapi.responses import HTMLResponse from fastapi.responses import HTMLResponse
from loguru import logger from loguru import logger
from ashare_dp.api.routers import stocks, kline, realtime, calendar from ashare_dp.apps.api.routers import stocks, kline, realtime, calendar
from ashare_dp.apps.api.dashboard.router import router as dashboard_router
from ashare_dp.apps.api.dashboard.html import DASHBOARD_HTML
from ashare_dp.core.models import Freq from ashare_dp.core.models import Freq
from ashare_dp.storage.repository import KLineRepository from ashare_dp.data.store.repository import KLineRepository
from ashare_dp.api.websocket.handlers import router as ws_router from ashare_dp.apps.api.websocket.handlers import router as ws_router
from ashare_dp.storage.database import get_db from ashare_dp.data.store.database import get_db
from ashare_dp.storage.schema import DDL_STATEMENTS from ashare_dp.data.store.schema import DDL_STATEMENTS
DOCS_HTML = r""" DOCS_HTML = r"""
<!DOCTYPE html> <!DOCTYPE html>
@@ -425,7 +427,7 @@ async def lifespan(app: FastAPI):
# Start realtime poller # Start realtime poller
try: try:
from ashare_dp.data.realtime import poller from ashare_dp.data.pipelines.realtime import poller
await poller.start() await poller.start()
app.state.poller = poller app.state.poller = poller
logger.info("Realtime poller started") logger.info("Realtime poller started")
@@ -565,6 +567,11 @@ def create_app() -> FastAPI:
"""EMA52 screening results page.""" """EMA52 screening results page."""
return EMA52_HTML return EMA52_HTML
@app.get("/dashboard", response_class=HTMLResponse)
async def dashboard_page():
"""Trading OS Dashboard — professional trader decision support."""
return DASHBOARD_HTML
@app.get("/api/v1/screening/ema52") @app.get("/api/v1/screening/ema52")
async def api_ema52_screening( async def api_ema52_screening(
freq: str = Query("1d", description="Frequency: 1d or 1w"), freq: str = Query("1d", description="Frequency: 1d or 1w"),
@@ -609,6 +616,7 @@ def create_app() -> FastAPI:
app.include_router(kline.router, prefix="/api/v1") app.include_router(kline.router, prefix="/api/v1")
app.include_router(realtime.router, prefix="/api/v1") app.include_router(realtime.router, prefix="/api/v1")
app.include_router(calendar.router, prefix="/api/v1") app.include_router(calendar.router, prefix="/api/v1")
app.include_router(dashboard_router, prefix="/api/v1")
app.include_router(ws_router) app.include_router(ws_router)
return app return app
@@ -0,0 +1 @@
"""Dashboard package."""
+359
View File
@@ -0,0 +1,359 @@
"""Dashboard HTML — Trading Command Center.
5 sections: COMMAND → ACTION → WHERE → WHY → RISK → EXPECTANCY
Everything else collapsed as Diagnostics.
"""
DASHBOARD_HTML = """<!DOCTYPE html>
<html lang="zh-CN">
<head>
<meta charset="UTF-8"><meta name="viewport" content="width=device-width,initial-scale=1.0">
<title>Trading OS</title>
<style>
:root{--bg:#0d1117;--card:#161b22;--border:#30363d;--text:#e6edf3;--muted:#8b949e;--accent:#58a6ff;--green:#3fb950;--red:#f85149;--orange:#d2991d;--purple:#a371f7}
*{box-sizing:border-box;margin:0;padding:0}
body{font-family:-apple-system,BlinkMacSystemFont,'Segoe UI','PingFang SC','Microsoft YaHei',sans-serif;background:var(--bg);color:var(--text);padding:18px;line-height:1.5;max-width:1040px;margin:0 auto}
.header{display:flex;justify-content:space-between;align-items:center;margin-bottom:8px}
.header h1{font-size:13px;font-weight:600;letter-spacing:-.3px}
.date{font-size:10px;color:var(--muted)}
.loading{text-align:center;padding:50px;color:var(--muted)}
.spinner{display:inline-block;width:22px;height:22px;border:2px solid rgba(255,255,255,.1);border-top-color:var(--accent);border-radius:50%;animation:spin .8s linear infinite}
@keyframes spin{to{transform:rotate(360deg)}}
.error{text-align:center;padding:30px;color:var(--red);font-size:13px}
.card{background:var(--card);border:1px solid var(--border);border-radius:8px;padding:14px;margin-bottom:8px}
.sec-label{font-size:9px;font-weight:700;color:var(--muted);text-transform:uppercase;letter-spacing:1px;margin-bottom:6px}
.good{color:var(--green)}.bad{color:var(--red)}.warn{color:var(--orange)}.muted{color:var(--muted)}
.tag{display:inline-block;padding:2px 8px;border-radius:4px;font-size:10px;font-weight:500;margin:1px 3px 1px 0}
.tag.buy{background:rgba(63,185,80,.15);color:var(--green)}.tag.avoid{background:rgba(248,81,73,.15);color:var(--red)}
.tag.bias{font-size:12px;font-weight:700;padding:3px 10px;border-radius:5px}
.tag.LONG{background:rgba(63,185,80,.18);color:var(--green)}
.tag.CASH{background:rgba(139,148,158,.18);color:var(--muted)}
/* Rows */
.row{display:grid;grid-template-columns:1fr 1fr;gap:8px;margin-bottom:8px}
@media(max-width:800px){.row{grid-template-columns:1fr}}
/* COMMAND */
.cmd-mode{font-size:26px;font-weight:800;margin-bottom:2px}
.cmd-play{font-size:12px;color:var(--muted);margin-bottom:6px}
.cmd-meta{display:flex;gap:12px;flex-wrap:wrap;font-size:10px;color:var(--muted);margin-top:6px;padding-top:6px;border-top:1px solid rgba(255,255,255,.06)}
/* WHERE */
.where-item{display:flex;align-items:center;gap:6px;padding:6px 8px;background:rgba(255,255,255,.02);border-radius:5px;margin-bottom:4px}
.where-stars{font-size:11px;color:var(--orange);width:55px;flex-shrink:0;letter-spacing:-1px}
.where-name{font-weight:600;font-size:12px;min-width:70px}
.where-detail{font-size:9px;color:var(--muted);flex:1}
.where-leader{font-size:9px;color:var(--accent)}
/* WHY — flow */
.flow-line{font-family:monospace;font-size:10px;line-height:1.6;white-space:pre;color:var(--muted)}
.flow-line .fr{color:var(--red)}.flow-line .to{color:var(--green)}.flow-line .ar{color:var(--muted)}
/* RISK */
.risk-item{padding:6px 8px;border-radius:4px;margin-bottom:3px;font-size:10px;display:flex;align-items:flex-start;gap:5px}
.risk-crit{background:rgba(248,81,73,.1);border:1px solid rgba(248,81,73,.2)}
.risk-warn{background:rgba(210,153,29,.1);border:1px solid rgba(210,153,29,.2)}
.risk-badge{font-size:7px;font-weight:700;padding:1px 4px;border-radius:2px;flex-shrink:0}
.risk-crit .risk-badge{background:rgba(248,81,73,.2);color:var(--red)}
.risk-warn .risk-badge{background:rgba(210,153,29,.2);color:var(--orange)}
/* EXPECTANCY */
.exp-grid{display:grid;grid-template-columns:repeat(4,1fr);gap:4px;text-align:center}
.exp-stat{padding:4px 2px;background:rgba(255,255,255,.02);border-radius:3px}
.exp-stat .val{font-size:15px;font-weight:700}.exp-stat .lbl{font-size:8px;color:var(--muted)}
.exp-table{width:100%;border-collapse:collapse;font-size:10px;margin-top:6px}
.exp-table th{color:var(--muted);text-align:left;padding:3px 4px;border-bottom:1px solid rgba(255,255,255,.06);font-weight:500}
.exp-table td{padding:2px 4px;border-bottom:1px solid rgba(255,255,255,.03)}
.exp-mini-bar{display:inline-block;height:8px;border-radius:2px;min-width:2px;vertical-align:middle;margin-right:4px}
/* Diagnostics */
.diag-toggle{cursor:pointer;user-select:none;font-size:9px;color:var(--muted);padding:3px 0;margin-bottom:8px}
.diag-toggle:hover{color:var(--text)}
.diag-content{display:none}
.diag-content.open{display:block}
.diag-grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(170px,1fr));gap:6px}
.diag-card{background:rgba(255,255,255,.015);border:1px solid rgba(255,255,255,.04);border-radius:5px;padding:6px}
.diag-card h4{font-size:9px;color:var(--muted);margin-bottom:4px}
.metric{display:flex;justify-content:space-between;padding:1px 0;font-size:9px;border-bottom:1px solid rgba(255,255,255,.02)}.metric:last-child{border:none}
</style>
</head>
<body>
<div class="header"><div><h1>Trading OS</h1></div><div class="date" id="trade-date">--</div></div>
<div id="app"><div class="loading"><div class="spinner"></div><p>Loading...</p></div></div>
<script>
var A='/api/v1/dashboard/state';
function S(n){var s='';for(var i=0;i<5;i++)s+=i<n?'':'';return s;}
function P(v,d){return v!=null&&!isNaN(v)?v:(d||0);}
function Q(v){return v==null?'--':v;}
function L(){
fetch(A).then(function(r){if(!r.ok)throw Error('HTTP '+r.status);return r.json();})
.then(function(d){R(d);}).catch(function(e){document.getElementById('app').innerHTML='<div class=\"error\">'+e.message+'</div>';setTimeout(L,10000);});
}
function R(d){
document.getElementById('trade-date').textContent=d.trade_date||'--';
var pb=d.playbook||{},exp=d.expectancy||{},alloc=d.allocation||{};
var opps=d.opportunities||[],risks=d.risks||[],leaders=d.leaders||{};
var mf=d.money_flow||{},flows=mf.flows||[],netIn=mf.net_inflow||{},netOut=mf.net_outflow||{};
var ev=d.evidence||{},state=ev.state||{},dims=state.dimensions||{},feat=ev.features||{};
var b=feat.breadth_vector||{},v=feat.volume_vector||{},tr=feat.trend_vector||{};
var wr=P(exp.win_rate*100,0),ar=P(exp.avg_return*100,0);
var dd=P(exp.max_drawdown*100,0),hold=P(exp.avg_holding_days,0);
var samples=P(exp.sample_count,0),sigType=Q(exp.signal_type);
var trend=P(dims.trend,0),fear=P(dims.fear,0),liq=P(dims.liquidity,0);
var conf=P(state.confidence||pb.confidence,0.5);
var cashP=P(alloc.cash_pct*100,30).toFixed(0),trendP=P(alloc.trend_pct*100,40).toFixed(0);
// Leaders
var alive=[],broken=[];
for(var k in leaders){var st=leaders[k].state||'';if(st==='leading'||st==='growing'||st==='birth')alive.push(k);if(st==='dead'||st==='breaking')broken.push(k);}
// Mode
var ms=conf>0.6&&wr>55?5:conf>0.45&&wr>40?4:conf>0.3?3:wr>35?2:1;
var ml=ms>=4?'积极做':ms>=3?'轻仓试错':ms>=2?'谨慎':'保命';
var mc=ms>=4?'good':ms>=3?'warn':'bad';
// ACTIONS — derive from Playbook
var acts=[],avs=[];
(pb.suitable_strategies||[]).forEach(function(s){acts.push('<span class=\"tag buy\">✓ '+s+'</span>');});
(pb.unsuitable_strategies||[]).forEach(function(s){avs.push('<span class=\"tag avoid\">✗ '+s+'</span>');});
// WHERE
var wh='';
opps.slice(0,6).forEach(function(o,i){
var ld=o.leader_stock||'';
wh+='<div class=\"where-item\">'+
'<span class=\"where-stars\">'+S(Math.round(o.score/20))+'</span>'+
'<span class=\"where-name\">'+o.name+'</span>'+
'<span class=\"where-detail\">'+(ld?'<span class=\"where-leader\">'+ld+'</span> · ':'')+o.lifecycle+' · '+o.persistence_days+'d</span>'+
'</div>';
});
// WHY — flow
var af=[];
for(var k in netIn)af.push({name:k,chg:netIn[k],dir:'in'});
for(var k in netOut)af.push({name:k,chg:netOut[k],dir:'out'});
af.sort(function(a,b){return Math.abs(b.chg)-Math.abs(a.chg);});
var flowL='',seen={};
af.slice(0,8).forEach(function(f){if(seen[f.name])return;seen[f.name]=1;
var arrow='',n=Math.round(Math.abs(f.chg)*12);
for(var i=0;i<Math.min(n,10);i++)arrow+=f.dir==='in'?'':'';
flowL+='<span class=\"'+(f.dir==='in'?'to':'fr')+'\">'+f.name+'</span> <span class=\"ar\">'+arrow+'</span> <span style=\"font-size:9px;color:var(--muted)\">'+(f.chg>0?'+':'')+(f.chg*100).toFixed(1)+'%</span>\\n';
});
// RISK
var rh='<span style=\"font-size:10px;color:var(--green)\">✓ None</span>';
if(risks.length){rh='';risks.forEach(function(r){var l=r.level==='danger'?'risk-crit':'risk-warn';rh+='<div class=\"risk-item '+l+'\"><span class=\"risk-badge\">'+r.level.toUpperCase().substring(0,4)+'</span>'+r.message+'</div>';})}
// Focus/Avoid
var ft='',at='';
(pb.focus_sectors||[]).forEach(function(s){ft+='<span class=\"tag buy\">'+s+'</span> ';});
(pb.avoid_sectors||[]).forEach(function(s){at+='<span class=\"tag avoid\">'+s+'</span> ';});
// WHY text
var wy=[];
if(trend>0.5)wy.push('趋势向上');else if(trend>0.3)wy.push('震荡');else wy.push('偏弱');
if(alive.length)wy.push('龙头健康');
if(broken.length)wy.push(broken.length+'龙头破位');
if(fear>0.6)wy.push('恐慌偏高');
if(v.vs_5d>1.1)wy.push('放量');
if(v.vs_5d<0.85)wy.push('缩量');
var sent=ev.sentiment||{};
if(sent.profit_effect&&sent.profit_effect!=='pending_data')wy.push('赚钱效应:'+sent.profit_effect);
// Build
var h='';
// ═══ COMMAND + WHERE ═══
h+='<div class=\"row\">';
h+='<div class=\"card\" style=\"background:linear-gradient(135deg,#1a1f2e,#161b22)\">';
h+='<div class=\"sec-label\">COMMAND</div>';
h+='<div class=\"cmd-mode\"><span class=\"'+mc+'\">'+S(ms)+'</span></div>';
h+='<div style=\"font-size:13px;font-weight:700\">'+ml+'</div>';
h+='<div class=\"cmd-play\">'+Q(pb.suitable_strategies[0]||pb.bias)+' · '+Q(pb.holding_time)+'</div>';
h+='<div style=\"margin-bottom:4px\">'+acts.join(' ')+'</div>';
if(avs.length)h+='<div style=\"margin-bottom:2px\">'+avs.join(' ')+'</div>';
h+='<div class=\"cmd-meta\"><span>'+wy.join(' · ')+'</span></div>';
h+='<div class=\"cmd-meta\" style=\"border-top:none;padding-top:2px\"><span>Agr '+(conf*100).toFixed(0)+' &nbsp;Hlth '+(trend*100).toFixed(0)+' &nbsp;Fear '+(fear*100).toFixed(0)+' &nbsp;Liq '+(liq*100).toFixed(0)+' &nbsp;'+Q(pb.bias)+' &nbsp;C'+cashP+'/T'+trendP+'/T20</span></div>';
h+='</div>';
h+='<div class=\"card\">';
h+='<div class=\"sec-label\">WHERE</div>';
h+=wh||'<span class=\"muted\" style=\"font-size:10px\">--</span>';
h+='</div></div>';
// ═══ WHY + RISK ═══
h+='<div class=\"row\">';
h+='<div class=\"card\">';
h+='<div class=\"sec-label\">WHY — Money Flow</div>';
h+='<div class=\"flow-line\">'+(flowL||'No flow data')+'</div>';
if(ft)h+='<div style=\"margin-top:6px;font-size:9px\"><span class=\"muted\">Focus:</span> '+ft+'</div>';
if(at)h+='<div style=\"font-size:9px\"><span class=\"muted\">Avoid:</span> '+at+'</div>';
h+='</div>';
// Theme Map
var tg=d.theme_graph||{},themes=tg.themes||[];
h+='<div class=\"card\">';
h+='<div class=\"sec-label\">THEMES</div>';
if(themes.length){
themes.slice(0,5).forEach(function(t,i){
var stars='';for(var j=0;j<5;j++)stars+=j<Math.round(t.score/20)?'':'';
h+='<div style=\"display:flex;justify-content:space-between;align-items:center;padding:3px 0;font-size:11px\">'+
'<span style=\"font-weight:600\">'+t.name+'</span>'+
'<span style=\"color:var(--orange);font-size:10px\">'+stars+'</span>'+
'<span style=\"font-size:10px;color:var(--muted)\">'+t.momentum+'</span></div>';
if(t.concepts&&t.concepts.length){
h+='<div style=\"font-size:9px;color:var(--muted);margin-bottom:2px;padding-left:8px\">'+
t.concepts.slice(0,4).map(function(c){return c.name}).join('')+'</div>';
}
});
}else{h+='<span class=\"muted\" style=\"font-size:10px\">No active themes</span>';}
if(tg.edges&&tg.edges.length){
h+='<div style=\"margin-top:4px;font-size:9px;color:var(--muted)\">'+
tg.edges.slice(0,2).map(function(e){return e.source+''+e.target}).join(' · ')+'</div>';
}
h+='</div></div>';
// Recommendations
var recs=d.recommendations||[];
h+='<div class=\"card\" style=\"margin-bottom:8px\">';
h+='<div class=\"sec-label\">RECOMMENDATIONS — Money Flow Picks</div>';
if(recs.length){
h+='<div style=\"display:grid;grid-template-columns:repeat(auto-fill,minmax(300px,1fr));gap:6px\">';
recs.forEach(function(r,i){
var cls=r.score>70?'good':r.score>50?'warn':'muted';
h+='<div style=\"padding:8px 10px;background:rgba(255,255,255,.02);border-radius:6px;font-size:11px\">';
h+='<div style=\"display:flex;justify-content:space-between;align-items:center;margin-bottom:4px\">';
h+='<span><strong>'+(r.name||r.ts_code)+'</strong> <span style=\"font-size:10px;color:var(--muted)\">'+r.ts_code+' · '+r.sector+'</span></span>';
h+='<span class=\"'+cls+'\" style=\"font-weight:700\">'+r.score.toFixed(0)+'</span></div>';
h+='<div style=\"display:flex;gap:12px;font-size:10px;color:var(--muted);margin-bottom:4px\">';
h+='<span>Entry: <span class=\"good\">'+r.entry+'</span></span>';
h+='<span>Stop: <span class=\"bad\">'+r.stop+'</span></span>';
h+='<span>T1: <span class=\"good\">'+r.target1+'</span></span>';
h+='<span>T2: '+r.target2+'</span></div>';
h+='<div style=\"display:flex;justify-content:space-between;font-size:10px\">';
h+='<span class=\"muted\">5d:'+(r.metrics.ret_5d>0?'+':'')+r.metrics.ret_5d.toFixed(1)+'% Vol:'+r.metrics.vol_ratio.toFixed(1)+'x '+r.metrics.trend+'</span>';
h+='<span style=\"color:var(--accent)\">R:R '+r.rr_ratio.toFixed(1)+'</span></div>';
h+='<div style=\"font-size:10px;color:var(--green);margin-top:3px\">'+r.action+'</div>';
h+='</div>';
});
h+='</div>';
}else{h+='<span class=\"muted\" style=\"font-size:11px\">No recommendations — insufficient data</span>';}
h+='</div>';
h+='<div class=\"row\"><div class=\"card\">';
h+='<div class=\"sec-label\">RISK</div>';
h+=rh;
h+='</div></div>';
// ═══ EXPECTANCY ═══
h+='<div class=\"card\" style=\"opacity:0.8\">';
var expLabel=samples>0?''+sigType+' · '+samples+' samples':'— Phase 1 Heuristic · 0 samples';
var expColor=samples>0?'var(--green)':'var(--orange)';
h+='<div class=\"sec-label\">EXPECTANCY <span style=\"color:'+expColor+';font-weight:400\">'+expLabel+'</span></div>';
h+='<div class=\"exp-grid\">';
h+='<div class=\"exp-stat\"><div class=\"val '+(wr>50?'good':'warn')+'\">'+wr.toFixed(0)+'%</div><div class=\"lbl\">Est. WR</div></div>';
h+='<div class=\"exp-stat\"><div class=\"val '+(ar>0?'good':'bad')+'\">'+(ar>0?'+':'')+ar.toFixed(1)+'%</div><div class=\"lbl\">Est. Ret</div></div>';
h+='<div class=\"exp-stat\"><div class=\"val bad\">'+dd.toFixed(1)+'%</div><div class=\"lbl\">Est. DD</div></div>';
h+='<div class=\"exp-stat\"><div class=\"val\">'+hold.toFixed(1)+'d</div><div class=\"lbl\">Est. Hold</div></div>';
h+='</div>';
// All expectancies comparison table
var allExp=d.all_expectancies||[];
if(allExp.length>0){
h+='<table class=\"exp-table\"><tr><th>Signal</th><th>WR</th><th>Samples</th><th>Avg Ret</th><th>Max DD</th></tr>';
allExp.forEach(function(e){
var ew=e.win_rate*100,ea=e.avg_return*100,ed=e.max_drawdown*100;
var barW=Math.max(2,ew);
h+='<tr>'+
'<td>'+e.signal_type+'</td>'+
'<td><span class=\"exp-mini-bar\" style=\"width:'+barW+'px;background:'+(ew>50?'var(--green)':ew>40?'var(--orange)':'var(--red)')+'\"></span>'+(e.sample_count>0?ew.toFixed(0)+'%':'--')+'</td>'+
'<td style=\"color:var(--muted)\">'+(e.sample_count>0?e.sample_count.toLocaleString():'heuristic')+'</td>'+
'<td class=\"'+(ea>0?'good':'bad')+'\">'+(e.sample_count>0?(ea>0?'+':'')+ea.toFixed(1)+'%':'--')+'</td>'+
'<td class=\"bad\">'+(e.sample_count>0?ed.toFixed(1)+'%':'--')+'</td>'+
'</tr>';
});
h+='</table>';
}
h+='</div>';
// ═══ DIAGNOSTICS (always collapsed) ═══
var idx=(tr.index_details||[]).slice(0,7);
var idxH='';idx.forEach(function(i){idxH+='<div class=\"metric\"><span>'+i.name+'</span><span class=\"'+(i.pct_chg>0?'good':'bad')+'\">'+(i.pct_chg>0?'+':'')+i.pct_chg.toFixed(2)+'%</span></div>';});
h+='<div class=\"diag-toggle\" onclick=\"var e=document.getElementById(\\'diag\\');e.classList.toggle(\\'open\\');this.textContent=e.classList.contains(\\'open\\')?\\'▼ Diagnostics\\':\\'▶ Diagnostics\\'\">▶ Diagnostics</div>';
h+='<div class=\"diag-content\" id=\"diag\"><div class=\"diag-grid\">';
h+='<div class=\"diag-card\"><h4>Breadth</h4><div class=\"metric\"><span>Adv/Dec</span><span><span class=\"good\">'+Q(b.advances)+'</span>/<span class=\"bad\">'+Q(b.declines)+'</span></span></div><div class=\"metric\"><span>Ratio</span><span>'+((b.advance_ratio||0)*100).toFixed(1)+'%</span></div><div class=\"metric\"><span>New High</span><span>'+((b.new_high_ratio||0)*100).toFixed(1)+'%</span></div></div>';
h+='<div class=\"diag-card\"><h4>Volume</h4><div class=\"metric\"><span>Turnover</span><span>'+Q(v.total_turnover_yi)+' 亿</span></div><div class=\"metric\"><span>vs 5d</span><span>'+((v.vs_5d-1)*100).toFixed(1)+'%</span></div><div class=\"metric\"><span>Tier</span><span>'+Q(v.tier)+'</span></div></div>';
h+='<div class=\"diag-card\"><h4>State</h4>';['trend','fear','liquidity','rotation','participation','volatility','breadth'].forEach(function(k){h+='<div class=\"metric\"><span>'+k+'</span><span>'+((dims[k]||0)*100).toFixed(0)+'</span></div>';});h+='</div>';
h+='<div class=\"diag-card\"><h4>Indices</h4>'+idxH+'<div style=\"font-size:8px;color:var(--muted);margin-top:2px\">Resonance: '+((tr.resonance||0)*100).toFixed(0)+'%</div></div>';
var sent=ev.sentiment||{};
if(sent.status==='live'){
h+='<div class=\"diag-card\"><h4>Sentiment</h4>';
h+='<div class=\"metric\"><span>Profit Effect</span><span class=\"'+(sent.profit_effect==='Strong'||sent.profit_effect==='Average'?'good':'bad')+'\">'+Q(sent.profit_effect)+' ('+Q(sent.profit_effect_score)+')</span></div>';
h+='<div class=\"metric\"><span>Limit Up/Down</span><span><span class=\"good\">'+Q(sent.limit_up_count)+'</span>/<span class=\"bad\">'+Q(sent.limit_down_count)+'</span></span></div>';
h+='<div class=\"metric\"><span>Broken Board</span><span>'+Q(sent.broken_board_count)+' ('+((sent.broken_board_rate||0)*100).toFixed(0)+'%)</span></div>';
h+='<div class=\"metric\"><span>Consecutive</span><span>'+Q(sent.consecutive_count)+' ('+((sent.consecutive_board_rate||0)*100).toFixed(0)+'%)</span></div>';
if(sent.max_consecutive)h+='<div class=\"metric\"><span>Max Board</span><span>'+sent.max_consecutive+' 连板</span></div>';
h+='</div>';
}
h+='</div></div>';
// Replay section
h+='<div id=\"replay-section\" style=\"margin-top:8px\"><span class=\"muted\" style=\"font-size:9px\">No intraday data (starts after first trading session)</span></div>';
document.getElementById('app').innerHTML=h;
// Load replay data
loadReplay(d.trade_date);
}
L();setInterval(L,60000);
// ═══ REPLAY ═══
function loadReplay(dateStr){
fetch('/api/v1/dashboard/replay?date_str='+dateStr)
.then(function(r){return r.json();})
.then(function(d){
if(d.snapshots&&d.snapshots.length>0)renderReplay(d.snapshots);
})
.catch(function(){});
}
function renderReplay(snaps){
if(snaps.length<2)return;
var el=document.getElementById('replay-section');
if(!el)return;
var h='<div class=\"sec-title\">REPLAY — Intraday Timeline</div>';
h+='<div style=\"display:flex;align-items:center;gap:8px;margin-bottom:4px\">';
h+='<input type=\"range\" min=\"0\" max=\"'+(snaps.length-1)+'\" value=\"'+(snaps.length-1)+'\" ';
h+='oninput=\"updateReplay(this.value)\" style=\"flex:1;accent-color:var(--accent)\" id=\"replay-slider\">';
h+='<span id=\"replay-time\" style=\"font-size:10px;color:var(--muted);width:45px\">'+snaps[snaps.length-1].timestamp.slice(11,16)+'</span></div>';
h+='<div id=\"replay-bars\" style=\"display:flex;flex-direction:column;gap:2px\">';
var last=snaps[snaps.length-1].dimensions;
for(var k in last){
h+='<div style=\"display:flex;align-items:center;gap:6px\"><span style=\"font-size:9px;color:var(--muted);width:55px\">'+k+'</span>';
h+='<div style=\"flex:1;height:8px;background:rgba(255,255,255,.06);border-radius:2px;overflow:hidden\">';
h+='<div id=\"replay-'+k+'\" style=\"height:100%;background:var(--accent);border-radius:2px;width:'+(last[k]*100).toFixed(0)+'%\" class=\"rp-bar\"></div></div>';
h+='<span id=\"replay-val-'+k+'\" style=\"font-size:9px;width:24px;text-align:right\">'+(last[k]*100).toFixed(0)+'</span></div>';
}
h+='</div>';
el.innerHTML=h;
// Store snaps for slider updates
window._replaySnaps=snaps;
}
function updateReplay(idx){
var snaps=window._replaySnaps;
if(!snaps)return;
var s=snaps[parseInt(idx)];
document.getElementById('replay-time').textContent=s.timestamp.slice(11,16);
var dims=s.dimensions;
for(var k in dims){
var bar=document.getElementById('replay-'+k);
var val=document.getElementById('replay-val-'+k);
if(bar)bar.style.width=(dims[k]*100).toFixed(0)+'%';
if(val)val.textContent=(dims[k]*100).toFixed(0);
}
}
</script>
</body>
</html>"""
+235
View File
@@ -0,0 +1,235 @@
"""Dashboard REST API — thin orchestrator.
Calls engines → assembles TradingBrief → serializes via brief_to_api_dict.
No inference, no manual JSON building.
"""
from __future__ import annotations
from datetime import date, datetime
from fastapi import APIRouter, HTTPException
from loguru import logger
from ashare_dp.core.calendar import BEIJING_TZ
from ashare_dp.data.store.database import get_db, analytics_conn, kline_glob
from ashare_dp.domain.context import Allocation
from ashare_dp.execution.brief import build_brief, brief_to_api_dict, generate_narrative
from ashare_dp.execution.playbook import build_playbook
from ashare_dp.execution.risk import evaluate_risks
from ashare_dp.features import registry
from ashare_dp.market.flow import compute_flow
from ashare_dp.market.leadership import assess_leaders
from ashare_dp.market.memory import state_store
from ashare_dp.market.opportunity import rank_opportunities
from ashare_dp.market.knowledge import activate_themes
from ashare_dp.market.recommendations import get_recommendations
from ashare_dp.market.sentiment import assess_sentiment
from ashare_dp.market.state import infer_market_state
from ashare_dp.signals.expectancy import get_expectancy, get_all_expectancies
router = APIRouter(tags=["dashboard"])
def _get_latest_trade_date() -> date:
conn = analytics_conn()
try:
row = conn.execute(
f"SELECT MAX(trade_date) FROM read_parquet('{kline_glob()}', "
f"hive_partitioning=true, union_by_name=true)"
).fetchone()
if row and row[0]:
return date.fromisoformat(str(row[0])[:10])
raise HTTPException(status_code=503, detail="No K-line data available")
finally:
conn.close()
@router.get("/dashboard/state")
async def dashboard_state():
"""Complete market state — single atomic endpoint, versioned schema."""
trade_date = _get_latest_trade_date()
now = datetime.now(BEIJING_TZ)
# ── L1: Market Understanding ──
try:
features = registry.compute_all(None, trade_date)
except Exception as e:
logger.error(f"Feature computation failed: {e}")
raise HTTPException(status_code=500, detail=f"Feature computation: {e}")
state = infer_market_state(features, trade_date, timestamp=now)
# ── L2: Market Interpretation (each engine independent, failures degrade) ──
leaders, opportunities, money_flow = {}, [], None
try:
leaders = assess_leaders(trade_date)
except Exception as e:
logger.warning(f"Leadership engine failed: {e}")
try:
opportunities = rank_opportunities(trade_date)
except Exception as e:
logger.warning(f"Opportunity engine failed: {e}")
try:
money_flow = compute_flow(trade_date)
except Exception as e:
logger.warning(f"Flow engine failed: {e}")
sentiment = assess_sentiment(trade_date)
# Knowledge Graph — theme activation
theme_graph = activate_themes(trade_date, leaders=leaders, opportunities=opportunities)
# Stock Recommendations
try:
recommendations = get_recommendations(trade_date, top_n=10)
except Exception as e:
logger.warning(f"Recommendations failed: {e}")
recommendations = []
# ── L3: Trading Intelligence ──
playbook = build_playbook(state, leaders, opportunities)
expectancy = get_expectancy(state) # fallback chain lives inside
all_expectancies = get_all_expectancies(state)
allocation = Allocation(
cash_pct=round(max(state.fear * 0.6, 0.1), 2),
trend_pct=round(max(state.trend * 0.6, 0.1), 2),
trial_pct=0.2,
reasoning="恐惧偏高,保留现金缓冲" if state.fear > 0.5 else "趋势明确,积极参与",
)
narrative = generate_narrative(state, leaders, opportunities, money_flow)
brief = build_brief(
trade_date=trade_date, state=state, playbook=playbook,
expectancy=expectancy, leaders=leaders, opportunities=opportunities,
money_flow=money_flow, allocation=allocation, narrative=narrative,
timestamp=now,
)
try:
risk_events = evaluate_risks(features, timestamp=now)
except Exception as e:
logger.warning(f"Risk evaluation failed: {e}")
risk_events = []
# Persist state snapshot for Market Memory
try:
with get_db(read_only=False) as db:
state_store.save(db, "market", "1d", now, state)
except Exception as e:
logger.warning(f"StateStore save failed: {e}")
return brief_to_api_dict(brief, risk_events, features, sentiment, all_expectancies, theme_graph, recommendations)
@router.post("/dashboard/trades")
async def record_trade(request: dict):
"""Record a trade for Trading Memory."""
if not request.get("ts_code") or not request.get("entry_price"):
raise HTTPException(status_code=400, detail="ts_code and entry_price are required")
from ashare_dp.data.store.database import get_db
with get_db(read_only=False) as db:
db.execute(
"""INSERT INTO trade_log (ts_code, trade_date, entry_price, signal_type,
position_pct, state_trend, state_fear, state_liquidity, state_breadth,
tags, notes)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
(request.get("ts_code"), request.get("trade_date"),
request.get("entry_price"), request.get("signal_type", ""),
request.get("position_pct", 0), request.get("state_trend", 0),
request.get("state_fear", 0), request.get("state_liquidity", 0),
request.get("state_breadth", 0), request.get("tags", ""),
request.get("notes", "")),
)
return {"status": "recorded"}
@router.get("/dashboard/trades")
async def list_trades(limit: int = 20, closed: bool | None = None):
"""List recent trades with performance stats."""
from ashare_dp.data.store.database import get_db
with get_db(read_only=True) as db:
wheres = []
params = []
if closed is not None:
wheres.append("closed = ?")
params.append(closed)
where = ("WHERE " + " AND ".join(wheres)) if wheres else ""
rows = db.query(
f"SELECT * FROM trade_log {where} ORDER BY created_at DESC LIMIT ?",
tuple(params) + (limit,),
)
# Stats
stats_rows = db.query(
"SELECT COUNT(*) as total, AVG(return_pct) as avg_ret, "
"AVG(CASE WHEN return_pct>0 THEN 1.0 ELSE 0.0 END) as win_rate "
"FROM trade_log WHERE closed=TRUE"
)
stats = {"total": 0, "avg_return": 0, "win_rate": 0}
if stats_rows and stats_rows[0][0] > 0:
stats = {"total": int(stats_rows[0][0]),
"avg_return": round(float(stats_rows[0][1] or 0), 2),
"win_rate": round(float(stats_rows[0][2] or 0), 4)}
trades = []
for r in rows:
trades.append({
"id": r[0], "ts_code": r[1], "trade_date": str(r[2]) if r[2] else "",
"entry_price": r[3], "signal_type": r[5],
"return_pct": r[10], "holding_days": r[11],
"closed": bool(r[20]), "tags": r[18], "notes": r[19],
})
return {"trades": trades, "stats": stats}
@router.put("/dashboard/trades/{trade_id}/close")
async def close_trade(trade_id: int, request: dict):
"""Close a trade with exit price and reason."""
from ashare_dp.data.store.database import get_db
with get_db(read_only=False) as db:
db.execute(
"""UPDATE trade_log SET exit_price=?, exit_reason=?, return_pct=?,
holding_days=?, closed=TRUE
WHERE id=?""",
(request.get("exit_price"), request.get("exit_reason", "manual"),
request.get("return_pct"), request.get("holding_days", 0), trade_id),
)
return {"status": "closed"}
@router.get("/dashboard/replay")
async def dashboard_replay(date_str: str = ""):
"""Return all intraday state snapshots for a given date (YYYY-MM-DD)."""
from datetime import date as _date
if not date_str:
date_str = _get_latest_trade_date().isoformat()
try:
with get_db(read_only=True) as db:
rows = db.query(
"SELECT timestamp, state FROM state_snapshot "
"WHERE entity = 'market' AND timeframe = '30min' "
"AND CAST(timestamp AS DATE) = ? "
"ORDER BY timestamp ASC",
(date_str,),
)
except Exception:
rows = []
snapshots = []
for ts, state_json in rows:
try:
import json
state = json.loads(state_json)
snapshots.append({
"timestamp": ts.isoformat() if hasattr(ts, 'isoformat') else str(ts),
"dimensions": state.get("dimensions", {}),
})
except Exception:
pass
return {
"date": date_str,
"count": len(snapshots),
"snapshots": snapshots,
}
@@ -1,6 +1,6 @@
"""FastAPI dependency injection.""" """FastAPI dependency injection."""
from ashare_dp.storage.repository import KLineRepository from ashare_dp.data.store.repository import KLineRepository
def get_repo() -> KLineRepository: def get_repo() -> KLineRepository:
@@ -6,9 +6,9 @@ from typing import Optional
from fastapi import APIRouter, Depends, HTTPException, Query from fastapi import APIRouter, Depends, HTTPException, Query
from loguru import logger from loguru import logger
from ashare_dp.api.deps import get_repo from ashare_dp.apps.api.deps import get_repo
from ashare_dp.core.models import BACKFILLABLE_FREQS, DERIVED_FREQS, INTRADAY_FREQS, Freq from ashare_dp.core.models import BACKFILLABLE_FREQS, DERIVED_FREQS, INTRADAY_FREQS, Freq
from ashare_dp.storage.repository import KLineRepository from ashare_dp.data.store.repository import KLineRepository
router = APIRouter(prefix="/klines", tags=["klines"]) router = APIRouter(prefix="/klines", tags=["klines"])
@@ -6,8 +6,9 @@ from typing import Optional
from fastapi import APIRouter, HTTPException, Query from fastapi import APIRouter, HTTPException, Query
from loguru import logger from loguru import logger
from ashare_dp.core.codes import to_ts_code
from ashare_dp.core.calendar import BEIJING_TZ, determine_market_state from ashare_dp.core.calendar import BEIJING_TZ, determine_market_state
from ashare_dp.data.akshare_client import AKShareClient from ashare_dp.data.sources.akshare_client import AKShareClient
router = APIRouter(prefix="/realtime", tags=["realtime"]) router = APIRouter(prefix="/realtime", tags=["realtime"])
@@ -45,14 +46,7 @@ async def get_spot(
# Build ts_code from code # Build ts_code from code
if "code" in df.columns: if "code" in df.columns:
def _to_ts_code(c): df["ts_code"] = df["code"].apply(to_ts_code)
c = str(c).zfill(6)
if c.startswith(("6", "9")):
return f"{c}.SH"
elif c.startswith(("8", "4")):
return f"{c}.BJ"
return f"{c}.SZ"
df["ts_code"] = df["code"].apply(_to_ts_code)
# Filter by requested codes # Filter by requested codes
if codes: if codes:
@@ -6,7 +6,7 @@ from typing import Optional
from fastapi import APIRouter, HTTPException, Query from fastapi import APIRouter, HTTPException, Query
from loguru import logger from loguru import logger
from ashare_dp.storage.database import get_db from ashare_dp.data.store.database import get_db
router = APIRouter(prefix="/stocks", tags=["stocks"]) router = APIRouter(prefix="/stocks", tags=["stocks"])
@@ -7,7 +7,7 @@ from datetime import datetime
from fastapi import APIRouter, WebSocket, WebSocketDisconnect from fastapi import APIRouter, WebSocket, WebSocketDisconnect
from loguru import logger from loguru import logger
from ashare_dp.api.websocket.manager import manager from ashare_dp.apps.api.websocket.manager import manager
from ashare_dp.core.calendar import BEIJING_TZ, determine_market_state from ashare_dp.core.calendar import BEIJING_TZ, determine_market_state
router = APIRouter() router = APIRouter()
+167
View File
@@ -0,0 +1,167 @@
"""Backfill CLI subcommands."""
from __future__ import annotations
import asyncio
from datetime import date, datetime
import typer
from loguru import logger
from ashare_dp.core.models import BACKFILLABLE_FREQS, INTRADAY_FREQS
from ashare_dp.data.sources.akshare_client import AKShareClient
from ashare_dp.data.pipelines.backfill import BackfillPipeline
from ashare_dp.data.sources.index import backfill_indices
from ashare_dp.data.sources.industry import fetch_industry_mapping, store_industry_mapping
backfill_app = typer.Typer()
@backfill_app.command("init")
def init_db():
"""Initialize database schema."""
pipeline = BackfillPipeline()
pipeline.init_db()
pipeline.load_stock_list()
pipeline.load_trading_calendar()
typer.echo("Database initialized with stock list and trading calendar")
@backfill_app.command("daily")
def backfill_daily(
start: str = typer.Option("19900101", help="Start date YYYYMMDD"),
end: str = typer.Option(None, help="End date YYYYMMDD (default: today)"),
workers: int = typer.Option(10, help="Number of worker threads"),
symbols: str = typer.Option(None, help="Comma-separated stock symbols (default: all)"),
skip_index: bool = typer.Option(False, help="Skip index K-line backfill"),
):
"""Backfill daily/weekly/monthly K-line data."""
start_date = datetime.strptime(start, "%Y%m%d").date()
end_date = datetime.strptime(end, "%Y%m%d").date() if end else date.today()
sym_list = [s.strip() for s in symbols.split(",")] if symbols else None
pipeline = BackfillPipeline(max_workers=workers)
pipeline.init_db()
pipeline.load_stock_list()
pipeline.load_trading_calendar()
results = pipeline.backfill_daily_weekly_monthly(
symbols=sym_list,
start_date=start_date,
end_date=end_date,
)
typer.echo(f"\nBackfill complete:")
for freq, stats in results.items():
typer.echo(
f" {freq}: {stats['records']} records, "
f"{stats['completed']} stocks ok, {stats['failed']} failed"
)
# Index backfill
if not skip_index and sym_list is None:
typer.echo("\nBackfilling index data...")
idx_results = backfill_indices(
start_date=start_date.strftime("%Y%m%d"),
end_date=end_date.strftime("%Y%m%d"),
)
for ts_code, count in idx_results.items():
typer.echo(f" {ts_code}: {count} bars")
elif skip_index:
typer.echo("\nSkipping index backfill (--skip-index)")
@backfill_app.command("minute")
def backfill_minute(
days: int = typer.Option(30, help="Number of calendar days to look back"),
workers: int = typer.Option(5, help="Number of worker threads"),
symbols: str = typer.Option(None, help="Comma-separated stock symbols (default: all)"),
):
"""Backfill recent minute K-line data (limited API history)."""
sym_list = [s.strip() for s in symbols.split(",")] if symbols else None
pipeline = BackfillPipeline(max_workers=workers)
pipeline.init_db()
pipeline.load_stock_list()
results = pipeline.backfill_minute(
symbols=sym_list,
days_back=days,
)
typer.echo(f"\nMinute backfill complete ({days} day lookback):")
for freq, stats in results.items():
typer.echo(
f" {freq}: {stats['records']} records, "
f"{stats['completed']} stocks ok, {stats['failed']} failed"
)
@backfill_app.command("industry")
def backfill_industry():
"""Fetch and store industry classifications for all stocks."""
from ashare_dp.data.store.database import get_db
from ashare_dp.data.store.schema import DDL_STATEMENTS
typer.echo("Ensuring schema...")
with get_db(read_only=False) as db:
for ddl in DDL_STATEMENTS:
try:
db.execute(ddl)
except Exception:
pass # Table may already exist
typer.echo("Fetching industry classifications...")
df = fetch_industry_mapping()
count = store_industry_mapping(df)
typer.echo(f"Industry data stored: {count} stocks classified")
@backfill_app.command("signals")
def backfill_signals(
days: int = typer.Option(500, help="Calendar days to scan for historical signals"),
):
"""Detect and store historical trading signals (all types)."""
from ashare_dp.data.store.database import get_db
from ashare_dp.data.store.schema import DDL_STATEMENTS
from ashare_dp.signals.detectors import (
detect_ema52_signals, detect_vegas_signals,
detect_chan_signals, detect_orb_signals,
detect_gap_signals, detect_nr7_signals, detect_ib_signals,
)
from ashare_dp.signals.store import store_signals
typer.echo("Ensuring schema...")
with get_db(read_only=False) as db:
for ddl in DDL_STATEMENTS:
try:
db.execute(ddl)
except Exception:
pass
detectors = [
("EMA52", detect_ema52_signals),
("Vegas", detect_vegas_signals),
("Chan", detect_chan_signals),
("ORB", detect_orb_signals),
("Gap", detect_gap_signals),
("NR7", detect_nr7_signals),
("IB", detect_ib_signals),
]
total = 0
for name, detector in detectors:
typer.echo(f"Scanning {days} days for {name} signals...")
df = detector(lookback_days=days)
if df.empty:
typer.echo(f" {name}: 0 signals")
continue
count = store_signals(df)
total += count
by_type = df["signal_type"].value_counts().to_dict()
typer.echo(f" {name}: {count} stored {dict(by_type)}")
typer.echo(f"Total signals stored: {total}")
@@ -8,7 +8,7 @@ import typer
from loguru import logger from loguru import logger
from ashare_dp.core.calendar import BEIJING_TZ from ashare_dp.core.calendar import BEIJING_TZ
from ashare_dp.data.eod import EODPipeline from ashare_dp.data.pipelines.eod import EODPipeline
eod_app = typer.Typer() eod_app = typer.Typer()
@@ -4,11 +4,11 @@ from __future__ import annotations
import typer import typer
from ashare_dp.cli.backfill_cmd import backfill_app from ashare_dp.apps.cli.backfill_cmd import backfill_app
from ashare_dp.cli.serve_cmd import serve_app from ashare_dp.apps.cli.serve_cmd import serve_app
from ashare_dp.cli.query_cmd import query_app from ashare_dp.apps.cli.query_cmd import query_app
from ashare_dp.cli.eod_cmd import eod_app from ashare_dp.apps.cli.eod_cmd import eod_app
from ashare_dp.cli.screening_cmd import screening_app from ashare_dp.apps.cli.screening_cmd import screening_app
app = typer.Typer( app = typer.Typer(
name="ashare-dp", name="ashare-dp",
@@ -7,8 +7,8 @@ from datetime import date, datetime
import typer import typer
from ashare_dp.core.models import Freq from ashare_dp.core.models import Freq
from ashare_dp.storage.database import get_db from ashare_dp.data.store.database import get_db
from ashare_dp.storage.repository import KLineRepository from ashare_dp.data.store.repository import KLineRepository
query_app = typer.Typer() query_app = typer.Typer()
+23
View File
@@ -0,0 +1,23 @@
"""Screening CLI subcommand — EMA52 scanning and other screens."""
from __future__ import annotations
import typer
screening_app = typer.Typer()
@screening_app.command("ema52")
def ema52(
threshold: float = typer.Option(
None, "--threshold", "-t",
help="EMA52 proximity threshold (e.g. 0.03 = ±3%%). Default from .env",
),
):
"""Run EMA52 screening for 1d and 1w."""
from ashare_dp.signals.detectors import run_ema52_screening
results = run_ema52_screening(threshold=threshold)
for freq, count in results.items():
typer.echo(f" {freq}: {count} stocks near EMA52")
typer.echo("EMA52 screening complete")
@@ -30,7 +30,7 @@ def serve(
# The FastAPI lifespan handles DB init, scheduler start, and poller start # The FastAPI lifespan handles DB init, scheduler start, and poller start
uvicorn.run( uvicorn.run(
"ashare_dp.api.app:create_app", "ashare_dp.apps.api.app:create_app",
host=h, host=h,
port=p, port=p,
reload=reload, reload=reload,
+68
View File
@@ -0,0 +1,68 @@
"""Scheduler job implementations."""
from __future__ import annotations
from datetime import date, datetime
from loguru import logger
from ashare_dp.core.calendar import BEIJING_TZ
from ashare_dp.core.models import Freq
from ashare_dp.data.pipelines.eod import EODPipeline
from ashare_dp.data.store.repository import KLineRepository
async def eod_pull_job():
"""End-of-day data pull. Triggered at 15:05 Beijing time on trading days."""
now = datetime.now(BEIJING_TZ)
today = now.date()
if now.weekday() >= 5:
logger.info("EOD: Weekend, skipping")
return
logger.info(f"EOD job starting for {today.isoformat()}...")
pipeline = EODPipeline()
try:
symbols = pipeline.get_active_symbols()
logger.info(f"EOD: {len(symbols)} active stocks")
except Exception as e:
logger.error(f"EOD: Failed to get stock list: {e}")
return
try:
pipeline.pull_daily(symbols, today)
except Exception as e:
logger.error(f"EOD daily pull failed: {e}")
try:
pipeline.pull_minute(symbols, today)
except Exception as e:
logger.error(f"EOD minute pull failed: {e}")
try:
pipeline.pull_weekly_monthly(symbols, today)
except Exception as e:
logger.error(f"EOD weekly/monthly pull failed: {e}")
logger.info(f"EOD job completed for {today.isoformat()}")
async def health_check_job():
"""Daily health check: report database statistics."""
logger.info("Health check running...")
repo = KLineRepository()
try:
logger.info(
f"Health: 1d={repo.count_records(Freq.d1)}, "
f"1h={repo.count_records(Freq.h1)}"
)
except Exception as e:
logger.error(f"Health check failed: {e}")
async def ema52_screening_job():
"""EOD EMA52 screening — delegates to shared implementation."""
from ashare_dp.signals.detectors import run_ema52_screening
run_ema52_screening()
@@ -7,7 +7,7 @@ from apscheduler.triggers.cron import CronTrigger
from loguru import logger from loguru import logger
from ashare_dp.core.calendar import BEIJING_TZ from ashare_dp.core.calendar import BEIJING_TZ
from ashare_dp.data.akshare_client import AKShareClient from ashare_dp.data.sources.akshare_client import AKShareClient
class Scheduler: class Scheduler:
@@ -19,7 +19,7 @@ class Scheduler:
def start(self): def start(self):
"""Start the scheduler and register jobs.""" """Start the scheduler and register jobs."""
from ashare_dp.scheduler.jobs import eod_pull_job, ema52_screening_job, health_check_job from ashare_dp.apps.scheduler.jobs import eod_pull_job, ema52_screening_job, health_check_job
# EOD job: 15:05 Beijing time, Mon-Fri # EOD job: 15:05 Beijing time, Mon-Fri
self._scheduler.add_job( self._scheduler.add_job(
-85
View File
@@ -1,85 +0,0 @@
"""Backfill CLI subcommands."""
from __future__ import annotations
import asyncio
from datetime import date, datetime
import typer
from loguru import logger
from ashare_dp.core.models import BACKFILLABLE_FREQS, INTRADAY_FREQS
from ashare_dp.data.akshare_client import AKShareClient
from ashare_dp.data.backfill import BackfillPipeline
backfill_app = typer.Typer()
@backfill_app.command("init")
def init_db():
"""Initialize database schema."""
pipeline = BackfillPipeline()
pipeline.init_db()
pipeline.load_stock_list()
pipeline.load_trading_calendar()
typer.echo("Database initialized with stock list and trading calendar")
@backfill_app.command("daily")
def backfill_daily(
start: str = typer.Option("19900101", help="Start date YYYYMMDD"),
end: str = typer.Option(None, help="End date YYYYMMDD (default: today)"),
workers: int = typer.Option(10, help="Number of worker threads"),
symbols: str = typer.Option(None, help="Comma-separated stock symbols (default: all)"),
):
"""Backfill daily/weekly/monthly K-line data."""
start_date = datetime.strptime(start, "%Y%m%d").date()
end_date = datetime.strptime(end, "%Y%m%d").date() if end else date.today()
sym_list = [s.strip() for s in symbols.split(",")] if symbols else None
pipeline = BackfillPipeline(max_workers=workers)
pipeline.init_db()
pipeline.load_stock_list()
pipeline.load_trading_calendar()
results = pipeline.backfill_daily_weekly_monthly(
symbols=sym_list,
start_date=start_date,
end_date=end_date,
)
typer.echo(f"\nBackfill complete:")
for freq, stats in results.items():
typer.echo(
f" {freq}: {stats['records']} records, "
f"{stats['completed']} stocks ok, {stats['failed']} failed"
)
@backfill_app.command("minute")
def backfill_minute(
days: int = typer.Option(30, help="Number of calendar days to look back"),
workers: int = typer.Option(5, help="Number of worker threads"),
symbols: str = typer.Option(None, help="Comma-separated stock symbols (default: all)"),
):
"""Backfill recent minute K-line data (limited API history)."""
sym_list = [s.strip() for s in symbols.split(",")] if symbols else None
pipeline = BackfillPipeline(max_workers=workers)
pipeline.init_db()
pipeline.load_stock_list()
results = pipeline.backfill_minute(
symbols=sym_list,
days_back=days,
)
typer.echo(f"\nMinute backfill complete ({days} day lookback):")
for freq, stats in results.items():
typer.echo(
f" {freq}: {stats['records']} records, "
f"{stats['completed']} stocks ok, {stats['failed']} failed"
)
-168
View File
@@ -1,168 +0,0 @@
"""Screening CLI subcommand: EMA52 scanning and other screens."""
from __future__ import annotations
import typer
from loguru import logger
from ashare_dp.storage.database import get_db
screening_app = typer.Typer()
@screening_app.command("ema52")
def ema52(
threshold: float = typer.Option(
None, "--threshold", "-t",
help="EMA52 proximity threshold (e.g. 0.03 = ±3%%). Default from .env",
),
):
"""Run EMA52 screening for 1d and 1w.
Scans all stocks, computes EMA52 from close prices,
and saves results near EMA52 to the ema52_screening table.
"""
import numpy as np
import pandas as pd
from ashare_dp.config import Settings
from ashare_dp.core.models import Freq
settings = Settings()
if threshold is None:
threshold = settings.ema52_threshold
with get_db(read_only=False) as db:
# Get all ts_codes and names
stocks = db.query(
"SELECT ts_code, name FROM stock_info ORDER BY ts_code"
)
symbols = [row[0] for row in stocks]
symbol_names = {row[0]: row[1] for row in stocks}
logger.info(
f"EMA52 screening: {len(symbols)} stocks, "
f"threshold=±{threshold * 100:.1f}%"
)
for freq, freq_label, bars_needed in [
(Freq.d1, "1d", 100),
(Freq.w1, "1w", 60),
]:
try:
from ashare_dp.storage.partitioning import partition_glob
glob = partition_glob(freq)
sql = f"""
WITH ranked AS (
SELECT *, ROW_NUMBER() OVER (
PARTITION BY ts_code ORDER BY trade_time DESC
) as rn
FROM read_parquet('{glob}', hive_partitioning=true, union_by_name=true)
)
SELECT * FROM ranked WHERE rn <= {bars_needed}
ORDER BY ts_code, trade_time ASC
"""
df = _query_parquet(sql)
if df.empty:
logger.warning(f"EMA52: no data for {freq_label}")
continue
except Exception as e:
logger.error(f"EMA52: failed to read {freq_label}: {e}")
continue
df["trade_time"] = pd.to_datetime(df["trade_time"])
df = df.sort_values(["ts_code", "trade_time"])
results = []
for ts_code, group in df.groupby("ts_code"):
if ts_code not in symbol_names:
continue
group = group.tail(bars_needed)
if len(group) < 26:
continue
closes = group["close"].astype(float).values
ema_values = _compute_ema(closes, 52)
if len(ema_values) == 0:
continue
latest_close = closes[-1]
latest_ema = ema_values[-1]
if latest_ema <= 0:
continue
distance = (latest_close - latest_ema) / latest_ema
# Compute daily amount in 亿 CNY
latest_amount = float(group["amount"].iloc[-1])
amount_yi = latest_amount / 100_000_000 # 元 → 亿
if abs(distance) <= threshold:
results.append({
"trade_date": group["trade_date"].iloc[-1],
"ts_code": ts_code,
"name": symbol_names[ts_code],
"freq": freq_label,
"close_price": round(latest_close, 2),
"ema52": round(latest_ema, 2),
"distance_pct": round(distance * 100, 2),
"amount": round(amount_yi, 2),
})
if results:
# Sort by amount DESC (largest turnover first)
results.sort(key=lambda r: r.get("amount", 0) or 0, reverse=True)
try:
with get_db(read_only=False) as wdb:
wdb.conn.execute(
"DELETE FROM ema52_screening WHERE freq = ?",
[freq_label],
)
wdb.conn.executemany(
"""INSERT INTO ema52_screening
(trade_date, ts_code, name, freq, close_price, ema52, distance_pct, amount)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT (trade_date, ts_code, freq) DO UPDATE SET
close_price=excluded.close_price,
ema52=excluded.ema52,
distance_pct=excluded.distance_pct,
amount=excluded.amount,
name=excluded.name,
updated_at=now()""",
[(r["trade_date"], r["ts_code"], r["name"],
r["freq"], r["close_price"], r["ema52"], r["distance_pct"], r["amount"])
for r in results],
)
except Exception as e:
logger.error(f"EMA52: failed to save {freq_label}: {e}")
continue
logger.info(
f"EMA52 {freq_label}: {len(results)} near "
f"{threshold * 100:.1f}%)"
)
logger.info("EMA52 screening complete")
def _query_parquet(sql: str) -> "pd.DataFrame":
"""Execute a SQL query against Parquet files using a transient in-memory DuckDB."""
import duckdb
import pandas as pd
conn = duckdb.connect()
try:
return conn.execute(sql).fetchdf()
finally:
conn.close()
def _compute_ema(values: "np.ndarray", period: int) -> "np.ndarray":
"""Compute EMA for a 1D array of values."""
import numpy as np
if len(values) < period:
return np.array([])
alpha = 2.0 / (period + 1)
ema = np.zeros(len(values))
ema[period - 1] = np.mean(values[:period])
for i in range(period, len(values)):
ema[i] = alpha * values[i] + (1 - alpha) * ema[i - 1]
return ema
+134
View File
@@ -0,0 +1,134 @@
"""ts_code conversion — the single implementation for the entire platform.
All code ↔ ts_code ↔ sina_symbol conversions live here.
Every function is IDEMPOTENT: feeding an already-converted value
returns it unchanged (this prevents the double-suffix corruption
that previously produced values like "000001.SZ.SZ").
Formats:
- bare code: "000001" (6-digit numeric string)
- ts_code: "000001.SZ" (code + exchange suffix)
- sina symbol: "sz000001" (lowercase exchange prefix + code)
"""
from __future__ import annotations
import re
_TS_CODE_RE = re.compile(r"^(\d{6})\.(SH|SZ|BJ)$")
_SINA_RE = re.compile(r"^(sh|sz|bj)(\d{6})$")
VALID_SUFFIXES = (".SH", ".SZ", ".BJ")
def exchange_for_code(code: str) -> str:
"""Determine exchange suffix from a bare 6-digit code.
Rules:
- 4xxxxx, 8xxxxx, 92xxxx → BJ (Beijing)
- 6xxxxx, 9xxxxx → SH (Shanghai)
- everything else → SZ (Shenzhen)
"""
code = str(code).strip()
if code.startswith(("4", "8")) or code.startswith("92"):
return "BJ"
if code.startswith(("6", "9")):
return "SH"
return "SZ"
def to_ts_code(value: str) -> str:
"""Convert any supported format to ts_code. IDEMPOTENT.
"000001""000001.SZ"
"000001.SZ""000001.SZ" (unchanged — never double-suffixes)
"sz000001""000001.SZ"
"000001.SZ.SZ""000001.SZ" (repairs legacy corruption)
"""
value = str(value).strip()
# Repair legacy double suffix first
value = normalize_ts_code(value)
# Already a valid ts_code?
if _TS_CODE_RE.match(value.upper()):
return value.upper()
# Sina symbol format?
m = _SINA_RE.match(value.lower())
if m:
prefix, code = m.groups()
return f"{code}.{prefix.upper()}"
# Bare numeric code
if value.isdigit():
code = value.zfill(6)
return f"{code}.{exchange_for_code(code)}"
# Unknown format — return as-is (caller's problem)
return value
def normalize_ts_code(ts_code: str) -> str:
"""Strip duplicated exchange suffixes. IDEMPOTENT.
"000001.SZ.SZ""000001.SZ"
"000001.SZ""000001.SZ"
"""
ts_code = str(ts_code).strip()
upper = ts_code.upper()
for suffix in VALID_SUFFIXES:
doubled = suffix + suffix
if upper.endswith(doubled):
return upper[: -len(suffix)]
return ts_code
def to_sina_symbol(value: str) -> str:
"""Convert to Sina symbol format. IDEMPOTENT.
"000001.SZ""sz000001"
"000001""sz000001"
"sz000001""sz000001" (unchanged)
"""
value = str(value).strip()
if _SINA_RE.match(value.lower()):
return value.lower()
ts = to_ts_code(value)
m = _TS_CODE_RE.match(ts)
if m:
code, exchange = m.groups()
return f"{exchange.lower()}{code}"
return value
def bare_code(value: str) -> str:
"""Extract the bare 6-digit code from any format.
"000001.SZ""000001"
"sz000001""000001"
"""
ts = to_ts_code(value)
m = _TS_CODE_RE.match(ts)
if m:
return m.group(1)
return value
def derive_market(ts_code: str) -> str:
"""Derive market segment (板块) from ts_code.
6xxxxx.SH (non-688) → 主板
688xxx.SH → 科创板
0xxxxx.SZ → 主板
3xxxxx.SZ → 创业板
*.BJ → 北交所
"""
ts = to_ts_code(ts_code).upper()
if ts.endswith(".SH"):
return "科创板" if ts.startswith("688") else "主板"
if ts.endswith(".SZ"):
return "创业板" if ts.startswith("3") else "主板"
if ts.endswith(".BJ"):
return "北交所"
return "其他"
+27
View File
@@ -85,3 +85,30 @@ INTRADAY_FREQS: tuple[Freq, ...] = (Freq.m1, Freq.m5, Freq.m15, Freq.m30, Freq.h
# Frequencies that are derived at query time # Frequencies that are derived at query time
DERIVED_FREQS: tuple[Freq, ...] = (Freq.h2,) DERIVED_FREQS: tuple[Freq, ...] = (Freq.h2,)
# ── Index codes ──
# ts_code format → Chinese name
INDEX_CODES: dict[str, str] = {
"000001.SH": "上证指数",
"399001.SZ": "深证成指",
"399006.SZ": "创业板指",
"000688.SH": "科创50",
"000300.SH": "沪深300",
"000905.SH": "中证500",
"000852.SH": "中证1000",
}
# Sina API symbol format for index backfill
INDEX_SINA_SYMBOLS: dict[str, str] = {
"sh000001": "000001.SH",
"sz399001": "399001.SZ",
"sz399006": "399006.SZ",
"sh000688": "000688.SH",
"sh000300": "000300.SH",
"sh000905": "000905.SH",
"sh000852": "000852.SH",
}
# derive_market moved to ashare_dp.core.codes
from ashare_dp.core.codes import derive_market # noqa: F401, E402
+1
View File
@@ -0,0 +1 @@
"""DATA PLATFORM — data acquisition, ETL, and storage. Knows nothing about trading."""
+1
View File
@@ -0,0 +1 @@
"""ETL pipelines: backfill, EOD, realtime."""
@@ -18,16 +18,17 @@ import pyarrow.parquet as pq
from loguru import logger from loguru import logger
from ashare_dp.config import Settings from ashare_dp.config import Settings
from ashare_dp.core.codes import to_ts_code
from ashare_dp.core.exceptions import StorageError from ashare_dp.core.exceptions import StorageError
from ashare_dp.core.models import ( from ashare_dp.core.models import (
BACKFILLABLE_FREQS, BACKFILLABLE_FREQS,
INTRADAY_FREQS, INTRADAY_FREQS,
Freq, Freq,
) )
from ashare_dp.data.akshare_client import AKShareClient from ashare_dp.data.sources.akshare_client import AKShareClient
from ashare_dp.storage.database import get_db from ashare_dp.data.store.database import get_db
from ashare_dp.storage.repository import KLineRepository, _standardize_df from ashare_dp.data.store.repository import KLineRepository, _standardize_df
from ashare_dp.storage.partitioning import partition_glob from ashare_dp.data.store.partitioning import partition_glob
settings = Settings() settings = Settings()
@@ -78,16 +79,6 @@ LEGACY_MIN_COLUMN_MAPPING = {
REQUIRED_COLS = ["ts_code", "trade_time", "open", "high", "low", "close", "volume", "amount"] REQUIRED_COLS = ["ts_code", "trade_time", "open", "high", "low", "close", "volume", "amount"]
def _to_ts_code(symbol: str) -> str:
"""Convert 6-digit symbol to ts_code with exchange suffix."""
code = str(symbol).zfill(6)
if code.startswith(("4", "8")) or code.startswith("92"):
return f"{code}.BJ"
elif code.startswith("6") or code.startswith("9"):
return f"{code}.SH"
return f"{code}.SZ"
def _normalize_hist_df(df: pd.DataFrame, symbol: str, freq: Freq) -> pd.DataFrame: def _normalize_hist_df(df: pd.DataFrame, symbol: str, freq: Freq) -> pd.DataFrame:
"""Normalize daily K-line output to standard K-line schema.""" """Normalize daily K-line output to standard K-line schema."""
df = df.copy() df = df.copy()
@@ -103,7 +94,7 @@ def _normalize_hist_df(df: pd.DataFrame, symbol: str, freq: Freq) -> pd.DataFram
break break
if "ts_code" not in df.columns: if "ts_code" not in df.columns:
df["ts_code"] = _to_ts_code(symbol) df["ts_code"] = to_ts_code(symbol)
if "trade_time" in df.columns: if "trade_time" in df.columns:
df["trade_time"] = pd.to_datetime(df["trade_time"]) df["trade_time"] = pd.to_datetime(df["trade_time"])
@@ -135,7 +126,7 @@ def _normalize_min_df(df: pd.DataFrame, symbol: str) -> pd.DataFrame:
break break
if "ts_code" not in df.columns: if "ts_code" not in df.columns:
df["ts_code"] = _to_ts_code(symbol) df["ts_code"] = to_ts_code(symbol)
if "trade_time" in df.columns: if "trade_time" in df.columns:
df["trade_time"] = pd.to_datetime(df["trade_time"]) df["trade_time"] = pd.to_datetime(df["trade_time"])
@@ -189,16 +180,6 @@ def _resample_daily_to_period(df: pd.DataFrame, freq: Freq) -> pd.DataFrame:
return result return result
def _code_to_ts_code(code: str) -> str:
"""Convert bare 6-digit code to ts_code format: '000001' -> '000001.SZ'."""
code = str(code).zfill(6)
if code.startswith(("4", "8")) or code.startswith("92"):
return f"{code}.BJ"
elif code.startswith("6") or code.startswith("9"):
return f"{code}.SH"
return f"{code}.SZ"
WRITE_BATCH = 500 # Write to disk every N stocks to limit memory WRITE_BATCH = 500 # Write to disk every N stocks to limit memory
@@ -217,7 +198,7 @@ class BackfillPipeline:
def init_db(self): def init_db(self):
"""Initialize database schema (tables and views).""" """Initialize database schema (tables and views)."""
from ashare_dp.storage.schema import DDL_STATEMENTS from ashare_dp.data.store.schema import DDL_STATEMENTS
logger.info("Initializing database schema...") logger.info("Initializing database schema...")
with get_db(read_only=False) as db: with get_db(read_only=False) as db:
@@ -394,8 +375,8 @@ class BackfillPipeline:
chunk = pd.concat(batch_frames, ignore_index=True) chunk = pd.concat(batch_frames, ignore_index=True)
# Standardize: add freq, keep only standard columns # Standardize: add freq, keep only standard columns
chunk["freq"] = Freq.d1.value chunk["freq"] = Freq.d1.value
# Convert bare codes (000001) to ts_code format (000001.SZ) # Normalize ts_code (idempotent — never double-suffixes)
chunk["ts_code"] = chunk["ts_code"].astype(str).apply(_code_to_ts_code) chunk["ts_code"] = chunk["ts_code"].astype(str).apply(to_ts_code)
chunk = _standardize_df(chunk) chunk = _standardize_df(chunk)
total_records += len(chunk) total_records += len(chunk)
table = pa.Table.from_pandas(chunk, preserve_index=False) table = pa.Table.from_pandas(chunk, preserve_index=False)
@@ -451,6 +432,7 @@ class BackfillPipeline:
logger.info(f"Merging {batch_idx} temp batches into final parquet...") logger.info(f"Merging {batch_idx} temp batches into final parquet...")
t0 = time.monotonic() t0 = time.monotonic()
temp_glob = str(temp_dir / "batch_*.parquet") temp_glob = str(temp_dir / "batch_*.parquet")
import duckdb
copy_conn = duckdb.connect() copy_conn = duckdb.connect()
copy_conn.execute(f""" copy_conn.execute(f"""
COPY ( COPY (
@@ -535,6 +517,7 @@ class BackfillPipeline:
logger.info(f"Deriving {freq.value} with single DuckDB scan...") logger.info(f"Deriving {freq.value} with single DuckDB scan...")
t0 = time.monotonic() t0 = time.monotonic()
try: try:
import duckdb
derive_conn = duckdb.connect() derive_conn = duckdb.connect()
df = derive_conn.execute(sql).fetchdf() df = derive_conn.execute(sql).fetchdf()
derive_conn.close() derive_conn.close()
@@ -14,9 +14,9 @@ from loguru import logger
from ashare_dp.core.calendar import BEIJING_TZ from ashare_dp.core.calendar import BEIJING_TZ
from ashare_dp.core.models import BACKFILLABLE_FREQS, INTRADAY_FREQS, Freq from ashare_dp.core.models import BACKFILLABLE_FREQS, INTRADAY_FREQS, Freq
from ashare_dp.data.akshare_client import AKShareClient from ashare_dp.data.sources.akshare_client import AKShareClient
from ashare_dp.data.backfill import _normalize_hist_df, _normalize_min_df, _resample_daily_to_period from ashare_dp.data.pipelines.backfill import _normalize_hist_df, _normalize_min_df, _resample_daily_to_period
from ashare_dp.storage.repository import KLineRepository from ashare_dp.data.store.repository import KLineRepository
class EODPipeline: class EODPipeline:
@@ -11,32 +11,13 @@ from datetime import datetime
from loguru import logger from loguru import logger
from ashare_dp.api.websocket.manager import manager from ashare_dp.core.codes import to_ts_code
from ashare_dp.apps.api.websocket.manager import manager
from ashare_dp.core.calendar import BEIJING_TZ, MarketState, determine_market_state from ashare_dp.core.calendar import BEIJING_TZ, MarketState, determine_market_state
from ashare_dp.data.akshare_client import AKShareClient from ashare_dp.data.sources.akshare_client import AKShareClient
def _code_to_ts_code(code: str) -> str:
"""Convert a raw stock code to ts_code format.
Handles both Sina format (with exchange prefix, e.g. 'sh600000')
and numeric-only format (e.g. '000001').
"""
code = str(code)
# Sina format: 'sh600000', 'sz000001', 'bj920000'
if code[:2].isalpha() and len(code) == 8:
prefix = code[:2].lower()
num = code[2:]
exchange_map = {"sh": "SH", "sz": "SZ", "bj": "BJ"}
return f"{num}.{exchange_map.get(prefix, prefix.upper())}"
# Numeric format
code = code.zfill(6)
if code.startswith(("4", "8")) or code.startswith("92"):
return f"{code}.BJ"
elif code.startswith("6") or code.startswith("9"):
return f"{code}.SH"
return f"{code}.SZ"
class RealtimePoller: class RealtimePoller:
"""Background task that polls market data and broadcasts to subscribers.""" """Background task that polls market data and broadcasts to subscribers."""
@@ -50,6 +31,7 @@ class RealtimePoller:
self._poll_interval = poll_interval self._poll_interval = poll_interval
self._running = False self._running = False
self._task: asyncio.Task | None = None self._task: asyncio.Task | None = None
self._last_snapshot: datetime | None = None
async def start(self): async def start(self):
"""Start the polling loop.""" """Start the polling loop."""
@@ -105,7 +87,7 @@ class RealtimePoller:
if df is not None and not df.empty: if df is not None and not df.empty:
# Normalize and filter # Normalize and filter
if "代码" in df.columns: if "代码" in df.columns:
df["ts_code"] = df["代码"].apply(_code_to_ts_code) df["ts_code"] = df["代码"].apply(to_ts_code)
spot_data = {} spot_data = {}
for _, row in df.iterrows(): for _, row in df.iterrows():
@@ -129,6 +111,17 @@ class RealtimePoller:
except Exception as e: except Exception as e:
logger.error(f"Spot poll error: {e}") logger.error(f"Spot poll error: {e}")
# Intraday state snapshot every 30 minutes
if state == MarketState.TRADING and (
self._last_snapshot is None
or (now - self._last_snapshot).total_seconds() >= 1800
):
try:
await asyncio.to_thread(_save_intraday_snapshot, now)
self._last_snapshot = now
except Exception as e:
logger.warning(f"Intraday snapshot failed: {e}")
# Heartbeat every 30 seconds # Heartbeat every 30 seconds
if (now - last_heartbeat).total_seconds() >= 30: if (now - last_heartbeat).total_seconds() >= 30:
await manager.broadcast({ await manager.broadcast({
@@ -144,5 +137,73 @@ class RealtimePoller:
await asyncio.sleep(self._get_poll_interval(state)) await asyncio.sleep(self._get_poll_interval(state))
def _save_intraday_snapshot(now: datetime):
"""Compute and save a lightweight intraday MarketState from spot data."""
import json
from ashare_dp.data.store.database import analytics_conn
from ashare_dp.data.sources.akshare_client import AKShareClient
client = AKShareClient()
df = client.get_spot()
if df is None or df.empty:
return
# Lightweight breadth from spot: advance/decline from pct_chg
pct_col = None
for c in ["涨跌幅", "pct_chg"]:
if c in df.columns:
pct_col = c
break
if pct_col is None:
return
pct_vals = df[pct_col].dropna().astype(float)
total = len(pct_vals)
if total == 0:
return
advances = int((pct_vals > 0).sum())
declines = int((pct_vals < 0).sum())
advance_ratio = advances / total
# Compute volume if available
total_amount = 0.0
amt_col = None
for c in ["成交额", "amount"]:
if c in df.columns:
amt_col = c
break
if amt_col:
total_amount = float(df[amt_col].dropna().sum())
# Lightweight state
state_dict = {
"version": "1.0",
"source": "intraday_spot",
"timestamp": now.isoformat(),
"trade_date": now.date().isoformat(),
"dimensions": {
"trend": round(advance_ratio, 4),
"fear": round(1.0 - advance_ratio, 4),
"liquidity": 0.5,
"rotation": 0.0,
"participation": round(advance_ratio, 4),
"volatility": 0.0,
"breadth": round(advance_ratio, 4),
},
"confidence": 0.5,
"quality": 0.5,
}
conn = analytics_conn(read_only=False)
try:
conn.execute(
"INSERT OR REPLACE INTO state_snapshot (entity, timeframe, timestamp, state) "
"VALUES (?, ?, ?, ?)",
("market", "30min", now, json.dumps(state_dict, ensure_ascii=False)),
)
finally:
conn.close()
# Global poller instance # Global poller instance
poller = RealtimePoller() poller = RealtimePoller()
+1
View File
@@ -0,0 +1 @@
"""External data sources: akshare, index, industry."""
@@ -24,6 +24,7 @@ import urllib.request
from loguru import logger from loguru import logger
from ashare_dp.config import Settings from ashare_dp.config import Settings
from ashare_dp.core.codes import to_sina_symbol, to_ts_code
from ashare_dp.core.exceptions import DataSourceError from ashare_dp.core.exceptions import DataSourceError
settings = Settings() settings = Settings()
@@ -66,33 +67,6 @@ def _to_akshare_date(d: date) -> str:
return d.strftime("%Y%m%d") return d.strftime("%Y%m%d")
def _code_to_sina_symbol(code: str) -> str:
"""Convert numeric stock code to Sina symbol format.
'000001' -> 'sz000001', '600000' -> 'sh600000', '920000' -> 'bj920000'
"""
code = str(code).zfill(6)
if code.startswith(("4", "8")) or code.startswith("92"):
return f"bj{code}"
elif code.startswith("6") or code.startswith("9"):
return f"sh{code}"
else:
return f"sz{code}"
def _sina_symbol_to_ts_code(symbol: str) -> str:
"""Convert Sina symbol to ts_code format.
'sh600000' -> '600000.SH', 'sz000001' -> '000001.SZ', 'bj920000' -> '920000.BJ'
"""
match = re.match(r'^([a-z]+)(\d{6})$', str(symbol))
if match:
prefix, code = match.groups()
exchange = prefix.upper()
return f"{code}.{exchange}"
return str(symbol)
class AKShareClient: class AKShareClient:
"""Wrapper around akshare with retry, error handling, and dual backend. """Wrapper around akshare with retry, error handling, and dual backend.
@@ -228,7 +202,7 @@ class AKShareClient:
self, symbol: str, start_date: str, end_date: str, adjust: str self, symbol: str, start_date: str, end_date: str, adjust: str
) -> pd.DataFrame: ) -> pd.DataFrame:
import akshare as ak import akshare as ak
sina_symbol = _code_to_sina_symbol(symbol) sina_symbol = to_sina_symbol(symbol)
return self._retry(lambda: ak.stock_zh_a_daily( return self._retry(lambda: ak.stock_zh_a_daily(
symbol=sina_symbol, symbol=sina_symbol,
start_date=start_date, start_date=start_date,
@@ -286,7 +260,7 @@ class AKShareClient:
start_date: str | None, end_date: str | None, start_date: str | None, end_date: str | None,
) -> pd.DataFrame: ) -> pd.DataFrame:
import akshare as ak import akshare as ak
sina_symbol = _code_to_sina_symbol(symbol) sina_symbol = to_sina_symbol(symbol)
df = self._retry(lambda: ak.stock_zh_a_minute( df = self._retry(lambda: ak.stock_zh_a_minute(
symbol=sina_symbol, period=period, symbol=sina_symbol, period=period,
)) ))
+117
View File
@@ -0,0 +1,117 @@
"""Index daily K-line backfill via Sina API.
Fetches 7 major A-share indices and stores them in the same
Hive-partitioned Parquet format as stock K-lines.
Uses akshare.stock_zh_index_daily() which works via Sina
(verified accessible outside China).
"""
from __future__ import annotations
from datetime import date, datetime
import pandas as pd
from loguru import logger
from ashare_dp.core.models import INDEX_SINA_SYMBOLS, Freq
from ashare_dp.data.store.repository import KLineRepository
def backfill_indices(
repo: KLineRepository | None = None,
start_date: str = "19900101",
end_date: str | None = None,
) -> dict[str, int]:
"""Fetch and store index daily K-line history.
Args:
repo: KLineRepository instance. Created if None.
start_date: Start date in YYYYMMDD format.
end_date: End date in YYYYMMDD format (default: today).
Returns:
Dict mapping ts_code → number of bars stored.
"""
if repo is None:
repo = KLineRepository()
if end_date is None:
end_date = date.today().strftime("%Y%m%d")
import akshare as ak
results: dict[str, int] = {}
for sina_sym, ts_code in INDEX_SINA_SYMBOLS.items():
try:
logger.info(f"Fetching index {sina_sym} ({ts_code})...")
df = ak.stock_zh_index_daily(symbol=sina_sym)
if df is None or df.empty:
logger.warning(f"No data for {sina_sym}")
results[ts_code] = 0
continue
# Normalize columns to match K-line standard schema
df = _normalize_index_df(df, ts_code)
# Filter by date range
df["trade_date"] = pd.to_datetime(df["trade_date"])
df = df[(df["trade_date"] >= start_date) & (df["trade_date"] <= end_date)]
if df.empty:
logger.warning(f"No data in range for {sina_sym}")
results[ts_code] = 0
continue
# Store
repo.write_klines(df, Freq.d1)
results[ts_code] = len(df)
logger.info(f" {sina_sym}: stored {len(df)} bars, "
f"{df['trade_date'].min().date()} ~ {df['trade_date'].max().date()}")
except Exception as e:
logger.error(f"Failed to backfill index {sina_sym}: {e}")
results[ts_code] = 0
total = sum(results.values())
logger.info(f"Index backfill complete: {total} total bars across {len(results)} indices")
return results
def _normalize_index_df(df: pd.DataFrame, ts_code: str) -> pd.DataFrame:
"""Normalize index DataFrame to match stock K-line standard schema.
Sina index columns: date, open, high, low, close, volume
Missing: amount (not available for indices from Sina)
"""
df = df.copy()
# Standardize column names
col_map = {
"date": "trade_time",
"open": "open",
"high": "high",
"low": "low",
"close": "close",
"volume": "volume",
}
df = df.rename(columns=col_map)
# Add required columns
df["ts_code"] = ts_code
df["trade_date"] = pd.to_datetime(df["trade_time"])
df["freq"] = Freq.d1.value
# Index data from Sina doesn't have amount
if "amount" not in df.columns:
df["amount"] = 0.0
# Ensure volume is int
df["volume"] = df["volume"].fillna(0).astype("int64")
# Select and order standard columns
std_cols = ["ts_code", "trade_time", "trade_date", "open", "high", "low", "close", "volume", "amount", "freq"]
df = df[[c for c in std_cols if c in df.columns]]
return df
+165
View File
@@ -0,0 +1,165 @@
"""Industry classification data fetch and storage.
Fetches industry classifications from akshare and stores in DuckDB.
Used by rotation features and dashboard sector analysis.
"""
from __future__ import annotations
import pandas as pd
from loguru import logger
from ashare_dp.core.codes import to_ts_code
from ashare_dp.data.store.database import get_db
def fetch_industry_mapping() -> pd.DataFrame:
"""Fetch stock-to-industry mapping from akshare.
Tries East Money first (richer data), falls back to Sina.
Returns DataFrame with columns: ts_code, industry_name
"""
logger.info("Fetching industry classifications...")
try:
import akshare as ak
# Try East Money industry constituent data
df = ak.stock_board_industry_cons_em()
if df is not None and not df.empty:
logger.info(f"East Money industry data: {len(df)} rows")
# Expected columns vary by akshare version
# Common: '代码', '名称', '所属行业' or similar
cols = df.columns.tolist()
logger.debug(f"Industry columns: {cols}")
# Normalize to ts_code + industry_name
code_col = None
name_col = None
for c in cols:
if "代码" in c or c.lower() in ("code", "symbol"):
code_col = c
if "行业" in c or c.lower() in ("industry", "board"):
name_col = c
if code_col and name_col:
result = pd.DataFrame({
"code": df[code_col].astype(str).str.zfill(6),
"industry_name": df[name_col].astype(str),
})
# Convert code to ts_code format
result["ts_code"] = result["code"].apply(to_ts_code)
result = result[["ts_code", "industry_name"]].drop_duplicates(subset=["ts_code"])
logger.info(f"Normalized {len(result)} stock-industry mappings")
return result
except Exception as e:
logger.warning(f"East Money industry fetch failed: {e}")
# Fallback: try Sina
try:
import akshare as ak
df = ak.stock_info_a_code_name()
if df is not None and not df.empty:
# Sina stock_info doesn't have industry, but we can try other endpoints
logger.info("Sina fallback: no industry data available from basic stock info")
except Exception as e2:
logger.warning(f"Sina fallback also failed: {e2}")
# Last resort: derive a simple industry grouping from stock names
logger.warning("No industry API available. Using minimal fallback.")
return _minimal_fallback()
def _minimal_fallback() -> pd.DataFrame:
"""Generate minimal industry mapping from existing stock_info.
Phase 1: if no API works, use market + simple heuristics.
The rotation module will fall back to market-based segments.
"""
with get_db(read_only=True) as db:
result = db.query("SELECT ts_code, name, market FROM stock_info")
rows = []
for ts_code, name, market in result:
industry = _guess_industry_from_name(name, ts_code)
rows.append({"ts_code": ts_code, "industry_name": industry})
df = pd.DataFrame(rows)
logger.info(f"Generated {len(df)} fallback industry mappings")
return df
def _guess_industry_from_name(name: str, ts_code: str) -> str:
"""Guess industry from stock name using keyword matching."""
name = str(name)
keywords = [
("银行", "银行"),
("证券", "证券"),
("保险", "保险"),
("房地产", "房地产"),
("医药", "医药"),
("汽车", "汽车"),
("钢铁", "钢铁"),
("煤炭", "煤炭"),
("电力", "电力"),
("石油", "石油"),
("化工", "化工"),
("有色", "有色"),
("食品", "食品饮料"),
("饮料", "食品饮料"),
("白酒", "白酒"),
("家电", "家电"),
("半导体", "半导体"),
("芯片", "半导体"),
("通信", "通信"),
("软件", "软件"),
("计算机", "计算机"),
("传媒", "传媒"),
("军工", "军工"),
("航空", "军工"),
("机器人", "机器人"),
("光伏", "光伏"),
("锂电", "新能源"),
("电池", "新能源"),
("风电", "新能源"),
("储能", "新能源"),
("新能源", "新能源"),
("AI", "AI"),
("算力", "AI"),
("光模块", "AI"),
("PCB", "PCB"),
("船舶", "船舶"),
("港口", "交通运输"),
("铁路", "交通运输"),
("公路", "交通运输"),
("航空运输", "交通运输"),
("建筑", "建筑"),
("建材", "建材"),
("环保", "环保"),
]
for keyword, industry in keywords:
if keyword in name:
return industry
return "其他"
def store_industry_mapping(df: pd.DataFrame) -> int:
"""Store industry mappings in DuckDB stock_industry table.
Returns number of rows stored.
"""
with get_db(read_only=False) as db:
try:
for _, row in df.iterrows():
db.execute(
"INSERT OR REPLACE INTO stock_industry (ts_code, industry_name) VALUES (?, ?)",
(row["ts_code"], row["industry_name"]),
)
count_result = db.query("SELECT COUNT(*) FROM stock_industry")
count = count_result[0][0] if count_result else 0
logger.info(f"Stored {count} industry mappings in stock_industry")
return int(count)
except Exception as e:
logger.error(f"Failed to store industry data: {e}")
return 0
+6
View File
@@ -0,0 +1,6 @@
"""Storage layer: DuckDB + Parquet."""
from ashare_dp.data.store.database import get_db, Database, analytics_conn
from ashare_dp.data.store.repository import KLineRepository
from ashare_dp.data.store.partitioning import partition_glob, partition_path
__all__ = ["get_db", "Database", "analytics_conn", "KLineRepository", "partition_glob", "partition_path"]
@@ -99,3 +99,31 @@ def get_db(read_only: bool = True) -> Iterator[Database]:
yield db yield db
finally: finally:
db.close() db.close()
def analytics_conn(read_only: bool = True) -> "duckdb.DuckDBPyConnection":
"""Raw DuckDB connection for analytics queries (features, engines).
The single sanctioned way to open a DuckDB connection outside the
Database wrapper. Uses configured duckdb_path never hardcode paths.
Caller must close() the connection.
"""
return duckdb.connect(settings.duckdb_path, read_only=read_only)
def kline_glob(freq=None) -> str:
"""Parquet glob for K-line data at the given frequency (default: daily).
Convenience re-export so analytics modules don't hardcode paths.
"""
from ashare_dp.core.models import Freq
from ashare_dp.data.store.partitioning import partition_glob
return partition_glob(freq or Freq.d1)
def read_parquet_sql(glob_var: str) -> str:
"""Standard read_parquet SQL snippet with hive partitioning.
Single place for the 28 duplicate 'read_parquet({glob}, hive_partitioning=true, union_by_name=true)' patterns.
"""
return f"read_parquet('{glob_var}', hive_partitioning=true, union_by_name=true)"
@@ -15,8 +15,8 @@ from loguru import logger
from ashare_dp.config import Settings from ashare_dp.config import Settings
from ashare_dp.core.exceptions import QueryError, StorageError from ashare_dp.core.exceptions import QueryError, StorageError
from ashare_dp.core.models import DERIVED_FREQS, Freq from ashare_dp.core.models import DERIVED_FREQS, Freq
from ashare_dp.storage.database import get_db from ashare_dp.data.store.database import get_db
from ashare_dp.storage.partitioning import ( from ashare_dp.data.store.partitioning import (
ensure_partition_dir, ensure_partition_dir,
partition_glob, partition_glob,
) )
+138
View File
@@ -0,0 +1,138 @@
"""Database schema: DDL statements for DuckDB tables and views."""
from __future__ import annotations
# DDL for persistent tables
DDL_STATEMENTS = [
"""
CREATE TABLE IF NOT EXISTS stock_info (
ts_code VARCHAR(9) PRIMARY KEY,
symbol VARCHAR(6) NOT NULL,
name VARCHAR(40) NOT NULL,
exchange VARCHAR(2) NOT NULL,
area VARCHAR(20),
industry VARCHAR(40),
list_date DATE,
delist_date DATE,
market VARCHAR(10),
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
""",
"""
CREATE INDEX IF NOT EXISTS idx_stock_symbol ON stock_info(symbol)
""",
"""
CREATE INDEX IF NOT EXISTS idx_stock_exchange ON stock_info(exchange)
""",
"""
CREATE TABLE IF NOT EXISTS trading_calendar (
trade_date DATE PRIMARY KEY,
is_trading_day BOOLEAN NOT NULL DEFAULT TRUE,
week_day TINYINT NOT NULL,
year SMALLINT NOT NULL,
month TINYINT NOT NULL
)
""",
"""
CREATE SEQUENCE IF NOT EXISTS seq_ema52_id
""",
"""
CREATE TABLE IF NOT EXISTS ema52_screening (
id BIGINT PRIMARY KEY DEFAULT nextval('seq_ema52_id'),
trade_date DATE NOT NULL,
ts_code VARCHAR(9) NOT NULL,
name VARCHAR(40),
freq VARCHAR(3) NOT NULL,
close_price DOUBLE NOT NULL,
ema52 DOUBLE NOT NULL,
distance_pct DOUBLE NOT NULL,
amount DOUBLE,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
UNIQUE(trade_date, ts_code, freq)
)
""",
"""
CREATE INDEX IF NOT EXISTS idx_ema52_date ON ema52_screening(trade_date)
""",
# Migration: add amount column if upgrading from older schema
"""
ALTER TABLE ema52_screening ADD COLUMN IF NOT EXISTS amount DOUBLE
""",
# ── Dashboard / Trading OS tables ──
"""
CREATE TABLE IF NOT EXISTS stock_industry (
ts_code VARCHAR(9) NOT NULL,
industry_name VARCHAR(40) NOT NULL,
industry_code VARCHAR(20),
PRIMARY KEY (ts_code)
)
""",
"""
CREATE INDEX IF NOT EXISTS idx_stock_industry_name ON stock_industry(industry_name)
""",
"""
CREATE TABLE IF NOT EXISTS state_snapshot (
entity VARCHAR(20) NOT NULL,
timeframe VARCHAR(10) NOT NULL,
timestamp TIMESTAMP NOT NULL,
state JSON NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (entity, timeframe, timestamp)
)
""",
# ── Signal Intelligence ──
"""
CREATE TABLE IF NOT EXISTS signal_instance (
id BIGINT PRIMARY KEY DEFAULT nextval('seq_signal_id'),
signal_type VARCHAR(30) NOT NULL,
ts_code VARCHAR(15) NOT NULL,
trade_date DATE NOT NULL,
signal_price DOUBLE NOT NULL,
-- Market state at signal time
state_trend DOUBLE,
state_fear DOUBLE,
state_liquidity DOUBLE,
state_rotation DOUBLE,
state_participation DOUBLE,
state_volatility DOUBLE,
state_breadth DOUBLE,
-- Forward outcomes
return_5d DOUBLE,
return_10d DOUBLE,
return_20d DOUBLE,
max_return_5d DOUBLE,
max_drawdown_5d DOUBLE,
holding_days INTEGER,
outcome_known BOOLEAN DEFAULT FALSE,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
UNIQUE(signal_type, ts_code, trade_date)
)
""",
"""
CREATE SEQUENCE IF NOT EXISTS seq_signal_id
""",
"""
CREATE INDEX IF NOT EXISTS idx_signal_type_date ON signal_instance(signal_type, trade_date)
""",
"""
CREATE INDEX IF NOT EXISTS idx_signal_ts_code ON signal_instance(ts_code, trade_date)
""",
# ── Trading Memory ──
"""
CREATE TABLE IF NOT EXISTS trade_log (
id BIGINT PRIMARY KEY DEFAULT nextval('seq_trade_id'),
ts_code VARCHAR(15) NOT NULL, trade_date DATE NOT NULL,
entry_price DOUBLE NOT NULL, entry_time TIMESTAMP,
signal_type VARCHAR(30), position_pct DOUBLE DEFAULT 0,
exit_price DOUBLE, exit_time TIMESTAMP, exit_reason VARCHAR(20),
return_pct DOUBLE, holding_days INTEGER,
max_favorable DOUBLE, max_adverse DOUBLE,
state_trend DOUBLE, state_fear DOUBLE, state_liquidity DOUBLE, state_breadth DOUBLE,
tags VARCHAR(200), notes VARCHAR(500),
closed BOOLEAN DEFAULT FALSE, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
""",
"""CREATE SEQUENCE IF NOT EXISTS seq_trade_id""",
"""CREATE INDEX IF NOT EXISTS idx_trade_date ON trade_log(trade_date)""",
"""CREATE INDEX IF NOT EXISTS idx_trade_signal ON trade_log(signal_type)""",
]
+35
View File
@@ -0,0 +1,35 @@
"""Domain objects for the A-Share Data Platform.
These are the foundational types shared by all modules:
Feature Store, Market State Engine, Decision Engine, Dashboard, and future
Signal Expectancy Engine / AI Agent / Backtester.
"""
from ashare_dp.domain.state import MarketState
from ashare_dp.domain.events import RiskEvent, RotationEdge, RotationGraph
from ashare_dp.domain.features import FeatureDefinition
from ashare_dp.domain.context import (
TradingContext, Playbook, Expectancy, Opportunity,
Allocation, FlowEdge, FlowGraph,
)
from ashare_dp.domain.leadership import LeaderState
from ashare_dp.domain.signal import SignalType, SignalInstance
__all__ = [
"MarketState",
"RiskEvent",
"RotationEdge",
"RotationGraph",
"FeatureDefinition",
"TradingContext",
"Playbook",
"Expectancy",
"Opportunity",
"Allocation",
"FlowEdge",
"FlowGraph",
"LeaderState",
"SignalType",
"SignalInstance",
]
+120
View File
@@ -0,0 +1,120 @@
"""TradingContext and related domain objects — the true center of the Trading OS.
MarketState describes the market. TradingContext describes a trading decision.
Signal Expectancy Engine queries history against this.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from datetime import datetime, date
from typing import Optional
from ashare_dp.domain.state import MarketState
@dataclass
class Allocation:
"""Suggested capital allocation based on market state."""
cash_pct: float = 0.4
trend_pct: float = 0.4
trial_pct: float = 0.2
reasoning: str = ""
@dataclass
class Playbook:
"""Complete trading playbook — not just a recommendation list.
Tells the trader: what strategies work, what to avoid,
how long to hold, how much to risk.
"""
suitable_strategies: list[str] = field(default_factory=list)
unsuitable_strategies: list[str] = field(default_factory=list)
holding_time: str = "2~4 Days"
risk_budget: str = "Medium"
position_sizing: str = "Standard"
focus_sectors: list[str] = field(default_factory=list)
avoid_sectors: list[str] = field(default_factory=list)
bias: str = "NEUTRAL"
confidence: float = 0.5
@dataclass
class Expectancy:
"""Historical expectancy for a signal in the current market context.
This is the moat. Query: Signal × MarketState → historical outcomes.
Phase 1: rule-based heuristic. Phase 2: real historical stats.
"""
signal_type: str = ""
win_rate: float = 0.0
avg_return: float = 0.0
max_drawdown: float = 0.0
avg_holding_days: float = 0.0
sample_count: int = 0
confidence: float = 0.0
similar_states: int = 0
@dataclass
class Opportunity:
"""A trading opportunity — not an industry score, a tradable object.
AI Agent can consume this directly without re-reasoning.
"""
name: str = ""
score: float = 0.0
lifecycle: str = "" # "Early" | "Accelerating" | "Peak" | "Declining"
persistence_days: int = 0
catalyst: str = "" # "资金轮动" | "政策催化" | "业绩驱动"
leader_stock: str = ""
invalidation: str = "" # what would invalidate this opportunity
risk: str = ""
top_stocks: list[str] = field(default_factory=list)
@dataclass
class FlowEdge:
"""Money flow edge in the flow graph."""
source: str = ""
target: str = ""
magnitude: str = "" # "↓↓→↑↑"
strength: float = 0.0
@dataclass
class FlowGraph:
"""Money flow as a directed graph."""
flows: list[FlowEdge] = field(default_factory=list)
net_inflow: dict[str, float] = field(default_factory=dict)
net_outflow: dict[str, float] = field(default_factory=dict)
@dataclass
class TradingContext:
"""The universal context for any trading decision.
Signal Expectancy Engine queries history against this object.
Every downstream consumer (Dashboard, AI, Backtest) reads this,
not raw features or individual engine outputs.
"""
timestamp: datetime = field(default_factory=datetime.now)
trade_date: date = field(default_factory=date.today)
# L1: Market State
state: Optional[MarketState] = None
# L2: Interpretation
leaders: dict = field(default_factory=dict) # industry → LeaderState
opportunities: list[Opportunity] = field(default_factory=list)
money_flow: Optional[FlowGraph] = None
# L3: Intelligence
playbook: Optional[Playbook] = None
expectancy: Optional[Expectancy] = None
allocation: Optional[Allocation] = None
risk_budget: str = "Medium"
# Narrative
narrative: str = ""
+73
View File
@@ -0,0 +1,73 @@
"""RiskEvent, RotationEdge, RotationGraph — structured domain events.
Dashboard renders them, AI explains them, Backtest counts them.
NOT raw dicts or strings.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from datetime import datetime
from typing import Any
@dataclass
class RiskEvent:
"""Structured risk warning emitted by the Risk Rule Engine.
Dashboard renders the message with color-coded severity.
AI Agent can explain the event context.
Backtester can count event frequency by type.
"""
type: str # "BreadthCollapse", "VolumeDrying", "LeaderFailure", etc.
severity: float # 0.01.0, how severe
level: str # "warning" | "danger"
affected: str # "All" | "主板" | "AI" | specific entity
confidence: float # 0.01.0, how confident the rule is in this detection
message: str # human-readable description
payload: dict = field(default_factory=dict) # type-specific data
timestamp: datetime = field(default_factory=datetime.now)
def to_dict(self) -> dict:
return {
"type": self.type,
"severity": round(self.severity, 4),
"level": self.level,
"affected": self.affected,
"confidence": round(self.confidence, 4),
"message": self.message,
"timestamp": self.timestamp.isoformat(),
}
@dataclass
class RotationEdge:
"""Directed edge in the rotation graph.
Represents capital flowing FROM one sector TO another,
inferred from turnover changes.
"""
source: str # sector name, e.g. "AI"
target: str # sector name, e.g. "机器人"
strength: float # 0.01.0, how strong the flow
confidence: float # 0.01.0
source_change: float # turnover change % for source (negative = outflow)
target_change: float # turnover change % for target (positive = inflow)
persistence_days: int # how many consecutive days this flow persisted
@dataclass
class RotationGraph:
"""Sector rotation as a directed graph.
Nodes = industries/concepts.
Edges = capital flow direction + strength.
Future: AI can do graph search over this structure.
"""
nodes: list[str] # all industry/concept names
edges: list[RotationEdge] # directed flows, sorted by strength desc
timestamp: datetime = field(default_factory=datetime.now)
+27
View File
@@ -0,0 +1,27 @@
"""FeatureDefinition — metadata for a registered feature.
Each feature in the Feature Registry is described by one of these.
The Engine code only knows about FeatureDefinitions, never imports
individual feature functions directly.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Callable
@dataclass
class FeatureDefinition:
"""Metadata for a registered market feature.
Adding a new feature means registering a FeatureDefinition.
No engine code changes needed.
"""
name: str # unique identifier, e.g. "breadth_vector"
category: str # "breadth", "volume", "trend", "fear", "rotation", "participation"
description: str # human-readable explanation
dependencies: list[str] = field(default_factory=list) # feature names this depends on
compute: Callable | None = None # the actual computation function (set after registration)
version: str = "1.0"
+95
View File
@@ -0,0 +1,95 @@
"""Knowledge Graph domain model — Theme → Concept → Industry → Leader.
The ontology that connects abstract trading themes (e.g. "AI", "新能源")
to concrete industries and stocks. Used by narrative generation, rotation
detection, and future AI agent reasoning.
"""
from __future__ import annotations
from dataclasses import dataclass, field
@dataclass
class Concept:
"""A mid-level concept within a theme — e.g. "算力" within "AI"."""
name: str # "算力"
industries: list[str] = field(default_factory=list) # industry names from stock_industry
keywords: list[str] = field(default_factory=list) # keywords to match stock names
leaders: list[str] = field(default_factory=list) # representative stocks
@dataclass
class Theme:
"""A high-level trading theme — e.g. "AI", "新能源", "机器人"."""
name: str
concepts: list[Concept] = field(default_factory=list)
relevance_score: float = 0.0 # 0-100, how active this theme is today
momentum: str = "" # "↑↑" | "↑" | "→" | "↓" | "↓↓"
@dataclass
class ThemeEdge:
"""Directed edge: money flowing from one theme to another."""
source: str
target: str
strength: float = 0.0
@dataclass
class ThemeGraph:
"""The complete theme landscape for today."""
themes: list[Theme] = field(default_factory=list)
edges: list[ThemeEdge] = field(default_factory=list)
# ═══════════════════════════════════════════════
# Curated A-Share Theme Knowledge Base
# ═══════════════════════════════════════════════
THEME_KNOWLEDGE: list[Theme] = [
Theme(name="AI", concepts=[
Concept(name="算力", industries=[], keywords=["算力", "服务器", "数据中心", "GPU", "CP", "曙光", "浪潮", "紫光", "中科曙光", "工业富联"], leaders=["工业富联", "中科曙光"]),
Concept(name="PCB", industries=[], keywords=["PCB", "电路板", "印制板", "沪电", "深南电路", "鹏鼎", "景旺", "胜宏"], leaders=["沪电股份", "深南电路"]),
Concept(name="光模块", industries=[], keywords=["光模块", "光通信", "中际", "新易盛", "天孚", "旭创", "剑桥", "光迅"], leaders=["中际旭创", "新易盛"]),
Concept(name="AI应用", industries=[], keywords=["AI应用", "大模型", "智能", "软件", "科大讯飞", "商汤", "云从", "格灵"], leaders=["科大讯飞"]),
Concept(name="半导体", industries=[], keywords=["半导体", "芯片", "晶圆", "封装", "设备", "韦尔", "兆易", "北方华创", "中微", "长电"], leaders=["北方华创"]),
]),
Theme(name="机器人", concepts=[
Concept(name="减速器", industries=[], keywords=["减速器", "谐波", "绿的", "双环传动", "中大力德"], leaders=["绿的谐波"]),
Concept(name="伺服电机", industries=[], keywords=["伺服", "电机", "汇川", "禾川", "埃斯顿", "雷赛"], leaders=["汇川技术", "埃斯顿"]),
Concept(name="传感器", industries=[], keywords=["传感器", "视觉", "力觉", "奥普特", "天准"], leaders=["奥普特"]),
Concept(name="本体集成", industries=[], keywords=["机器人", "新松", "拓斯达", "埃夫特", "博实"], leaders=["新松机器人"]),
]),
Theme(name="新能源", concepts=[
Concept(name="光伏", industries=[], keywords=["光伏", "隆基", "通威", "晶澳", "天合", "晶科", "阳光电源", "锦浪"], leaders=["隆基绿能", "阳光电源"]),
Concept(name="锂电", industries=[], keywords=["锂电", "宁德", "比亚迪", "亿纬", "国轩", "赣锋", "天齐", "华友", "中伟"], leaders=["宁德时代"]),
Concept(name="风电", industries=[], keywords=["风电", "金风", "明阳", "运达", "东方电缆", "大金重工"], leaders=["金风科技"]),
Concept(name="储能", industries=[], keywords=["储能", "派能", "科士达", "盛弘", "固德威", "德业"], leaders=["派能科技"]),
]),
Theme(name="消费", concepts=[
Concept(name="白酒", industries=[], keywords=["白酒", "茅台", "五粮液", "泸州老窖", "汾酒", "洋河", "古井"], leaders=["贵州茅台"]),
Concept(name="家电", industries=[], keywords=["家电", "美的", "格力", "海尔", "海信", "老板"], leaders=["美的集团"]),
Concept(name="食品饮料", industries=[], keywords=["食品", "饮料", "伊利", "海天", "双汇", "桃李", "安井"], leaders=["伊利股份"]),
]),
Theme(name="证券金融", concepts=[
Concept(name="证券", industries=[], keywords=["证券", "中信", "华泰", "国泰", "海通", "东方财富"], leaders=["东方财富"]),
Concept(name="保险", industries=[], keywords=["保险", "平安", "太保", "人寿", "新华"], leaders=["中国平安"]),
Concept(name="银行", industries=[], keywords=["银行", "招商", "兴业", "平安银行", "宁波银行"], leaders=["招商银行"]),
]),
Theme(name="军工", concepts=[
Concept(name="航空", industries=[], keywords=["航空", "发动机", "飞机", "沈飞", "西飞", "航发", "中航"], leaders=["中航沈飞"]),
Concept(name="船舶", industries=[], keywords=["船舶", "中国船舶", "中船", "重工"], leaders=["中国船舶"]),
Concept(name="导弹武器", industries=[], keywords=["导弹", "兵器", "光电", "航天"], leaders=["航天电器"]),
]),
Theme(name="医药", concepts=[
Concept(name="创新药", industries=[], keywords=["创新药", "恒瑞", "百济", "信达", "君实", "荣昌"], leaders=["恒瑞医药"]),
Concept(name="CXO", industries=[], keywords=["CXO", "药明", "康龙", "泰格", "凯莱英"], leaders=["药明康德"]),
Concept(name="医疗器械", industries=[], keywords=["医疗", "器械", "迈瑞", "联影", "鱼跃", "微创"], leaders=["迈瑞医疗"]),
]),
Theme(name="汽车", concepts=[
Concept(name="整车", industries=[], keywords=["汽车", "整车", "比亚迪", "长城", "长安", "吉利", "上汽"], leaders=["比亚迪"]),
Concept(name="零部件", industries=[], keywords=["零部件", "配件", "华域", "均胜", "拓普", "三花"], leaders=["拓普集团"]),
Concept(name="智能驾驶", industries=[], keywords=["智能驾驶", "自动驾驶", "ADAS", "德赛西威", "中科创达", "经纬恒润"], leaders=["德赛西威"]),
]),
]
+21
View File
@@ -0,0 +1,21 @@
"""LeaderLifecycle — full lifecycle model for sector leaders.
Not just Alive/Dead. Traders track the entire lifecycle:
Birth → Growing → Leading → Exhausted → Breaking → Dead → Recovering
"""
from __future__ import annotations
from enum import Enum
class LeaderState(str, Enum):
"""Full leader lifecycle — not binary Alive/Dead."""
BIRTH = "birth" # first breakout from base
GROWING = "growing" # accelerating, above MAs, volume expanding
LEADING = "leading" # at peak, driving sector, strongest momentum
EXHAUSTED = "exhausted" # extended from MA, volume fading
BREAKING = "breaking" # breaking below key MA (MA20)
DEAD = "dead" # below MA60, no recovery signal
RECOVERING = "recovering" # reclaiming MA20 after a break
UNKNOWN = "unknown" # insufficient data to classify
+82
View File
@@ -0,0 +1,82 @@
"""Signal domain objects — the core of Signal Intelligence.
Signal × MarketState → Expectancy. This is the moat.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from datetime import datetime, date
from enum import Enum
from typing import Optional
class SignalType(str, Enum):
"""Known signal types. Each has its own detector and expectancy profile."""
# EMA52 family
EMA52_CROSS_UP = "ema52_cross_up"
EMA52_CROSS_DOWN = "ema52_cross_down"
EMA52_NEAR = "ema52_near"
# Vegas Channel family
VEGAS_LONG = "vegas_long" # price > EMA144, EMA12>EMA50>EMA144, pullback to EMA50
VEGAS_SHORT = "vegas_short" # price < EMA144, EMA12<EMA50<EMA144, rally to EMA50
# Chan Theory
CHAN_1BUY = "chan_1buy" # MACD bottom divergence (price↓ MACD↑)
CHAN_1SELL = "chan_1sell" # MACD top divergence (price↑ MACD↓)
# ORB (Opening Range Breakout — daily level)
ORB_UP = "orb_up" # close breaks above yesterday's high
ORB_DOWN = "orb_down" # close breaks below yesterday's low
# Gap signals
GAP_UP = "gap_up" # open > yesterday's high (breakaway gap)
GAP_DOWN = "gap_down" # open < yesterday's low (breakaway gap)
# NR7 (Narrow Range 7)
NR7 = "nr7" # narrowest range in 7 days (coiling)
# Inside Bar
IB_LONG = "ib_long" # today's range inside yesterday's, close > mid
IB_SHORT = "ib_short" # today's range inside yesterday's, close < mid
@classmethod
def bullish_signals(cls) -> list["SignalType"]:
return [cls.EMA52_CROSS_UP, cls.VEGAS_LONG, cls.CHAN_1BUY, cls.ORB_UP]
@classmethod
def bearish_signals(cls) -> list["SignalType"]:
return [cls.EMA52_CROSS_DOWN, cls.VEGAS_SHORT, cls.CHAN_1SELL, cls.ORB_DOWN]
@dataclass
class SignalInstance:
"""A single signal occurrence with its context and outcome.
Stored in signal_instance table. This is the training data
for the Expectancy Engine — every signal, its market state,
and what happened afterward.
"""
# Identity
signal_type: str # SignalType value
ts_code: str # stock code
trade_date: date # when signal fired
# Signal details
signal_price: float = 0.0 # close price at signal
# Market context at signal time (state vector snapshot)
state_trend: float = 0.0
state_fear: float = 0.0
state_liquidity: float = 0.0
state_rotation: float = 0.0
state_participation: float = 0.0
state_volatility: float = 0.0
state_breadth: float = 0.0
# Outcome (forward-looking)
return_5d: Optional[float] = None # return after 5 trading days
return_10d: Optional[float] = None
return_20d: Optional[float] = None
max_return_5d: Optional[float] = None # max favorable excursion
max_drawdown_5d: Optional[float] = None # max adverse excursion
holding_days: Optional[int] = None # days until signal invalidates
outcome_known: bool = False # True if outcome data is complete
# Metadata
created_at: datetime = field(default_factory=datetime.now)
+184
View File
@@ -0,0 +1,184 @@
"""MarketState — the central domain object of the entire platform.
Every downstream consumer (Dashboard, AI Agent, Backtester, Signal Engine)
uses this same type. NOT a dict.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from datetime import date, datetime
from typing import Optional
@dataclass
class MarketState:
"""Continuous market state vector.
Each dimension is 0.01.0, representing the strength/intensity
of that market property. NOT a discrete classification.
Consumers read the vector directly; the human-readable label
is derived by the presentation layer only.
"""
timestamp: datetime
trade_date: date
version: str = "1.0"
source: str = "ashare_dp"
# ── State dimensions (0.01.0 continuous) ──
trend: float = 0.0 # trend strength (breadth + MA alignment + resonance)
fear: float = 0.0 # fear/greed inverse (drawdown + range + decline breadth)
liquidity: float = 0.0 # liquidity health (volume vs avg + participation)
rotation: float = 0.0 # sector rotation intensity
participation: float = 0.0 # market participation breadth
volatility: float = 0.0 # normalized volatility
breadth: float = 0.0 # advance/decline health
# ── Quality metadata ──
confidence: float = 0.0 # how clearly the vector matches a regime (01)
quality: float = 0.0 # data quality / completeness score (01)
def to_dict(self) -> dict:
"""Serialize to dict for JSON API responses."""
return {
"version": self.version,
"source": self.source,
"timestamp": self.timestamp.isoformat(),
"trade_date": self.trade_date.isoformat(),
"dimensions": {
"trend": self.trend,
"fear": self.fear,
"liquidity": self.liquidity,
"rotation": self.rotation,
"participation": self.participation,
"volatility": self.volatility,
"breadth": self.breadth,
},
"confidence": self.confidence,
"quality": self.quality,
}
def to_vector(self) -> list[float]:
"""Return state as a plain float vector for ML / similarity search."""
return [
self.trend,
self.fear,
self.liquidity,
self.rotation,
self.participation,
self.volatility,
self.breadth,
]
@property
def dimension_names(self) -> list[str]:
return [
"trend", "fear", "liquidity", "rotation",
"participation", "volatility", "breadth",
]
@classmethod
def from_features(
cls,
features: dict,
trade_date: date,
timestamp: Optional[datetime] = None,
) -> "MarketState":
"""Construct MarketState from computed feature vectors.
Each feature category contributes to one or more state dimensions.
"""
import time as _time
ts = timestamp or datetime.now()
# ── Breadth → breadth, participation ──
b = features.get("breadth_vector", {})
advance_ratio = b.get("advance_ratio", 0.5)
new_high_ratio = b.get("new_high_ratio", 0.0)
new_low_ratio = b.get("new_low_ratio", 0.0)
breadth_val = _clamp(advance_ratio, 0, 1)
# ── Volume → liquidity ──
v = features.get("volume_vector", {})
vs_5d = v.get("vs_5d", 1.0)
vs_20d = v.get("vs_20d", 1.0)
total_turnover = v.get("total_turnover", 0)
# Liquidity: volume relative to average, mapped to 0-1
# 1.0x avg → 0.5, 2.0x+ → 1.0, <0.5x → 0.0
liquidity_val = _clamp((min(vs_5d, vs_20d) - 0.5) / 1.5, 0, 1)
# ── Trend → trend ──
t = features.get("trend_vector", {})
above_ma20 = t.get("above_ma20_pct", 0.5)
resonance = t.get("resonance", 0.5)
divergence = t.get("divergence", 0.0)
# Trend: MA alignment + index resonance - divergence penalty
trend_val = _clamp(
above_ma20 * 0.5 + resonance * 0.4 - divergence * 0.3,
0, 1,
)
# ── Fear → fear ──
f = features.get("fear_vector", {})
median_drawdown = abs(f.get("median_drawdown", 0.0))
pct_deep = f.get("pct_deep_drawdown", 0.0)
daily_range = f.get("daily_range_pct", 0.02)
# Fear: drawdown depth + breadth + intraday range
fear_val = _clamp(
median_drawdown * 2.0 + pct_deep * 1.5 + daily_range * 5.0,
0, 1,
)
# ── Rotation → rotation ──
r = features.get("rotation_vector", {})
speed = r.get("speed", 0.0)
persistence = r.get("persistence", 0.5)
rotation_val = _clamp(speed * 0.6 + persistence * 0.4, 0, 1)
# ── Participation → participation ──
p = features.get("participation_vector", {})
pct_above_ma20 = p.get("pct_above_ma20", 0.5)
pct_above_ma60 = p.get("pct_above_ma60", 0.5)
participation_val = _clamp((pct_above_ma20 + pct_above_ma60) / 2, 0, 1)
# ── Volatility ──
vol_val = _clamp(daily_range / 0.06, 0, 1) # 6% daily range → 1.0
# ── Confidence: how internally consistent the vector is ──
# High trend + low fear → high confidence
# High trend + high fear → conflicting, lower confidence
consistency = 1.0 - abs(trend_val - (1.0 - fear_val)) * 0.5
confidence = _clamp(consistency * breadth_val, 0, 1)
# ── Quality: data completeness ──
quality = 1.0 # Phase 1: assume full quality; future: check for gaps
return cls(
timestamp=ts,
trade_date=trade_date,
trend=round(trend_val, 4),
fear=round(fear_val, 4),
liquidity=round(liquidity_val, 4),
rotation=round(rotation_val, 4),
participation=round(participation_val, 4),
volatility=round(vol_val, 4),
breadth=round(breadth_val, 4),
confidence=round(confidence, 4),
quality=round(quality, 4),
)
def _clamp(value: float, lo: float, hi: float) -> float:
"""Clamp value to [lo, hi], with NaN safety."""
import math
if value is None or (isinstance(value, float) and math.isnan(value)):
return (lo + hi) / 2
return max(lo, min(hi, value))
+46
View File
@@ -0,0 +1,46 @@
"""Trade domain — recording actions and outcomes for system learning."""
from __future__ import annotations
from dataclasses import dataclass, field
from datetime import datetime, date
from typing import Optional
@dataclass
class TradeRecord:
"""A single trade from entry to exit. The training data for system learning."""
# Identity
ts_code: str
trade_date: date
# Entry
entry_price: float
entry_time: datetime | None = None
signal_type: str = "" # which signal triggered the entry
position_pct: float = 0.0 # position size as % of portfolio
# Exit
exit_price: float | None = None
exit_time: datetime | None = None
exit_reason: str = "" # "target" | "stop" | "signal" | "time" | "manual"
# Outcome
return_pct: float | None = None # (exit - entry) / entry * 100
holding_days: int | None = None
max_favorable: float | None = None # best price during hold
max_adverse: float | None = None # worst price during hold
# Context (market state at entry)
state_trend: float = 0.0
state_fear: float = 0.0
state_liquidity: float = 0.0
state_breadth: float = 0.0
# Tags
tags: str = "" # comma-separated: "breakout,trend,momentum"
notes: str = ""
# Metadata
created_at: datetime = field(default_factory=datetime.now)
closed: bool = False
+6
View File
@@ -0,0 +1,6 @@
"""EXECUTION INTELLIGENCE — how to trade: playbook, risk, brief assembly."""
from ashare_dp.execution.playbook import build_playbook
from ashare_dp.execution.risk import evaluate_risks
from ashare_dp.execution.brief import build_brief
__all__ = ["build_playbook", "evaluate_risks", "build_brief"]
+190
View File
@@ -0,0 +1,190 @@
"""TradingBrief Builder — assembles TradingContext + serializes for API.
NOT an engine. Pure assembly and serialization — no inference, no rules.
The single place where TradingContext becomes API JSON.
"""
from __future__ import annotations
from datetime import date, datetime
from ashare_dp.domain.context import (
TradingContext, Playbook, Expectancy, Allocation,
)
from ashare_dp.domain.state import MarketState
def build_brief(
trade_date: date,
state: MarketState,
playbook: Playbook,
expectancy: Expectancy | None,
leaders: dict,
opportunities: list,
money_flow,
allocation: Allocation | None,
narrative: str,
timestamp: datetime | None = None,
) -> TradingContext:
"""Assemble TradingContext from all engine outputs. Pure assembly."""
return TradingContext(
timestamp=timestamp or datetime.now(),
trade_date=trade_date,
state=state,
leaders=leaders,
opportunities=opportunities,
money_flow=money_flow,
playbook=playbook,
expectancy=expectancy,
allocation=allocation,
risk_budget=playbook.risk_budget if playbook else "Medium",
narrative=narrative,
)
def generate_narrative(state, leaders, opportunities, flow_graph) -> str:
"""Rule-based narrative. Template from features — not LLM."""
parts = []
t = state.trend
f = state.fear
if t > 0.5:
parts.append("市场处于趋势行情")
elif t > 0.3:
parts.append("市场震荡")
else:
parts.append("市场偏弱")
if flow_graph and flow_graph.flows:
top_flow = flow_graph.flows[0]
parts.append(f"资金从{top_flow.source}流向{top_flow.target}")
if opportunities:
top_opps = [o.name for o in opportunities[:3]]
parts.append(f"热点集中在{''.join(top_opps)}")
alive = [ind for ind, ld in leaders.items()
if isinstance(ld, dict) and ld.get("state") in ("leading", "growing", "birth")]
broken = [ind for ind, ld in leaders.items()
if isinstance(ld, dict) and ld.get("state") in ("dead", "breaking")]
if alive:
parts.append(f"龙头板块{''.join(alive[:3])}保持强势")
if broken:
parts.append(f"{''.join(broken[:3])}龙头走弱需回避")
if f > 0.5:
parts.append("市场恐慌情绪较高,注意风险控制")
elif f < 0.3:
parts.append("风险情绪可控")
return "".join(parts) + ""
def brief_to_api_dict(
brief: TradingContext,
risk_events: list,
features: dict,
sentiment: dict,
all_expectancies: list | None = None,
theme_graph = None,
recommendations: list | None = None,
) -> dict:
"""Serialize TradingContext to the versioned API response.
The SINGLE serialization point. Router calls this, never builds JSON by hand.
Response schema is stable (version 1.0) — consumers depend on it.
"""
pb = brief.playbook
exp = brief.expectancy
alloc = brief.allocation
mf = brief.money_flow
return {
"version": "1.0",
"timestamp": brief.timestamp.isoformat(),
"trade_date": brief.trade_date.isoformat(),
"playbook": {
"suitable_strategies": pb.suitable_strategies,
"unsuitable_strategies": pb.unsuitable_strategies,
"holding_time": pb.holding_time,
"risk_budget": pb.risk_budget,
"position_sizing": pb.position_sizing,
"bias": pb.bias,
"confidence": pb.confidence,
} if pb else {},
"expectancy": {
"signal_type": exp.signal_type,
"win_rate": exp.win_rate,
"avg_return": exp.avg_return,
"max_drawdown": exp.max_drawdown,
"avg_holding_days": exp.avg_holding_days,
"sample_count": exp.sample_count,
"similar_states": exp.similar_states,
"confidence": exp.confidence,
} if exp else {},
"allocation": {
"cash_pct": alloc.cash_pct,
"trend_pct": alloc.trend_pct,
"trial_pct": alloc.trial_pct,
"reasoning": alloc.reasoning,
} if alloc else {},
"opportunities": [
{
"name": o.name,
"score": o.score,
"lifecycle": o.lifecycle,
"persistence_days": o.persistence_days,
"leader_stock": o.leader_stock,
"top_stocks": o.top_stocks,
}
for o in (brief.opportunities or [])[:10]
],
"money_flow": {
"flows": [
{"source": e.source, "target": e.target,
"magnitude": e.magnitude, "strength": e.strength}
for e in mf.flows
] if mf else [],
"net_inflow": mf.net_inflow if mf else {},
"net_outflow": mf.net_outflow if mf else {},
},
"leaders": brief.leaders,
"risks": [e.to_dict() for e in risk_events],
"evidence": {
"state": brief.state.to_dict() if brief.state else {},
"features": features,
"sentiment": sentiment,
},
"theme_graph": {
"themes": [
{
"name": t.name,
"score": t.relevance_score,
"momentum": t.momentum,
"concepts": [
{"name": c.name, "score": c.relevance_score}
for c in (t.concepts or []) if c.relevance_score > 0
],
}
for t in (theme_graph.themes if theme_graph else [])
if t.relevance_score > 0
],
"edges": [
{"source": e.source, "target": e.target, "strength": e.strength}
for e in (theme_graph.edges if theme_graph else [])
],
} if theme_graph else {},
"recommendations": recommendations or [],
"narrative": brief.narrative,
"all_expectancies": [
{
"signal_type": e.signal_type,
"win_rate": e.win_rate,
"avg_return": e.avg_return,
"max_drawdown": e.max_drawdown,
"sample_count": e.sample_count,
"confidence": e.confidence,
}
for e in (all_expectancies or [])
],
}
+112
View File
@@ -0,0 +1,112 @@
"""Playbook Engine — maps MarketState + Leaders + Opportunities to a complete trading playbook.
Not a recommendation list. A complete playbook: what strategies work,
what to avoid, how long to hold, how much to risk.
"""
from __future__ import annotations
from ashare_dp.domain.context import Playbook
from ashare_dp.domain.leadership import LeaderState
from ashare_dp.domain.state import MarketState
def build_playbook(
state: MarketState,
leaders: dict[str, dict],
opportunities: list,
) -> Playbook:
"""Build trading playbook from market state + leaders + opportunities.
Rules:
- High trend + low fear + high participation → Trend Following / Breakout
- Mid trend + high rotation → Sector Rotation
- Low trend + high fear → Defensive / Cash
- Leader state affects sector focus
"""
t = state.trend
f = state.fear
l = state.liquidity
r = state.rotation
p = state.participation
suitable = []
unsuitable = []
# ── Strategy selection ──
if t > 0.55 and f < 0.4 and l > 0.5:
suitable.extend(["Trend Following", "Breakout", "Momentum"])
unsuitable.extend(["Mean Reversion", "Bottom Fishing"])
elif t > 0.4 and r > 0.4:
suitable.extend(["Sector Rotation", "Breakout"])
unsuitable.extend(["Counter Trend"])
elif t < 0.3 and f > 0.5:
suitable.append("Cash")
unsuitable.extend(["Trend Following", "Breakout", "Momentum", "Bottom Fishing"])
else:
suitable.extend(["Range Trading"])
unsuitable.extend(["Trend Following"])
if p < 0.3:
unsuitable.append("Heavy Position")
if l < 0.3:
unsuitable.append("Large Cap Breakout")
# ── Holding time ──
if t > 0.6 and l > 0.6:
holding_time = "3~5 Days"
elif t > 0.4:
holding_time = "2~4 Days"
else:
holding_time = "1~2 Days"
# ── Risk budget ──
if f > 0.6:
risk_budget = "Low"
elif f > 0.35:
risk_budget = "Medium"
else:
risk_budget = "High"
# ── Position sizing ──
if state.confidence > 0.7 and f < 0.3:
position_sizing = "Standard"
elif f > 0.5:
position_sizing = "Reduced"
else:
position_sizing = "Standard"
# ── Focus sectors ──
focus_sectors = []
avoid_sectors = []
for opp in opportunities[:5]:
if opp.score > 60:
focus_sectors.append(opp.name)
for ind, ld in leaders.items():
if isinstance(ld, dict) and ld.get("state") in (
LeaderState.DEAD.value, LeaderState.BREAKING.value
):
avoid_sectors.append(ind)
# ── Bias ──
if t > 0.5 and f < 0.4:
bias = "LONG"
elif f > 0.6:
bias = "CASH"
else:
bias = "NEUTRAL"
return Playbook(
suitable_strategies=list(dict.fromkeys(suitable))[:4],
unsuitable_strategies=list(dict.fromkeys(unsuitable))[:4],
holding_time=holding_time,
risk_budget=risk_budget,
position_sizing=position_sizing,
focus_sectors=focus_sectors[:5],
avoid_sectors=avoid_sectors[:5],
bias=bias,
confidence=round(state.confidence, 4),
)
+218
View File
@@ -0,0 +1,218 @@
"""Risk Rule Engine — pluggable rules that emit RiskEvents.
Each rule evaluates features independently. Adding a new risk
means adding a new RiskRule — no engine code changes.
Dashboard renders, AI explains, Backtest counts.
"""
from __future__ import annotations
from datetime import datetime
from typing import Any, Callable
from ashare_dp.domain.events import RiskEvent
# Rule condition signature:
# (features: dict) -> (triggered: bool, severity: float, payload: dict)
class RiskRule:
"""A single risk detection rule."""
def __init__(
self,
name: str,
event_type: str,
level: str, # "warning" | "danger"
affected: str,
condition: Callable[[dict], tuple[bool, float, dict]],
message_template: str,
):
self.name = name
self.event_type = event_type
self.level = level
self.affected = affected
self.condition = condition
self.message_template = message_template
# ── Rule definitions ──
def _check_drawdown_spread(features: dict) -> tuple[bool, float, dict]:
"""High-position stocks starting to drop broadly."""
fear = features.get("fear_vector", {})
pct_deep = fear.get("pct_deep_drawdown_20d", fear.get("pct_deep_drawdown", 0))
median_dd = abs(fear.get("median_drawdown_20d", fear.get("median_drawdown", 0)))
triggered = pct_deep > 0.30 or median_dd > 0.08
severity = min(max(pct_deep * 2 + median_dd * 5, 0), 1.0)
return triggered, severity, {
"pct_deep_drawdown": pct_deep,
"median_drawdown": median_dd,
}
def _check_volume_drying(features: dict) -> tuple[bool, float, dict]:
"""Volume drying up significantly vs. recent average."""
vol = features.get("volume_vector", {})
vs_5d = vol.get("vs_5d", 1.0)
vs_20d = vol.get("vs_20d", 1.0)
triggered = vs_5d < 0.80 or vs_20d < 0.75
severity = min(max((1.0 - min(vs_5d, vs_20d)) * 2, 0), 1.0)
return triggered, severity, {
"vs_5d": vs_5d,
"vs_20d": vs_20d,
}
def _check_breadth_collapse(features: dict) -> tuple[bool, float, dict]:
"""Market breadth collapsing — very few stocks advancing."""
breadth = features.get("breadth_vector", {})
advance_ratio = breadth.get("advance_ratio", 0.5)
down_prev = breadth.get("down_from_prev", 0)
up_prev = breadth.get("up_from_prev", 0)
total = breadth.get("advances", 0) + breadth.get("declines", 0) + breadth.get("unchanged", 0)
triggered = advance_ratio < 0.25
severity = min(max((0.5 - advance_ratio) * 2, 0), 1.0)
down_pct = round(down_prev / total * 100, 1) if total > 0 else 0
return triggered, severity, {
"advance_ratio": advance_ratio,
"down_pct": down_pct,
}
def _check_high_volatility(features: dict) -> tuple[bool, float, dict]:
"""Market volatility spiking — unusual daily ranges."""
fear = features.get("fear_vector", {})
avg_range = fear.get("daily_range_pct", fear.get("avg_daily_range", 0.02))
triggered = avg_range > 0.05
severity = min(max((avg_range - 0.03) / 0.05, 0), 1.0)
return triggered, severity, {
"avg_daily_range": avg_range,
}
def _check_divergence(features: dict) -> tuple[bool, float, dict]:
"""Indices diverging — some up, some down strongly."""
trend = features.get("trend_vector", {})
divergence = trend.get("divergence", 0.0)
triggered = divergence > 0.7
severity = min(divergence, 1.0)
return triggered, severity, {
"divergence": divergence,
}
RISK_RULES: list[RiskRule] = [
RiskRule(
name="DrawdownSpread",
event_type="DrawdownSpread",
level="danger",
affected="All",
condition=_check_drawdown_spread,
message_template="高位股开始补跌:{pct_deep}% 的股票从高点回撤超 10%",
),
RiskRule(
name="BreadthCollapse",
event_type="BreadthCollapse",
level="danger",
affected="All",
condition=_check_breadth_collapse,
message_template="上涨家数仅 {advance_pct}%{down_pct}% 的股票下跌",
),
RiskRule(
name="VolumeDrying",
event_type="VolumeDrying",
level="warning",
affected="All",
condition=_check_volume_drying,
message_template="成交额较5日均值下降 {vol_change}%",
),
RiskRule(
name="HighVolatility",
event_type="HighVolatility",
level="warning",
affected="All",
condition=_check_high_volatility,
message_template="市场波动加剧,日均振幅达 {range_pct}%",
),
RiskRule(
name="Divergence",
event_type="Divergence",
level="warning",
affected="All",
condition=_check_divergence,
message_template="指数走势分化,共振度低",
),
]
def evaluate_risks(
features: dict[str, dict[str, Any]],
timestamp: datetime | None = None,
) -> list[RiskEvent]:
"""Evaluate all risk rules against current features.
Returns triggered RiskEvents sorted by severity descending.
"""
if timestamp is None:
timestamp = datetime.now()
events: list[RiskEvent] = []
for rule in RISK_RULES:
try:
triggered, severity, payload = rule.condition(features)
except Exception:
continue # Skip rules that fail to evaluate
if not triggered:
continue
# Build human-readable message from template + payload
message = rule.message_template
# Build a formatting context with common derived values
fmt_ctx = {}
for k, v in payload.items():
if isinstance(v, float):
fmt_ctx[k] = round(v, 3)
else:
fmt_ctx[k] = v
# Add derived values for common templates
if "pct_deep_drawdown" in payload:
fmt_ctx["pct_deep"] = round(payload["pct_deep_drawdown"] * 100, 1)
if "pct_deep_drawdown_20d" in payload:
fmt_ctx["pct_deep"] = round(payload["pct_deep_drawdown_20d"] * 100, 1)
if "advance_ratio" in payload:
fmt_ctx["advance_pct"] = round(payload["advance_ratio"] * 100, 1)
if "vs_5d" in payload:
fmt_ctx["vol_change"] = round((1 - payload["vs_5d"]) * 100, 1)
if "avg_daily_range" in payload:
fmt_ctx["range_pct"] = round(payload["avg_daily_range"] * 100, 1)
try:
message = message.format(**fmt_ctx)
except (KeyError, ValueError):
pass # Keep template as-is if formatting fails
events.append(RiskEvent(
type=rule.event_type,
severity=round(severity, 4),
level=rule.level,
affected=rule.affected,
confidence=round(severity, 4), # Phase 1: severity ≈ confidence
message=message,
payload=payload,
timestamp=timestamp,
))
# Sort: danger first, then by severity desc
events.sort(key=lambda e: (0 if e.level == "danger" else 1, -e.severity))
return events
+21
View File
@@ -0,0 +1,21 @@
"""Feature Store — centralized market feature computation.
All feature modules register themselves with the FeatureRegistry on import.
Engine code imports only the registry, never individual feature functions.
Usage:
from ashare_dp.features import registry
features = registry.compute_all(db, trade_date)
"""
from ashare_dp.features.registry import registry
# Import feature modules to trigger registration
from ashare_dp.features import breadth # noqa: F401
from ashare_dp.features import volume # noqa: F401
from ashare_dp.features import trend # noqa: F401
from ashare_dp.features import fear # noqa: F401
from ashare_dp.features import rotation # noqa: F401
from ashare_dp.features import participation # noqa: F401
__all__ = ["registry"]
+151
View File
@@ -0,0 +1,151 @@
"""Breadth features: advance/decline, new highs/lows, up/down volume ratio."""
from __future__ import annotations
from datetime import date
from typing import Any
import duckdb
from ashare_dp.domain.features import FeatureDefinition
from ashare_dp.features.registry import registry
from ashare_dp.data.store.database import analytics_conn, kline_glob
def _compute_breadth_vector(db: Any, trade_date: date, _results: dict) -> dict[str, Any]:
"""Compute market breadth metrics for a single trading day.
Uses DuckDB to aggregate over ALL stocks' daily K-line in one query.
"""
parquet_glob = kline_glob()
conn = analytics_conn()
try:
# Single query: advance/decline, new highs/lows, up/down volume
sql = f"""
WITH latest AS (
SELECT *, ROW_NUMBER() OVER (
PARTITION BY ts_code ORDER BY trade_time DESC
) as rn
FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true)
WHERE trade_date <= $trade_date
),
today AS (
SELECT * FROM latest WHERE rn = 1
),
prev AS (
SELECT * FROM latest WHERE rn = 2
),
with_prev AS (
SELECT
t.ts_code,
t.close AS close_today,
t.open AS open_today,
t.volume AS vol_today,
t.amount AS amt_today,
t.high AS high_today,
t.low AS low_today,
p.close AS close_prev,
-- 20-day and 60-day rolling high
MAX(t.close) OVER (
PARTITION BY t.ts_code
ORDER BY t.trade_date
ROWS BETWEEN 19 PRECEDING AND CURRENT ROW
) AS high_20d,
MAX(t.close) OVER (
PARTITION BY t.ts_code
ORDER BY t.trade_date
ROWS BETWEEN 59 PRECEDING AND CURRENT ROW
) AS high_60d,
MIN(t.close) OVER (
PARTITION BY t.ts_code
ORDER BY t.trade_date
ROWS BETWEEN 19 PRECEDING AND CURRENT ROW
) AS low_20d,
MIN(t.close) OVER (
PARTITION BY t.ts_code
ORDER BY t.trade_date
ROWS BETWEEN 59 PRECEDING AND CURRENT ROW
) AS low_60d
FROM today t
LEFT JOIN prev p ON t.ts_code = p.ts_code
)
SELECT
COUNT(*) AS total,
COUNT(*) FILTER (WHERE close_today > open_today) AS advances,
COUNT(*) FILTER (WHERE close_today < open_today) AS declines,
COUNT(*) FILTER (WHERE close_today = open_today) AS unchanged,
COUNT(*) FILTER (WHERE close_today > close_prev AND close_prev IS NOT NULL) AS up_from_prev,
COUNT(*) FILTER (WHERE close_today < close_prev AND close_prev IS NOT NULL) AS down_from_prev,
COUNT(*) FILTER (WHERE close_today >= high_20d * 0.98) AS near_20d_high,
COUNT(*) FILTER (WHERE close_today <= low_20d * 1.02) AS near_20d_low,
COUNT(*) FILTER (WHERE close_today >= high_60d * 0.98) AS near_60d_high,
COUNT(*) FILTER (WHERE close_today <= low_60d * 1.02) AS near_60d_low,
SUM(vol_today) AS total_volume,
SUM(amt_today) AS total_amount,
SUM(CASE WHEN close_today > open_today THEN vol_today ELSE 0 END) AS up_volume,
SUM(CASE WHEN close_today < open_today THEN vol_today ELSE 0 END) AS down_volume,
AVG((close_today - open_today) / NULLIF(open_today, 0) * 100) AS avg_pct_chg,
AVG((high_today - low_today) / NULLIF(close_today, 0)) AS avg_daily_range
FROM with_prev
"""
result = conn.execute(sql, {"trade_date": trade_date.isoformat()}).fetchone()
finally:
conn.close()
if result is None or result[0] == 0:
return {
"advance_ratio": 0.5,
"advances": 0,
"declines": 0,
"unchanged": 0,
"up_from_prev": 0,
"down_from_prev": 0,
"new_high_ratio": 0.0,
"new_low_ratio": 0.0,
"up_vol_ratio": 0.5,
"total_volume": 0,
"total_amount": 0,
"avg_pct_chg": 0.0,
"avg_daily_range": 0.02,
}
total = result[0]
advances = result[1]
declines = result[2]
unchanged = result[3]
up_prev = result[4]
down_prev = result[5]
near_20h = result[6]
near_20l = result[7]
near_60h = result[8]
near_60l = result[9]
total_vol = result[10] or 0
total_amt = result[11] or 0
up_vol = result[12] or 0
down_vol = result[13] or 0
return {
"advance_ratio": round(advances / total, 4) if total else 0.5,
"advances": advances,
"declines": declines,
"unchanged": unchanged,
"up_from_prev": up_prev,
"down_from_prev": down_prev,
"new_high_ratio": round(max(near_20h, near_60h) / total, 4) if total else 0.0,
"new_low_ratio": round(max(near_20l, near_60l) / total, 4) if total else 0.0,
"up_vol_ratio": round(up_vol / (up_vol + down_vol), 4) if (up_vol + down_vol) > 0 else 0.5,
"total_volume": int(total_vol),
"total_amount": float(total_amt),
"avg_pct_chg": round(float(result[14] or 0), 4),
"avg_daily_range": round(float(result[15] or 0.02), 4),
}
registry.register(FeatureDefinition(
name="breadth_vector",
category="breadth",
description="Advance/decline ratio, new high/low ratios, up/down volume ratio, total turnover",
dependencies=[],
compute=_compute_breadth_vector,
))
+137
View File
@@ -0,0 +1,137 @@
"""Fear features: drawdown depth, breadth of deep drawdowns, intraday range."""
from __future__ import annotations
from datetime import date
from typing import Any
import duckdb
from ashare_dp.domain.features import FeatureDefinition
from ashare_dp.features.registry import registry
from ashare_dp.data.store.database import analytics_conn, kline_glob
def _compute_fear_vector(db: Any, trade_date: date, _results: dict) -> dict[str, Any]:
"""Compute fear/greed metrics: drawdown distribution, intraday volatility.
Uses a single DuckDB window query to compute drawdown from 20d and 60d
highs for all stocks, then aggregates the distribution.
"""
parquet_glob = kline_glob()
conn = analytics_conn()
try:
sql = f"""
WITH ranked AS (
SELECT *,
ROW_NUMBER() OVER (PARTITION BY ts_code ORDER BY trade_time DESC) as rn_desc,
ROW_NUMBER() OVER (PARTITION BY ts_code ORDER BY trade_time ASC) as rn_asc
FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true)
WHERE trade_date >= $start_date
),
latest AS (
SELECT * FROM ranked WHERE rn_desc = 1
),
-- 20-day high per stock
high20 AS (
SELECT ts_code, MAX(high) as h20
FROM ranked
WHERE rn_desc <= 20
GROUP BY ts_code
),
-- 60-day high per stock
high60 AS (
SELECT ts_code, MAX(high) as h60
FROM ranked
WHERE rn_desc <= 60
GROUP BY ts_code
),
-- Join with latest
combined AS (
SELECT
l.ts_code, l.close, l.high, l.low, l.open,
h20.h20, h60.h60,
(l.high - l.low) / NULLIF(l.close, 0) AS daily_range
FROM latest l
JOIN high20 h20 ON l.ts_code = h20.ts_code
JOIN high60 h60 ON l.ts_code = h60.ts_code
),
drawdowns AS (
SELECT *,
(close - h20) / NULLIF(h20, 0) AS dd_20d,
(close - h60) / NULLIF(h60, 0) AS dd_60d
FROM combined
)
SELECT
PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY dd_20d) AS dd20_p25,
PERCENTILE_CONT(0.50) WITHIN GROUP (ORDER BY dd_20d) AS dd20_p50,
PERCENTILE_CONT(0.75) WITHIN GROUP (ORDER BY dd_20d) AS dd20_p75,
PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY dd_60d) AS dd60_p25,
PERCENTILE_CONT(0.50) WITHIN GROUP (ORDER BY dd_60d) AS dd60_p50,
PERCENTILE_CONT(0.75) WITHIN GROUP (ORDER BY dd_60d) AS dd60_p75,
COUNT(*) FILTER (WHERE dd_20d < -0.10) AS deep_dd_20_count,
COUNT(*) FILTER (WHERE dd_60d < -0.20) AS deep_dd_60_count,
COUNT(*) AS total,
PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY daily_range) AS range_p25,
PERCENTILE_CONT(0.50) WITHIN GROUP (ORDER BY daily_range) AS range_p50,
PERCENTILE_CONT(0.75) WITHIN GROUP (ORDER BY daily_range) AS range_p75,
AVG(daily_range) AS avg_daily_range
FROM drawdowns
WHERE h20 IS NOT NULL
"""
# Look back ~60 trading days ≈ 90 calendar days
start_date = trade_date.strftime("%Y-%m-%d")
# Use a window that covers at least 60 trading days
result = conn.execute(
sql.replace("$start_date", f"'{_date_sub_calendar(trade_date, 95)}'")
).fetchone()
finally:
conn.close()
if result is None or result[8] == 0:
return _empty_fear()
total = int(result[8])
return {
"median_drawdown_20d": round(float(result[1] or 0), 4),
"median_drawdown_60d": round(float(result[4] or 0), 4),
"pct_deep_drawdown_20d": round(int(result[6]) / total, 4),
"pct_deep_drawdown_60d": round(int(result[7]) / total, 4),
# Legacy keys for backward compat with MarketState.from_features()
"median_drawdown": round(float(result[1] or 0), 4),
"pct_deep_drawdown": round(int(result[6]) / total, 4),
"daily_range_pct": round(float(result[12] or 0.02), 4),
"daily_range_p50": round(float(result[10] or 0.02), 4),
"total_stocks": total,
}
def _date_sub_calendar(trade_date: date, days: int) -> str:
"""Approximate calendar date N days before trade_date."""
from datetime import timedelta
return (trade_date - timedelta(days=days)).strftime("%Y-%m-%d")
def _empty_fear() -> dict:
return {
"median_drawdown_20d": 0.0,
"median_drawdown_60d": 0.0,
"pct_deep_drawdown_20d": 0.0,
"pct_deep_drawdown_60d": 0.0,
"median_drawdown": 0.0,
"pct_deep_drawdown": 0.0,
"daily_range_pct": 0.02,
"daily_range_p50": 0.02,
"total_stocks": 0,
}
registry.register(FeatureDefinition(
name="fear_vector",
category="fear",
description="Drawdown distribution (20d/60d), deep drawdown breadth, intraday range",
dependencies=[],
compute=_compute_fear_vector,
))
+93
View File
@@ -0,0 +1,93 @@
"""Participation features: breadth of market participation."""
from __future__ import annotations
from datetime import date
from typing import Any
import duckdb
from ashare_dp.domain.features import FeatureDefinition
from ashare_dp.features.registry import registry
from ashare_dp.data.store.database import analytics_conn, kline_glob
def _compute_participation_vector(db: Any, trade_date: date, _results: dict) -> dict[str, Any]:
"""Compute market participation: % stocks above key MAs, volume participation rate."""
parquet_glob = kline_glob()
conn = analytics_conn()
try:
sql = f"""
WITH ranked AS (
SELECT *,
ROW_NUMBER() OVER (PARTITION BY ts_code ORDER BY trade_time DESC) as rn_desc
FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true)
WHERE trade_date <= $trade_date
),
with_ma AS (
SELECT
ts_code,
close,
volume,
AVG(close) OVER (
PARTITION BY ts_code ORDER BY trade_time
ROWS BETWEEN 19 PRECEDING AND CURRENT ROW
) AS ma20,
AVG(close) OVER (
PARTITION BY ts_code ORDER BY trade_time
ROWS BETWEEN 59 PRECEDING AND CURRENT ROW
) AS ma60,
AVG(volume) OVER (
PARTITION BY ts_code ORDER BY trade_time
ROWS BETWEEN 19 PRECEDING AND CURRENT ROW
) AS vol_ma20,
rn_desc
FROM ranked
),
latest AS (
SELECT * FROM with_ma WHERE rn_desc = 1
)
SELECT
COUNT(*) AS total,
COUNT(*) FILTER (WHERE close > ma20) AS above_ma20,
COUNT(*) FILTER (WHERE close > ma60) AS above_ma60,
COUNT(*) FILTER (WHERE volume > vol_ma20 AND vol_ma20 > 0) AS above_vol_ma20,
AVG(CASE WHEN vol_ma20 > 0 THEN volume / vol_ma20 ELSE NULL END) AS avg_vol_ratio
FROM latest
"""
result = conn.execute(sql, {"trade_date": trade_date.isoformat()}).fetchone()
finally:
conn.close()
if result is None or result[0] == 0:
return _empty_participation()
total = int(result[0])
return {
"pct_above_ma20": round(int(result[1]) / total, 4),
"pct_above_ma60": round(int(result[2]) / total, 4),
"pct_above_vol_ma20": round(int(result[3]) / total, 4),
"avg_vol_ratio": round(float(result[4] or 1.0), 4),
"total_stocks": total,
}
def _empty_participation() -> dict:
return {
"pct_above_ma20": 0.5,
"pct_above_ma60": 0.5,
"pct_above_vol_ma20": 0.5,
"avg_vol_ratio": 1.0,
"total_stocks": 0,
}
registry.register(FeatureDefinition(
name="participation_vector",
category="participation",
description="% stocks above MA20/MA60, volume participation rate",
dependencies=[],
compute=_compute_participation_vector,
))
+128
View File
@@ -0,0 +1,128 @@
"""Feature Registry — centralized management of all market features.
Engine code only knows about the registry; individual feature functions
are looked up by name. Adding a new feature = register it, no engine changes.
"""
from __future__ import annotations
from datetime import date
from typing import Any, Callable
from ashare_dp.domain.features import FeatureDefinition
class FeatureRegistry:
"""Central registry for all market features.
Usage:
from ashare_dp.features.registry import registry
# Register a feature
registry.register(FeatureDefinition(
name="breadth_vector",
category="breadth",
description="Advance/decline ratio and related breadth metrics",
dependencies=[],
compute=_compute_breadth_vector,
))
# Compute all features for a date
features = registry.compute_all(db, trade_date)
# Get features by category
breadth_features = registry.get_by_category("breadth")
"""
def __init__(self):
self._features: dict[str, FeatureDefinition] = {}
def register(self, feature: FeatureDefinition) -> None:
"""Register a feature definition. Overwrites if name exists."""
self._features[feature.name] = feature
def get(self, name: str) -> FeatureDefinition | None:
"""Get a feature definition by name."""
return self._features.get(name)
def get_by_category(self, category: str) -> list[FeatureDefinition]:
"""Get all features in a category."""
return [f for f in self._features.values() if f.category == category]
def compute_all(self, db: Any, trade_date: date) -> dict[str, dict[str, Any]]:
"""Compute all registered features for a given trading date.
Args:
db: Database instance (read_only DuckDB connection).
trade_date: The trading date to compute features for.
Returns:
Dict mapping feature name → feature value dict.
Features are computed in dependency order (simple topological sort).
"""
results: dict[str, dict[str, Any]] = {}
computed: set[str] = set()
pending: set[str] = set(self._features.keys())
while pending:
ready = [
name for name in pending
if all(dep in computed for dep in self._features[name].dependencies)
]
if not ready:
# Circular dependency or missing dependency
remaining = ", ".join(sorted(pending))
raise RuntimeError(
f"Cannot resolve feature dependencies. "
f"Remaining: {remaining}. Computed: {computed}"
)
for name in ready:
feature = self._features[name]
if feature.compute is not None:
results[name] = feature.compute(db, trade_date, results)
computed.add(name)
pending.discard(name)
return results
def compute_category(
self, db: Any, trade_date: date, category: str
) -> dict[str, dict[str, Any]]:
"""Compute all features in a specific category."""
features = self.get_by_category(category)
# Compute dependencies first
all_needed: set[str] = set()
for f in features:
all_needed.add(f.name)
all_needed.update(f.dependencies)
full_results = self.compute_all(db, trade_date)
return {k: v for k, v in full_results.items() if k in all_needed}
def list_categories(self) -> list[str]:
"""List all unique feature categories."""
return sorted({f.category for f in self._features.values()})
def list_features(self) -> list[dict]:
"""List all registered features with metadata."""
return [
{
"name": f.name,
"category": f.category,
"description": f.description,
"dependencies": f.dependencies,
"version": f.version,
}
for f in sorted(self._features.values(), key=lambda x: x.name)
]
def __len__(self) -> int:
return len(self._features)
def __contains__(self, name: str) -> bool:
return name in self._features
# Module-level singleton
registry = FeatureRegistry()
+316
View File
@@ -0,0 +1,316 @@
"""Rotation features: sector rotation speed, persistence, concentration.
Also computes the RotationGraph (nodes + edges) from industry turnover changes.
"""
from __future__ import annotations
from datetime import date
from typing import Any
import duckdb
from ashare_dp.domain.events import RotationEdge, RotationGraph
from ashare_dp.domain.features import FeatureDefinition
from ashare_dp.features.registry import registry
from ashare_dp.data.store.database import analytics_conn, kline_glob
def _compute_rotation_vector(db: Any, trade_date: date, _results: dict) -> dict[str, Any]:
"""Compute rotation intensity metrics.
Measures how fast money is rotating between industries and how
persistent the current leaders are.
"""
# Use the compute_rotation_graph result for rotation metrics
rg = compute_rotation_graph_from_db(db, trade_date, lookback=20)
momentum = compute_industry_momentum_from_db(db, trade_date)
if not rg.edges:
return _empty_rotation()
# Speed: average edge strength (higher = faster rotation)
speed = sum(e.strength for e in rg.edges) / len(rg.edges) if rg.edges else 0.0
# Persistence: fraction of top-5 industries today that were also top-5 5 days ago
# (simplified: use the strongest edge's persistence_days)
persistence = max((e.persistence_days / 10 for e in rg.edges), default=0.0)
persistence = min(persistence, 1.0)
# Concentration: how concentrated are flows in the top 3 edges?
if len(rg.edges) >= 3:
top3 = sum(e.strength for e in rg.edges[:3])
total = sum(e.strength for e in rg.edges)
concentration = top3 / total if total > 0 else 0.5
else:
concentration = 0.5
return {
"speed": round(speed, 4),
"persistence": round(persistence, 4),
"concentration": round(concentration, 4),
"edge_count": len(rg.edges),
"node_count": len(rg.nodes),
}
def compute_rotation_graph_from_db(
db: Any, trade_date: date, lookback: int = 20
) -> RotationGraph:
"""Build rotation graph from industry-level turnover changes.
Uses stock_industry table joined with daily K-line to compute
per-industry turnover trends, then generates directed edges
where money flows from declining industries to rising ones.
"""
parquet_glob = kline_glob()
conn = analytics_conn()
try:
# Get industries and their recent turnover trends
sql = f"""
WITH daily AS (
SELECT
k.ts_code AS ts_code,
k.trade_date,
k.amount,
si.industry_name
FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true) k
JOIN stock_industry si ON k.ts_code = si.ts_code
WHERE k.trade_date >= $start_date AND k.trade_date <= $end_date
),
industry_daily AS (
SELECT
industry_name,
trade_date,
SUM(amount) AS total_amount,
COUNT(*) AS stock_count
FROM daily
GROUP BY industry_name, trade_date
),
-- Recent (last 5 trading days) vs prior (5 trading days before that)
ranked AS (
SELECT *,
ROW_NUMBER() OVER (PARTITION BY industry_name ORDER BY trade_date DESC) as rn
FROM industry_daily
),
recent AS (
SELECT industry_name, SUM(total_amount) as recent_amount,
SUM(stock_count) as recent_count
FROM ranked WHERE rn <= 5
GROUP BY industry_name
),
prior AS (
SELECT industry_name, SUM(total_amount) as prior_amount
FROM ranked WHERE rn > 5 AND rn <= 10
GROUP BY industry_name
),
changes AS (
SELECT
COALESCE(r.industry_name, p.industry_name) AS industry_name,
COALESCE(r.recent_amount, 0) AS recent_amount,
COALESCE(p.prior_amount, 0) AS prior_amount,
CASE
WHEN COALESCE(p.prior_amount, 0) > 0
THEN (COALESCE(r.recent_amount, 0) - p.prior_amount) / p.prior_amount
ELSE 0
END AS turnover_change,
COALESCE(r.recent_count, 0) AS stock_count
FROM recent r
FULL OUTER JOIN prior p ON r.industry_name = p.industry_name
WHERE COALESCE(r.recent_amount, 0) + COALESCE(p.prior_amount, 0) > 0
)
SELECT
industry_name,
recent_amount,
prior_amount,
turnover_change,
stock_count
FROM changes
ORDER BY turnover_change DESC
"""
start_str = (trade_date - __timedelta(lookback + 10)).strftime("%Y-%m-%d")
end_str = trade_date.strftime("%Y-%m-%d")
result = conn.execute(
sql.replace("$start_date", f"'{start_str}'").replace("$end_date", f"'{end_str}'")
).fetchdf()
finally:
conn.close()
if result.empty:
return RotationGraph(nodes=[], edges=[])
industries = result["industry_name"].tolist()
changes = result.set_index("industry_name")["turnover_change"].to_dict()
# Generate edges: from declining industries to rising ones
edges = []
gainers = sorted(
[(k, v) for k, v in changes.items() if v > 0.05],
key=lambda x: x[1], reverse=True,
)
losers = sorted(
[(k, v) for k, v in changes.items() if v < -0.05],
key=lambda x: x[1],
)
# Create edges from top losers to top gainers
max_edges = min(len(gainers), len(losers), 5)
for i in range(max_edges):
los_name, los_change = losers[i]
win_name, win_change = gainers[i]
strength = min((win_change - los_change) / 2.0, 1.0)
edges.append(RotationEdge(
source=los_name,
target=win_name,
strength=round(strength, 4),
confidence=round(min(abs(win_change), abs(los_change)) * 2, 4),
source_change=round(los_change, 4),
target_change=round(win_change, 4),
persistence_days=1, # Phase 1: simplified
))
return RotationGraph(
nodes=sorted(industries),
edges=sorted(edges, key=lambda e: e.strength, reverse=True),
)
def compute_industry_momentum_from_db(
db: Any, trade_date: date, windows: list[int] | None = None
) -> list[dict]:
"""Compute per-industry momentum across multiple time windows.
Returns list of dicts with turnover change % for 5d, 10d, 20d windows.
"""
if windows is None:
windows = [5, 10, 20]
parquet_glob = kline_glob()
conn = analytics_conn()
try:
code_list = "', '".join(
f"'{c}'" for c in ["主板", "创业板", "科创板", "北交所"]
)
# Phase 1: use derived market as industry proxy if stock_industry is empty
# Check if stock_industry has data
check_sql = "SELECT COUNT(*) FROM stock_industry"
has_industry = conn.execute(check_sql).fetchone()[0] > 0
if has_industry:
join_clause = "JOIN stock_industry si ON k.ts_code = si.ts_code"
group_col = "si.industry_name"
else:
# Fallback: derive market from ts_code
join_clause = ""
group_col = """
CASE
WHEN k.ts_code LIKE '6%SH' OR k.ts_code LIKE '0%SZ' THEN '主板'
WHEN k.ts_code LIKE '3%SZ' THEN '创业板'
WHEN k.ts_code LIKE '688%SH' THEN '科创板'
WHEN k.ts_code LIKE '4%BJ' OR k.ts_code LIKE '8%BJ' OR k.ts_code LIKE '92%BJ' THEN '北交所'
ELSE '其他'
END
"""
sql = f"""
WITH daily AS (
SELECT
k.ts_code AS ts_code,
k.trade_date,
k.amount,
{group_col} AS segment
FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true) k
{join_clause}
WHERE k.trade_date >= $start_date AND k.trade_date <= $end_date
),
segment_daily AS (
SELECT
segment,
trade_date,
SUM(amount) AS total_amount,
COUNT(DISTINCT ts_code) AS stock_count
FROM daily
GROUP BY segment, trade_date
)
SELECT
segment,
MAX(trade_date) AS latest_date,
AVG(CASE WHEN trade_date >= $recent_5 THEN total_amount END) AS avg_5d,
AVG(CASE WHEN trade_date < $recent_5 AND trade_date >= $prior_5 THEN total_amount END) AS avg_prior_5d,
AVG(CASE WHEN trade_date >= $recent_10 THEN total_amount END) AS avg_10d,
AVG(CASE WHEN trade_date < $recent_10 AND trade_date >= $prior_10 THEN total_amount END) AS avg_prior_10d,
MAX(stock_count) AS stock_count
FROM segment_daily
GROUP BY segment
ORDER BY avg_5d DESC
"""
end_str = trade_date.strftime("%Y-%m-%d")
# Approximate trading day offsets
from datetime import timedelta
recent_5 = (trade_date - timedelta(days=7)).strftime("%Y-%m-%d")
prior_5 = (trade_date - timedelta(days=14)).strftime("%Y-%m-%d")
recent_10 = (trade_date - timedelta(days=14)).strftime("%Y-%m-%d")
prior_10 = (trade_date - timedelta(days=28)).strftime("%Y-%m-%d")
result = conn.execute(
sql
.replace("$start_date", f"'{prior_10}'")
.replace("$end_date", f"'{end_str}'")
.replace("$recent_5", f"'{recent_5}'")
.replace("$prior_5", f"'{prior_5}'")
.replace("$recent_10", f"'{recent_10}'")
.replace("$prior_10", f"'{prior_10}'")
).fetchdf()
finally:
conn.close()
industries = []
for _, row in result.iterrows():
avg_5 = float(row["avg_5d"] or 0)
avg_p5 = float(row["avg_prior_5d"] or 0)
avg_10 = float(row["avg_10d"] or 0)
avg_p10 = float(row["avg_prior_10d"] or 0)
chg_5d = round((avg_5 - avg_p5) / avg_p5, 4) if avg_p5 > 0 else 0.0
chg_10d = round((avg_10 - avg_p10) / avg_p10, 4) if avg_p10 > 0 else 0.0
industries.append({
"segment": row["segment"],
"stock_count": int(row["stock_count"]),
"turnover_change_5d": chg_5d,
"turnover_change_10d": chg_10d,
})
return industries
def _empty_rotation() -> dict:
return {
"speed": 0.0,
"persistence": 0.5,
"concentration": 0.5,
"edge_count": 0,
"node_count": 0,
}
def __timedelta(days: int):
from datetime import timedelta
return timedelta(days=days)
# Exported for use by router
compute_rotation_graph = compute_rotation_graph_from_db
compute_industry_momentum = compute_industry_momentum_from_db
registry.register(FeatureDefinition(
name="rotation_vector",
category="rotation",
description="Sector rotation speed, persistence, concentration",
dependencies=[],
compute=_compute_rotation_vector,
))
+179
View File
@@ -0,0 +1,179 @@
"""Trend features: index MA positioning, resonance/divergence, trend strength."""
from __future__ import annotations
from datetime import date
from typing import Any
import duckdb
from ashare_dp.core.models import INDEX_CODES
from ashare_dp.domain.features import FeatureDefinition
from ashare_dp.features.registry import registry
from ashare_dp.data.store.database import analytics_conn, kline_glob
# Sina symbol format for indices used in Parquet ts_code
# The index codes stored in Parquet use format like "000001.SH"
INDEX_TS_CODES = list(INDEX_CODES.keys())
def _compute_trend_vector(db: Any, trade_date: date, _results: dict) -> dict[str, Any]:
"""Compute trend metrics from index daily K-line data.
Analyzes 7 major indices for MA positioning, slope, and mutual alignment.
"""
parquet_glob = kline_glob()
conn = analytics_conn()
code_list = "', '".join(INDEX_TS_CODES)
try:
sql = f"""
WITH idx_data AS (
SELECT
ts_code,
trade_date,
trade_time,
close,
volume,
AVG(close) OVER (
PARTITION BY ts_code ORDER BY trade_time
ROWS BETWEEN 4 PRECEDING AND CURRENT ROW
) AS ma5,
AVG(close) OVER (
PARTITION BY ts_code ORDER BY trade_time
ROWS BETWEEN 9 PRECEDING AND CURRENT ROW
) AS ma10,
AVG(close) OVER (
PARTITION BY ts_code ORDER BY trade_time
ROWS BETWEEN 19 PRECEDING AND CURRENT ROW
) AS ma20,
AVG(close) OVER (
PARTITION BY ts_code ORDER BY trade_time
ROWS BETWEEN 59 PRECEDING AND CURRENT ROW
) AS ma60,
AVG(volume) OVER (
PARTITION BY ts_code ORDER BY trade_time
ROWS BETWEEN 19 PRECEDING AND CURRENT ROW
) AS vol_ma20,
ROW_NUMBER() OVER (
PARTITION BY ts_code ORDER BY trade_time DESC
) as rn
FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true)
WHERE ts_code IN ('{code_list}')
),
latest AS (
SELECT * FROM idx_data WHERE rn = 1
),
prev AS (
SELECT * FROM idx_data WHERE rn = 2
)
SELECT
l.ts_code,
l.close,
l.ma5, l.ma10, l.ma20, l.ma60,
l.volume,
l.vol_ma20,
p.close AS prev_close,
-- price vs MAs
CASE WHEN l.close > l.ma5 THEN 1 ELSE 0 END AS above_ma5,
CASE WHEN l.close > l.ma10 THEN 1 ELSE 0 END AS above_ma10,
CASE WHEN l.close > l.ma20 THEN 1 ELSE 0 END AS above_ma20,
CASE WHEN l.close > l.ma60 THEN 1 ELSE 0 END AS above_ma60,
-- MA alignment (bullish = ma5 > ma10 > ma20 > ma60)
CASE WHEN l.ma5 > l.ma10 THEN 1 ELSE 0 END AS ma5_gt_ma10,
CASE WHEN l.ma10 > l.ma20 THEN 1 ELSE 0 END AS ma10_gt_ma20,
CASE WHEN l.ma20 > l.ma60 THEN 1 ELSE 0 END AS ma20_gt_ma60,
-- slope (5d change / 5)
(l.ma5 - p.ma5) / NULLIF(p.ma5, 0) AS ma5_slope,
(l.close - p.close) / NULLIF(p.close, 0) AS pct_chg,
-- volume ratio
l.volume / NULLIF(l.vol_ma20, 0) AS vol_ratio
FROM latest l
LEFT JOIN prev p ON l.ts_code = p.ts_code
"""
result = conn.execute(sql).fetchdf()
finally:
conn.close()
if result.empty:
return _empty_trend()
# ── Per-index stats ──
above_ma20_count = int(result["above_ma20"].sum())
above_ma60_count = int(result["above_ma60"].sum())
total = len(result)
# ── MA alignment score (bullish alignment) ──
alignment_cols = ["ma5_gt_ma10", "ma10_gt_ma20", "ma20_gt_ma60"]
avg_alignment = float(result[alignment_cols].mean().mean()) # fraction of alignments bullish
# ── Resonance: are indices moving in the same direction? ──
pct_chgs = result["pct_chg"].dropna()
if len(pct_chgs) >= 4:
positive = (pct_chgs > 0).sum()
negative = (pct_chgs < 0).sum()
resonance = max(positive, negative) / len(pct_chgs)
else:
resonance = 0.5
# ── Divergence: std of index returns (high = diverging) ──
divergence = float(pct_chgs.std()) if len(pct_chgs) > 1 else 0.0
# Normalize: 2% std → 1.0
divergence = min(divergence / 0.02, 1.0)
# ── Volume: what fraction of indices have above-avg volume? ──
vol_ratios = result["vol_ratio"].dropna()
vol_expanding = (vol_ratios > 1.0).sum() / len(vol_ratios) if len(vol_ratios) > 0 else 0.5
# ── Trend slope: average MA5 slope across indices ──
slopes = result["ma5_slope"].dropna()
avg_slope = float(slopes.mean()) if len(slopes) > 0 else 0.0
# Normalize: 2% slope → 1.0
slope_normalized = min(max(avg_slope / 0.02 + 0.5, 0.0), 1.0)
return {
"above_ma20_pct": round(above_ma20_count / total, 4),
"above_ma60_pct": round(above_ma60_count / total, 4),
"resonance": round(resonance, 4),
"divergence": round(divergence, 4),
"ma_alignment": round(avg_alignment, 4),
"vol_expanding_pct": round(vol_expanding, 4),
"avg_slope": round(avg_slope, 6),
"index_count": total,
"index_details": [
{
"ts_code": row["ts_code"],
"name": INDEX_CODES.get(row["ts_code"], row["ts_code"]),
"close": round(float(row["close"]), 2),
"pct_chg": round(float(row["pct_chg"] or 0) * 100, 2),
"above_ma20": bool(row["above_ma20"]),
"above_ma60": bool(row["above_ma60"]),
"vol_ratio": round(float(row["vol_ratio"] or 1.0), 2),
}
for _, row in result.iterrows()
],
}
def _empty_trend() -> dict:
return {
"above_ma20_pct": 0.5,
"above_ma60_pct": 0.5,
"resonance": 0.5,
"divergence": 0.5,
"ma_alignment": 0.5,
"vol_expanding_pct": 0.5,
"avg_slope": 0.0,
"index_count": 0,
"index_details": [],
}
registry.register(FeatureDefinition(
name="trend_vector",
category="trend",
description="Index MA positioning, resonance/divergence, bullish alignment, volume expansion",
dependencies=[],
compute=_compute_trend_vector,
))
+128
View File
@@ -0,0 +1,128 @@
"""Volume features: total turnover, vs 5d/20d average, turnover tier."""
from __future__ import annotations
from datetime import date, timedelta
from typing import Any
import duckdb
from ashare_dp.domain.features import FeatureDefinition
from ashare_dp.features.registry import registry
from ashare_dp.data.store.database import analytics_conn, kline_glob
def _compute_volume_vector(db: Any, trade_date: date, _results: dict) -> dict[str, Any]:
"""Compute volume profile: total turnover, comparison to 5d/20d averages.
Also reads the trading_calendar to get prior trading days.
"""
parquet_glob = kline_glob()
conn = analytics_conn()
try:
# Get the last 20 trading days from calendar
cal_sql = """
SELECT trade_date FROM trading_calendar
WHERE trade_date <= $trade_date AND is_trading_day = true
ORDER BY trade_date DESC
LIMIT 22
"""
cal_dates = conn.execute(
cal_sql, {"trade_date": trade_date.isoformat()}
).fetchall()
if not cal_dates:
return _empty_volume()
cal_dates = [row[0] for row in cal_dates]
today_str = cal_dates[0]
# Compute daily turnover for each of the last 20+ trading days
# Use a single DuckDB scan instead of repeated queries
date_list = "', '".join(str(d) for d in cal_dates)
sql = f"""
SELECT
trade_date,
SUM(amount) as daily_amount,
SUM(volume) as daily_volume,
COUNT(*) as stock_count
FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true)
WHERE trade_date IN ('{date_list}')
GROUP BY trade_date
ORDER BY trade_date DESC
"""
result = conn.execute(sql).fetchdf()
finally:
conn.close()
if result.empty:
return _empty_volume()
today_row = result[result["trade_date"] == today_str]
if today_row.empty:
return _empty_volume()
today_amount = float(today_row["daily_amount"].iloc[0])
today_volume = float(today_row["daily_volume"].iloc[0])
today_count = int(today_row["stock_count"].iloc[0])
# 5d avg (today + prior 4 trading days)
recent_5 = result.head(5)
avg_5d_amount = float(recent_5["daily_amount"].mean())
avg_5d_volume = float(recent_5["daily_volume"].mean())
# 20d avg
recent_20 = result.head(min(20, len(result)))
avg_20d_amount = float(recent_20["daily_amount"].mean())
avg_20d_volume = float(recent_20["daily_volume"].mean())
vs_5d = round(today_amount / avg_5d_amount, 4) if avg_5d_amount > 0 else 1.0
vs_20d = round(today_amount / avg_20d_amount, 4) if avg_20d_amount > 0 else 1.0
# Turnover tier
amount_yi = today_amount / 1e8 # 亿
if amount_yi < 8000:
tier = "缩量"
elif amount_yi < 12000:
tier = "正常"
elif amount_yi < 15000:
tier = "放量"
else:
tier = "巨量"
return {
"total_turnover": today_amount,
"total_turnover_yi": round(amount_yi, 0),
"total_volume": int(today_volume),
"stock_count": today_count,
"vs_5d": vs_5d,
"vs_20d": vs_20d,
"avg_5d_amount": avg_5d_amount,
"avg_20d_amount": avg_20d_amount,
"tier": tier,
}
def _empty_volume() -> dict:
return {
"total_turnover": 0,
"total_turnover_yi": 0,
"total_volume": 0,
"stock_count": 0,
"vs_5d": 1.0,
"vs_20d": 1.0,
"avg_5d_amount": 0,
"avg_20d_amount": 0,
"tier": "未知",
}
registry.register(FeatureDefinition(
name="volume_vector",
category="volume",
description="Total market turnover, vs 5d/20d average comparison, turnover tier classification",
dependencies=[],
compute=_compute_volume_vector,
))
+15
View File
@@ -0,0 +1,15 @@
"""MARKET INTELLIGENCE — market state inference and interpretation.
Knows nothing about signals or execution.
"""
from ashare_dp.market.state import infer_market_state
from ashare_dp.market.leadership import assess_leaders
from ashare_dp.market.opportunity import rank_opportunities
from ashare_dp.market.flow import compute_flow
from ashare_dp.market.sentiment import assess_sentiment
from ashare_dp.market.memory import StateStore, state_store
__all__ = [
"infer_market_state", "assess_leaders", "rank_opportunities",
"compute_flow", "assess_sentiment", "StateStore", "state_store",
]
+119
View File
@@ -0,0 +1,119 @@
"""Flow Engine — money flow graph from turnover changes.
Phase 1: uses turnover change as proxy for capital flow.
Phase 2: integrates 北向/ETF/主力/融资/龙虎榜 data.
"""
from __future__ import annotations
from datetime import date, timedelta
import duckdb
from loguru import logger
from ashare_dp.domain.context import FlowEdge, FlowGraph
from ashare_dp.data.store.database import analytics_conn, kline_glob
def compute_flow(trade_date: date) -> FlowGraph:
"""Compute money flow graph from industry turnover changes.
Identifies industries with rising vs falling turnover,
creates directed edges from losers to gainers.
"""
parquet_glob = kline_glob()
conn = analytics_conn()
try:
end_str = trade_date.strftime("%Y-%m-%d")
start_str = (trade_date - timedelta(days=20)).strftime("%Y-%m-%d")
sql = f"""
WITH normalized AS (
SELECT
k.ts_code AS ts_code,
k.trade_date, k.amount
FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true) k
WHERE k.trade_date >= $start_date AND k.trade_date <= $end_date
),
daily AS (
SELECT n.trade_date, n.amount, si.industry_name
FROM normalized n
JOIN stock_industry si ON n.ts_code = si.ts_code
),
industry_daily AS (
SELECT industry_name, trade_date, SUM(amount) AS total_amount
FROM daily GROUP BY industry_name, trade_date
),
ranked AS (
SELECT *,
ROW_NUMBER() OVER (PARTITION BY industry_name ORDER BY trade_date DESC) as rn
FROM industry_daily
),
recent AS (
SELECT industry_name, SUM(total_amount) AS recent_amount
FROM ranked WHERE rn <= 5 GROUP BY industry_name
),
prior AS (
SELECT industry_name, SUM(total_amount) AS prior_amount
FROM ranked WHERE rn > 5 AND rn <= 10 GROUP BY industry_name
)
SELECT
COALESCE(r.industry_name, p.industry_name) AS industry_name,
COALESCE(r.recent_amount, 0) AS recent_amount,
COALESCE(p.prior_amount, 0) AS prior_amount,
CASE WHEN COALESCE(p.prior_amount, 0) > 0
THEN (r.recent_amount - p.prior_amount) / p.prior_amount
ELSE 0 END AS change
FROM recent r
FULL OUTER JOIN prior p ON r.industry_name = p.industry_name
WHERE COALESCE(r.recent_amount, 0) + COALESCE(p.prior_amount, 0) > 0
"""
result = conn.execute(
sql.replace("$start_date", f"'{start_str}'").replace("$end_date", f"'{end_str}'")
).fetchdf()
except Exception as e:
logger.warning(f"Flow computation failed: {e}")
conn.close()
return FlowGraph()
finally:
conn.close()
if result.empty:
return FlowGraph()
# Build flow edges from losers to gainers
changes = {}
for _, row in result.iterrows():
changes[row["industry_name"]] = float(row["change"])
gainers = sorted([(k, v) for k, v in changes.items() if v > 0.05], key=lambda x: -x[1])
losers = sorted([(k, v) for k, v in changes.items() if v < -0.05], key=lambda x: x[1])
edges = []
for i in range(min(len(gainers), len(losers), 5)):
los_name, los_chg = losers[i]
win_name, win_chg = gainers[i]
# Magnitude string
los_mag = "↓↓" if los_chg < -0.15 else ""
win_mag = "↑↑" if win_chg > 0.15 else ""
magnitude = f"{los_mag}{win_mag}"
strength = min((win_chg - los_chg) / 2, 1.0)
edges.append(FlowEdge(
source=los_name,
target=win_name,
magnitude=magnitude,
strength=round(strength, 4),
))
net_inflow = {k: round(v, 4) for k, v in changes.items() if v > 0}
net_outflow = {k: round(v, 4) for k, v in changes.items() if v < 0}
return FlowGraph(
flows=sorted(edges, key=lambda e: -e.strength),
net_inflow=net_inflow,
net_outflow=net_outflow,
)
+166
View File
@@ -0,0 +1,166 @@
"""Knowledge Graph Engine — Theme activation, concept flow, narrative integration.
Maps daily market data (industry turnover, leader states, opportunities)
to the curated theme knowledge graph. Computes theme relevance scores
and detects theme-level rotation.
"""
from __future__ import annotations
from datetime import date
from loguru import logger
from ashare_dp.domain.knowledge import (
Theme, Concept, ThemeEdge, ThemeGraph, THEME_KNOWLEDGE,
)
def activate_themes(
trade_date: date,
leaders: dict | None = None,
opportunities: list | None = None,
) -> ThemeGraph:
"""Compute theme relevance from today's market data.
For each theme in the knowledge graph:
- Match concepts to industries by keyword search on stock names
- Compute relevance from: leader health, opportunity score, turnover
- Build theme-level rotation edges from flow data
Returns a ThemeGraph with scored themes and edges.
"""
themes = []
leaders = leaders or {}
opps = opportunities or []
for theme_tmpl in THEME_KNOWLEDGE:
theme = Theme(name=theme_tmpl.name, concepts=[])
total_score = 0.0
concept_count = 0
for conc_tmpl in theme_tmpl.concepts:
concept = Concept(
name=conc_tmpl.name,
keywords=conc_tmpl.keywords,
leaders=conc_tmpl.leaders,
)
# Find matching opportunities for this concept
concept_score = 0.0
for opp in (opps or []):
# Check if opportunity name matches concept keywords
opp_name = opp.name if hasattr(opp, 'name') else str(opp)
for kw in conc_tmpl.keywords:
if kw in opp_name:
concept_score = max(concept_score, opp.score if hasattr(opp, 'score') else 50)
break
# Check leader health by matching concept keywords against industry names
for ind, ld_data in leaders.items():
if not isinstance(ld_data, dict):
continue
state = ld_data.get("state", "unknown")
# Match concept keywords against industry name
for kw in conc_tmpl.keywords:
if kw in ind or kw in str(ld_data.get("leader", "")):
if state in ("leading", "growing"):
concept_score = max(concept_score, 80)
elif state in ("birth", "recovering"):
concept_score = max(concept_score, 60)
elif state in ("exhausted",):
concept_score = max(concept_score, 40)
elif state in ("breaking", "dead"):
concept_score = max(concept_score, 10)
break
# Match concept keywords against opportunity names
for opp in (opps or []):
opp_name = opp.name if hasattr(opp, 'name') else str(opp)
top_stocks = getattr(opp, 'top_stocks', []) or []
for kw in conc_tmpl.keywords:
if kw in opp_name:
concept_score = max(concept_score, opp.score if hasattr(opp, 'score') else 50)
break
for s in top_stocks:
if kw in str(s):
concept_score = max(concept_score, opp.score if hasattr(opp, 'score') else 50)
break
concept.relevance_score = round(concept_score, 1)
if concept_score > 0:
theme.concepts.append(concept)
total_score += concept_score
concept_count += 1
if concept_count > 0:
theme.relevance_score = round(total_score / concept_count, 1)
else:
theme.relevance_score = 0.0
# Momentum label
if theme.relevance_score >= 70:
theme.momentum = "↑↑"
elif theme.relevance_score >= 50:
theme.momentum = ""
elif theme.relevance_score >= 30:
theme.momentum = ""
elif theme.relevance_score > 0:
theme.momentum = ""
else:
theme.momentum = ""
themes.append(theme)
# Sort by relevance
themes.sort(key=lambda t: t.relevance_score, reverse=True)
# Build theme edges from top-to-bottom flow
edges = []
active = [t for t in themes if t.relevance_score > 30]
if len(active) >= 2:
for i in range(min(len(active) - 1, 3)):
edges.append(ThemeEdge(
source=active[-1 - i].name,
target=active[i].name,
strength=round(active[i].relevance_score / 100, 4),
))
return ThemeGraph(themes=themes, edges=edges)
def describe_theme_graph(tg: ThemeGraph) -> str:
"""Generate a human-readable description of the theme landscape."""
active = [t for t in tg.themes if t.relevance_score > 0]
if not active:
return "无明显主题热点"
parts = []
top3 = active[:3]
parts.append(f"主题集中在{''.join(t.name for t in top3)}")
# Describe top theme's concepts
if top3 and top3[0].concepts:
concepts = [c.name for c in top3[0].concepts if c.relevance_score > 0]
if concepts:
parts.append(f"{top3[0].name}主题下{''.join(concepts[:3])}最活跃")
# Describe rotation
if tg.edges:
e = tg.edges[0]
parts.append(f"资金从{e.source}流向{e.target}")
return "".join(parts) + ""
def get_theme_for_stock(stock_name: str, ts_code: str = "") -> list[str]:
"""Find which themes a stock belongs to."""
matched = []
for theme in THEME_KNOWLEDGE:
for concept in theme.concepts:
for kw in concept.keywords:
if kw in stock_name or kw in ts_code:
if theme.name not in matched:
matched.append(theme.name)
return matched
+137
View File
@@ -0,0 +1,137 @@
"""Leadership Engine — assess leader lifecycle for each major industry.
Not just Alive/Dead. Full lifecycle: Birth → Growing → Leading →
Exhausted → Breaking → Dead → Recovering.
Uses daily K-line data to determine leader state from price/MA/volume.
"""
from __future__ import annotations
from datetime import date
import duckdb
from loguru import logger
from ashare_dp.domain.leadership import LeaderState
from ashare_dp.data.store.database import analytics_conn, kline_glob
def assess_leaders(trade_date: date) -> dict[str, dict]:
"""For each major industry, identify the leader stock and its lifecycle state.
Returns:
{industry_name: {leader, state, close, above_ma20, above_ma60,
momentum, volume_ratio, days_above_ma20}}
Phase 1: uses top stock by amount per industry from stock_industry join.
"""
parquet_glob = kline_glob()
conn = analytics_conn()
try:
sql = f"""
WITH normalized AS (
SELECT
k.ts_code AS ts_code,
k.trade_date, k.trade_time, k.close, k.volume, k.amount, k.high, k.low
FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true) k
WHERE k.trade_date <= $trade_date
),
with_ma AS (
SELECT *,
AVG(close) OVER (PARTITION BY ts_code ORDER BY trade_time ROWS BETWEEN 19 PRECEDING AND CURRENT ROW) AS ma20,
AVG(close) OVER (PARTITION BY ts_code ORDER BY trade_time ROWS BETWEEN 59 PRECEDING AND CURRENT ROW) AS ma60,
AVG(volume) OVER (PARTITION BY ts_code ORDER BY trade_time ROWS BETWEEN 19 PRECEDING AND CURRENT ROW) AS vol_ma20,
ROW_NUMBER() OVER (PARTITION BY ts_code ORDER BY trade_time DESC) as rn
FROM normalized
),
latest AS (
SELECT * FROM with_ma WHERE rn = 1
),
prev AS (
SELECT * FROM with_ma WHERE rn = 21
),
ranked AS (
SELECT
l.ts_code, l.close, l.volume, l.amount,
l.ma20, l.ma60, l.vol_ma20,
p.close AS close_20d_ago,
(l.close - p.close) / NULLIF(p.close, 0) AS momentum_20d,
l.volume / NULLIF(l.vol_ma20, 0) AS vol_ratio,
si.industry_name,
ROW_NUMBER() OVER (PARTITION BY si.industry_name ORDER BY l.amount DESC) as amt_rank
FROM latest l
JOIN stock_industry si ON l.ts_code = si.ts_code
LEFT JOIN prev p ON l.ts_code = p.ts_code
WHERE l.amount > 1e8
)
SELECT * FROM ranked WHERE amt_rank = 1
ORDER BY industry_name
"""
result = conn.execute(sql, {"trade_date": trade_date.isoformat()}).fetchdf()
except Exception as e:
logger.warning(f"Leadership assessment failed: {e}")
conn.close()
return {}
finally:
conn.close()
leaders = {}
for _, row in result.iterrows():
industry = row["industry_name"]
close = float(row["close"])
ma20 = float(row["ma20"] or close)
ma60 = float(row["ma60"] or close)
vol_ratio = float(row["vol_ratio"] or 1.0)
import math as _math
momentum_raw = row["momentum_20d"]
momentum = float(momentum_raw) if (momentum_raw is not None and not (isinstance(momentum_raw, float) and _math.isnan(momentum_raw))) else 0.0
above_ma20 = close > ma20
above_ma60 = close > ma60
state = _classify_leader_state(
above_ma20=above_ma20,
above_ma60=above_ma60,
vol_ratio=vol_ratio,
momentum=momentum,
)
leaders[industry] = {
"leader": row["ts_code"],
"state": state.value,
"close": round(close, 2),
"above_ma20": above_ma20,
"above_ma60": above_ma60,
"momentum_20d": round(momentum, 4),
"volume_ratio": round(vol_ratio, 2),
}
return leaders
def _classify_leader_state(
above_ma20: bool,
above_ma60: bool,
vol_ratio: float,
momentum: float,
) -> LeaderState:
"""Classify leader lifecycle from technical indicators."""
if not above_ma60:
return LeaderState.DEAD
if not above_ma20:
# Was it above MA20 recently? If so, breaking. If not, dead.
return LeaderState.BREAKING
# Above both MAs
if momentum > 0.05 and vol_ratio > 1.2:
return LeaderState.LEADING
elif momentum > 0.02:
return LeaderState.GROWING
elif momentum > 0:
return LeaderState.BIRTH if vol_ratio > 0.8 else LeaderState.EXHAUSTED
elif momentum > -0.02:
return LeaderState.EXHAUSTED
else:
return LeaderState.BREAKING # losing momentum fast
+154
View File
@@ -0,0 +1,154 @@
"""Time Series State Store — persists MarketState snapshots for historical lookup.
NOT limited to daily. Schema: (entity, timeframe, timestamp, state).
Supports find_similar() for historical pattern matching (future Signal Engine).
"""
from __future__ import annotations
import json
from datetime import datetime
from typing import Any
from loguru import logger
from ashare_dp.domain.state import MarketState
class StateStore:
"""Persist and query MarketState snapshots.
Uses the Database class API: db.execute() for writes, db.query() for reads.
"""
def save(
self,
db: Any,
entity: str,
timeframe: str,
timestamp: datetime,
state: MarketState,
) -> None:
"""Save a MarketState snapshot."""
try:
state_json = json.dumps(state.to_dict(), ensure_ascii=False)
db.execute(
"INSERT OR REPLACE INTO state_snapshot (entity, timeframe, timestamp, state) "
"VALUES (?, ?, ?, ?)",
(entity, timeframe, timestamp, state_json),
)
except Exception as e:
logger.warning(f"StateStore.save failed: {e}")
def get_latest(
self, db: Any, entity: str, timeframe: str
) -> MarketState | None:
"""Get the most recent MarketState for an entity/timeframe."""
try:
rows = db.query(
"SELECT state, timestamp FROM state_snapshot "
"WHERE entity = ? AND timeframe = ? "
"ORDER BY timestamp DESC LIMIT 1",
(entity, timeframe),
)
if rows:
state_dict = json.loads(rows[0][0])
return _dict_to_state(state_dict)
return None
except Exception as e:
logger.warning(f"StateStore.get_latest failed: {e}")
return None
def query(
self,
db: Any,
entity: str,
timeframe: str,
start: datetime,
end: datetime,
) -> list[MarketState]:
"""Query MarketState snapshots in a time range."""
try:
rows = db.query(
"SELECT state, timestamp FROM state_snapshot "
"WHERE entity = ? AND timeframe = ? "
"AND timestamp >= ? AND timestamp <= ? "
"ORDER BY timestamp ASC",
(entity, timeframe, start, end),
)
return [_dict_to_state(json.loads(row[0])) for row in rows]
except Exception as e:
logger.warning(f"StateStore.query failed: {e}")
return []
def find_similar(
self, db: Any, state: MarketState, top_k: int = 10
) -> list[tuple[MarketState, float]]:
"""Find historically similar MarketStates by cosine similarity."""
try:
rows = db.query(
"SELECT state, timestamp FROM state_snapshot "
"WHERE entity = 'market' AND timeframe = '1d' "
"ORDER BY timestamp DESC LIMIT 1000"
)
target = state.to_vector()
scored = []
for row in rows:
hist_dict = json.loads(row[0])
hist = _dict_to_state(hist_dict)
hist_vec = hist.to_vector()
sim = _cosine_similarity(target, hist_vec)
scored.append((hist, sim))
scored.sort(key=lambda x: x[1], reverse=True)
return scored[:top_k]
except Exception as e:
logger.warning(f"StateStore.find_similar failed: {e}")
return []
def count(self, db: Any, entity: str, timeframe: str) -> int:
"""Count stored snapshots."""
try:
rows = db.query(
"SELECT COUNT(*) FROM state_snapshot WHERE entity = ? AND timeframe = ?",
(entity, timeframe),
)
return int(rows[0][0]) if rows else 0
except Exception:
return 0
def _dict_to_state(d: dict) -> MarketState:
"""Reconstruct MarketState from serialized dict."""
dims = d.get("dimensions", {})
return MarketState(
timestamp=datetime.fromisoformat(d["timestamp"]),
trade_date=datetime.fromisoformat(d["trade_date"]).date()
if "trade_date" in d
else datetime.fromisoformat(d["timestamp"]).date(),
version=d.get("version", "1.0"),
source=d.get("source", "ashare_dp"),
trend=dims.get("trend", 0.0),
fear=dims.get("fear", 0.0),
liquidity=dims.get("liquidity", 0.0),
rotation=dims.get("rotation", 0.0),
participation=dims.get("participation", 0.0),
volatility=dims.get("volatility", 0.0),
breadth=dims.get("breadth", 0.0),
confidence=d.get("confidence", 0.0),
quality=d.get("quality", 0.0),
)
def _cosine_similarity(a: list[float], b: list[float]) -> float:
"""Compute cosine similarity between two vectors."""
import math
dot = sum(x * y for x, y in zip(a, b))
na = math.sqrt(sum(x * x for x in a))
nb = math.sqrt(sum(x * x for x in b))
if na == 0 or nb == 0:
return 0.0
return dot / (na * nb)
# Module-level singleton
state_store = StateStore()
+198
View File
@@ -0,0 +1,198 @@
"""Opportunity Engine — ranks tradable opportunities by expected participation.
Not industry returns. Not leader rankings. Forward-looking scores for
"where can I make money in the next 1-3 days?"
Scoring: turnover trend + persistence + breadth within industry + leader state.
"""
from __future__ import annotations
from datetime import date, timedelta
import duckdb
from loguru import logger
from ashare_dp.domain.context import Opportunity
from ashare_dp.data.store.database import analytics_conn, kline_glob
def rank_opportunities(trade_date: date, top_n: int = 10) -> list[Opportunity]:
"""Score industries by multi-factor opportunity model.
Factors:
- Turnover momentum (5d change vs prior 5d)
- Persistence (consecutive days as turnover leader)
- Internal breadth (% of stocks in industry advancing)
- Leader health (from Leadership Engine)
Returns sorted list of Opportunity objects.
"""
parquet_glob = kline_glob()
conn = analytics_conn()
try:
# Compute per-industry turnover trends + internal breadth
sql = f"""
WITH normalized AS (
SELECT
k.ts_code AS ts_code,
k.trade_date, k.close, k.open, k.amount
FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true) k
WHERE k.trade_date >= $start_date AND k.trade_date <= $end_date
),
daily AS (
SELECT
n.ts_code, n.trade_date, n.close, n.open, n.amount,
si.industry_name
FROM normalized n
JOIN stock_industry si ON n.ts_code = si.ts_code
),
industry_daily AS (
SELECT
industry_name,
trade_date,
SUM(amount) AS total_amount,
COUNT(*) AS stock_count,
COUNT(*) FILTER (WHERE close > open) AS advance_count
FROM daily
GROUP BY industry_name, trade_date
),
ranked AS (
SELECT *,
ROW_NUMBER() OVER (PARTITION BY industry_name ORDER BY trade_date DESC) as rn
FROM industry_daily
),
recent AS (
SELECT industry_name,
SUM(total_amount) AS recent_amount,
SUM(stock_count) AS recent_stocks,
SUM(advance_count) * 1.0 / NULLIF(SUM(stock_count), 0) AS advance_ratio
FROM ranked WHERE rn <= 5
GROUP BY industry_name
),
prior AS (
SELECT industry_name,
SUM(total_amount) AS prior_amount
FROM ranked WHERE rn > 5 AND rn <= 10
GROUP BY industry_name
),
persistence AS (
SELECT
industry_name,
COUNT(*) FILTER (WHERE daily_rank <= 3) AS lead_days
FROM (
SELECT *,
ROW_NUMBER() OVER (PARTITION BY trade_date ORDER BY total_amount DESC) as daily_rank
FROM industry_daily
WHERE trade_date >= $recent_10
) sub
WHERE daily_rank <= 3
GROUP BY industry_name
)
SELECT
COALESCE(r.industry_name, p.industry_name) AS industry_name,
COALESCE(r.recent_amount, 0) AS recent_amount,
COALESCE(p.prior_amount, 0) AS prior_amount,
CASE WHEN COALESCE(p.prior_amount, 0) > 0
THEN (r.recent_amount - p.prior_amount) / p.prior_amount
ELSE 0 END AS turnover_change,
COALESCE(r.recent_stocks, 0) AS stock_count,
COALESCE(r.advance_ratio, 0.5) AS advance_ratio,
COALESCE(ps.lead_days, 0) AS persistence_days
FROM recent r
FULL OUTER JOIN prior p ON r.industry_name = p.industry_name
LEFT JOIN persistence ps ON r.industry_name = ps.industry_name
WHERE COALESCE(r.recent_amount, 0) + COALESCE(p.prior_amount, 0) > 0
"""
end_str = trade_date.strftime("%Y-%m-%d")
start_str = (trade_date - timedelta(days=30)).strftime("%Y-%m-%d")
recent_10 = (trade_date - timedelta(days=14)).strftime("%Y-%m-%d")
result = conn.execute(
sql.replace("$start_date", f"'{start_str}'")
.replace("$end_date", f"'{end_str}'")
.replace("$recent_10", f"'{recent_10}'")
).fetchdf()
except Exception as e:
logger.warning(f"Opportunity ranking failed: {e}")
conn.close()
return []
finally:
conn.close()
if result.empty:
return []
# Score each industry
opportunities = []
for _, row in result.iterrows():
chg = float(row["turnover_change"])
persistence = int(row["persistence_days"])
advance_ratio = float(row["advance_ratio"])
stock_count = int(row["stock_count"])
# Skip tiny industries
if stock_count < 3:
continue
# Composite score (0-100)
momentum_score = min(max(chg * 100 + 50, 0), 100) # normalize to 0-100
persistence_score = min(persistence * 20, 40) # up to 40 points
breadth_score = advance_ratio * 30 # up to 30 points
score = momentum_score * 0.5 + persistence_score + breadth_score
# Lifecycle classification
if persistence >= 3 and chg > 0.1:
lifecycle = "Accelerating"
elif persistence >= 1 and chg > 0.05:
lifecycle = "Early"
elif chg > 0:
lifecycle = "Peak"
else:
lifecycle = "Declining"
opportunities.append(Opportunity(
name=row["industry_name"],
score=round(min(score, 100), 1),
lifecycle=lifecycle,
persistence_days=persistence,
catalyst="资金轮动" if chg > 0.1 else "",
top_stocks=[], # filled by router
))
# Sort by score desc
opportunities.sort(key=lambda o: o.score, reverse=True)
# Get top stocks for top opportunities (fresh connection needed)
if opportunities:
import duckdb
fresh_conn = duckdb.connect("data/duckdb/ashare.db")
try:
_fill_top_stocks(fresh_conn, trade_date, opportunities[:top_n])
finally:
fresh_conn.close()
return opportunities[:top_n]
def _fill_top_stocks(conn, trade_date: date, opportunities: list[Opportunity]):
"""For each opportunity, find the top 3 stocks by amount."""
parquet_glob = kline_glob()
for opp in opportunities:
try:
sql = f"""
SELECT
k.ts_code AS ts_code
FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true) k
JOIN stock_industry si ON k.ts_code = si.ts_code
WHERE k.trade_date = ? AND si.industry_name = ?
ORDER BY k.amount DESC
LIMIT 3
"""
rows = conn.execute(sql, [trade_date.isoformat(), opp.name]).fetchall()
opp.top_stocks = [r[0] for r in rows]
except Exception:
pass
+246
View File
@@ -0,0 +1,246 @@
"""Stock Recommendation Engine — money flow driven stock picks.
Identifies top stocks in sectors with strong capital inflow,
ranks by momentum/volume/trend, generates actionable trade plans
with entry/stop/target levels.
"""
from __future__ import annotations
from datetime import date, datetime
import duckdb
from loguru import logger
from ashare_dp.data.store.database import analytics_conn, kline_glob
def get_recommendations(
trade_date: date,
top_n: int = 10,
min_amount_yi: float = 2.0, # 最低日成交额(亿)
) -> list[dict]:
"""Generate ranked stock recommendations based on money flow.
1. Find sectors with strongest capital inflow
2. Within each sector, rank stocks by momentum + volume + trend
3. Generate trade plans (entry, stop, targets)
Returns list of recommendation dicts sorted by composite score.
"""
parquet_glob = kline_glob()
conn = analytics_conn()
# ═══ Step 1: Find top inflow sectors ═══
try:
flow_sql = f"""
WITH normalized AS (
SELECT ts_code, trade_date, amount
FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true)
WHERE trade_date >= $start_date AND trade_date <= $end_date
),
daily AS (
SELECT n.trade_date, n.amount, si.industry_name
FROM normalized n
JOIN stock_industry si ON n.ts_code = si.ts_code
),
industry_daily AS (
SELECT industry_name, trade_date, SUM(amount) AS total_amount
FROM daily GROUP BY industry_name, trade_date
),
ranked AS (
SELECT *, ROW_NUMBER() OVER (PARTITION BY industry_name ORDER BY trade_date DESC) AS rn
FROM industry_daily
),
recent AS (
SELECT industry_name, SUM(total_amount) AS recent
FROM ranked WHERE rn <= 5 GROUP BY industry_name
),
prior AS (
SELECT industry_name, SUM(total_amount) AS prior
FROM ranked WHERE rn > 5 AND rn <= 10 GROUP BY industry_name
)
SELECT COALESCE(r.industry_name, p.industry_name) AS industry_name,
CASE WHEN COALESCE(p.prior,0) > 0 THEN (COALESCE(r.recent,0) - p.prior) / p.prior ELSE 0 END AS flow_change
FROM recent r FULL OUTER JOIN prior p ON r.industry_name = p.industry_name
WHERE COALESCE(r.recent, 0) + COALESCE(p.prior, 0) > 0
ORDER BY flow_change DESC
LIMIT 5
"""
end_str = trade_date.strftime("%Y-%m-%d")
from datetime import timedelta
start_str = (trade_date - timedelta(days=20)).strftime("%Y-%m-%d")
flow_df = conn.execute(
flow_sql.replace("$start_date", f"'{start_str}'").replace("$end_date", f"'{end_str}'")
).fetchdf()
top_sectors = flow_df["industry_name"].tolist() if not flow_df.empty else []
except Exception as e:
logger.warning(f"Flow sector query failed: {e}")
top_sectors = []
finally:
conn.close()
if not top_sectors:
return []
# ═══ Step 2: Find best stocks in top sectors ═══
conn = analytics_conn()
recommendations = []
for sector in top_sectors[:3]: # top 3 inflow sectors
try:
sector_list = "', '".join(top_sectors)
stock_sql = f"""
WITH normalized AS (
SELECT ts_code, trade_date, trade_time, close, volume, amount, high, low, open
FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true)
WHERE trade_date <= $trade_date
),
with_ma AS (
SELECT *,
AVG(close) OVER (PARTITION BY ts_code ORDER BY trade_time ROWS BETWEEN 19 PRECEDING AND CURRENT ROW) AS ma20,
AVG(close) OVER (PARTITION BY ts_code ORDER BY trade_time ROWS BETWEEN 59 PRECEDING AND CURRENT ROW) AS ma60,
AVG(volume) OVER (PARTITION BY ts_code ORDER BY trade_time ROWS BETWEEN 19 PRECEDING AND CURRENT ROW) AS vol_ma20,
(high - low) AS day_range,
ROW_NUMBER() OVER (PARTITION BY ts_code ORDER BY trade_time DESC) AS rn
FROM normalized
),
with_atr AS (
SELECT *,
AVG(day_range) OVER (PARTITION BY ts_code ORDER BY trade_time ROWS BETWEEN 19 PRECEDING AND CURRENT ROW) AS atr20
FROM with_ma
),
latest AS (
SELECT w.*, si.industry_name
FROM with_atr w
JOIN stock_industry si ON w.ts_code = si.ts_code
WHERE w.rn = 1 AND si.industry_name = $sector
),
prev AS (
SELECT ts_code, close AS close_prev
FROM with_ma WHERE rn = 6
),
scored AS (
SELECT
l.ts_code, l.industry_name, l.close, l.volume, l.amount,
l.ma20, l.ma60, l.vol_ma20, l.atr20,
l.high, l.low, l.open,
p.close_prev,
-- Momentum: 5d return
(l.close - p.close_prev) / NULLIF(p.close_prev, 0) AS ret_5d,
-- Volume expansion
l.volume / NULLIF(l.vol_ma20, 0) AS vol_ratio,
-- Trend: above MAs
CASE WHEN l.close > l.ma20 THEN 1 ELSE 0 END + CASE WHEN l.close > l.ma60 THEN 1 ELSE 0 END AS trend_score
FROM latest l
LEFT JOIN prev p ON l.ts_code = p.ts_code
WHERE l.amount > $min_amt
)
SELECT *,
(COALESCE(ret_5d, 0) * 40 + LEAST(vol_ratio, 3.0) / 3.0 * 30 + trend_score * 15) AS composite
FROM scored
ORDER BY composite DESC
LIMIT 4
"""
min_amt = min_amount_yi * 1e8
stock_df = conn.execute(
stock_sql,
{"trade_date": trade_date.isoformat(), "sector": sector, "min_amt": min_amt},
).fetchdf()
# Look up stock names
stock_names = {}
try:
nc = duckdb.connect("data/duckdb/ashare.db", read_only=True)
for c in stock_df["ts_code"].tolist():
r = nc.execute("SELECT name FROM stock_info WHERE ts_code=?", [c]).fetchone()
stock_names[c] = r[0] if r else c
nc.close()
except Exception:
pass
for _, row in stock_df.iterrows():
import math
entry = float(row["close"])
atr_raw = row.get("atr20")
atr = float(atr_raw) if atr_raw and not (isinstance(atr_raw, float) and math.isnan(atr_raw)) else entry * 0.03
# Stop: MA20 or 2 ATR below entry
ma20 = float(row["ma20"] or entry)
stop = round(min(ma20 * 0.97, entry - 2 * atr), 2)
# Targets
target1 = round(entry + 1.5 * atr, 2)
target2 = round(entry + 3.0 * atr, 2)
# Risk/reward
risk = entry - stop
reward = target1 - entry
rr_ratio = round(reward / risk, 1) if risk > 0 else 0
# Position sizing guidance
position_advice = "Standard"
if float(row["trend_score"]) >= 2:
position_advice = "Aggressive"
elif float(row.get("vol_ratio", 1)) < 0.8:
position_advice = "Reduced"
import math as _m
def _safe(v, d):
try:
f = float(v)
return d if _m.isnan(f) else f
except (ValueError, TypeError):
return d
_ret5 = _safe(row.get("ret_5d"), 0.0)
_volr = _safe(row.get("vol_ratio"), 1.0)
_trnd = _safe(row.get("trend_score"), 0.0)
_comp = _safe(row.get("composite"), 0.0)
if _m.isnan(_ret5): _ret5 = 0.0
if _m.isnan(_volr): _volr = 1.0
if _m.isnan(_trnd): _trnd = 0.0
if _m.isnan(_comp): _comp = 0.0
ts = row["ts_code"]
recommendations.append({
"ts_code": ts,
"name": stock_names.get(ts, ts),
"sector": sector,
"entry": round(entry, 2),
"stop": stop,
"target1": target1,
"target2": target2,
"rr_ratio": round(float(rr_ratio), 1) if not _m.isnan(rr_ratio) else 0.0,
"position": position_advice,
"score": round(_comp, 1),
"metrics": {
"ret_5d": round(_ret5 * 100, 1),
"vol_ratio": round(_volr, 2),
"trend": "Strong" if _trnd >= 2 else "Weak",
"amount_yi": round(float(row["amount"]) / 1e8, 1),
},
"action": _recommend_action(_ret5, _volr, _trnd),
})
except Exception as e:
logger.warning(f"Stock scoring failed for sector {sector}: {e}")
conn.close()
# Sort by composite score
recommendations.sort(key=lambda r: r["score"], reverse=True)
return recommendations[:top_n]
def _recommend_action(ret_5d: float, vol_ratio: float, trend_score: float) -> str:
"""Generate trading action recommendation."""
if trend_score >= 2 and ret_5d > 0.03 and vol_ratio > 1.2:
return "买入 — 趋势强势,放量上涨"
elif trend_score >= 2 and ret_5d > 0:
return "关注 — 趋势健康,等待回踩"
elif trend_score >= 1 and vol_ratio > 1.0:
return "观察 — 趋势形成中,可轻仓试"
elif ret_5d < -0.03:
return "回避 — 短期偏弱,等待企稳"
else:
return "观望 — 方向不明,暂不参与"
+195
View File
@@ -0,0 +1,195 @@
"""Sentiment Engine — real limit-up/down data from akshare.
Phase 2: fetches 涨停/跌停/炸板 data via EM API,
stores daily snapshots, computes profit effect score.
"""
from __future__ import annotations
from datetime import date, datetime
from loguru import logger
def fetch_daily_sentiment(trade_date: date | None = None) -> dict:
"""Fetch today's sentiment data from akshare.
Uses East Money APIs (accessible outside China — different from stock data).
Returns structured sentiment dict for API/dashboard consumption.
"""
import akshare as ak
if trade_date is None:
trade_date = date.today()
date_str = trade_date.strftime("%Y%m%d")
# ── Fetch limit-up pool ──
try:
zt_df = ak.stock_zt_pool_em(date=date_str)
limit_up_count = len(zt_df)
# Extract board stats
board_stats = _extract_board_stats(zt_df)
except Exception as e:
logger.warning(f"Limit-up fetch failed: {e}")
limit_up_count = 0
board_stats = {}
# ── Fetch limit-down pool ──
try:
dt_df = ak.stock_zt_pool_dtgc_em(date=date_str)
limit_down_count = len(dt_df)
except Exception as e:
logger.warning(f"Limit-down fetch failed: {e}")
limit_down_count = 0
# ── Fetch broken-board pool (炸板) ──
try:
zb_df = ak.stock_zt_pool_zbgc_em(date=date_str)
broken_board_count = len(zb_df)
except Exception as e:
logger.warning(f"Broken-board fetch failed: {e}")
broken_board_count = 0
# ── Compute derived metrics ──
board_total = limit_up_count + broken_board_count
broken_board_rate = round(broken_board_count / board_total, 4) if board_total > 0 else 0
# Board progression rate: 连板数>=2 / total limit-up
consecutive_count = board_stats.get("consecutive_count", 0)
consecutive_board_rate = round(consecutive_count / limit_up_count, 4) if limit_up_count > 0 else 0
# First-board count
first_board_count = board_stats.get("first_board_count", 0)
# Profit effect score (0-100)
profit_score = _compute_profit_score(
limit_up_count=limit_up_count,
limit_down_count=limit_down_count,
broken_board_rate=broken_board_rate,
consecutive_board_rate=consecutive_board_rate,
total_stocks=board_stats.get("total_analyzed", 5000),
)
# Profit effect label
if profit_score >= 80:
profit_effect = "Strong"
elif profit_score >= 60:
profit_effect = "Average"
elif profit_score >= 40:
profit_effect = "Weak"
else:
profit_effect = "Collapse"
return {
"status": "live",
"trade_date": trade_date.isoformat(),
"profit_effect": profit_effect,
"profit_effect_score": profit_score,
"limit_up_count": limit_up_count,
"limit_down_count": limit_down_count,
"broken_board_count": broken_board_count,
"broken_board_rate": broken_board_rate,
"first_board_count": first_board_count,
"consecutive_count": consecutive_count,
"consecutive_board_rate": consecutive_board_rate,
"max_consecutive": board_stats.get("max_consecutive", 0),
"top_industries": board_stats.get("top_industries", []),
}
def _extract_board_stats(zt_df) -> dict:
"""Extract board statistics from limit-up DataFrame."""
total = len(zt_df)
if total == 0:
return {}
consecutive_count = 0
first_board_count = 0
max_consecutive = 0
industry_counts = {}
for _, row in zt_df.iterrows():
# Parse 连板数
lb = row.get("连板数", 0)
try:
lb = int(lb)
except (ValueError, TypeError):
lb = 1
if lb >= 2:
consecutive_count += 1
if lb == 1:
first_board_count += 1
max_consecutive = max(max_consecutive, lb)
# Industry
ind = row.get("所属行业", "其他")
industry_counts[ind] = industry_counts.get(ind, 0) + 1
top_industries = sorted(industry_counts.items(), key=lambda x: -x[1])[:5]
return {
"consecutive_count": consecutive_count,
"first_board_count": first_board_count,
"max_consecutive": max_consecutive,
"top_industries": [{"name": n, "count": c} for n, c in top_industries],
"total_analyzed": total,
}
def _compute_profit_score(
limit_up_count: int,
limit_down_count: int,
broken_board_rate: float,
consecutive_board_rate: float,
total_stocks: int = 5000,
) -> float:
"""Compute profit effect score (0-100).
Factors:
- Limit-up ratio vs total stocks (more = better, up to 50 points)
- Limit-up/down ratio (more ups than downs = better, up to 25 points)
- Broken board rate penalty (higher = worse, up to -15 points)
- Consecutive board rate (higher = stronger sentiment, up to 10 points)
"""
score = 50.0
# Limit-up breadth
zt_pct = limit_up_count / max(total_stocks, 1) * 100
score += min(zt_pct * 3, 20) # 0-20 bonus
# Up/down ratio
if limit_down_count > 0:
ratio = limit_up_count / limit_down_count
score += min(ratio * 2, 15) # 0-15 bonus
else:
score += 15
# Broken board penalty
score -= broken_board_rate * 20 # 0 to -20
# Consecutive board bonus
score += consecutive_board_rate * 15 # 0 to 15
return round(max(0, min(100, score)), 1)
def assess_sentiment(trade_date: date | None = None) -> dict:
"""Main entry point — fetch sentiment for a trading date.
Tries akshare first. Falls back to stored data if API unavailable.
"""
try:
return fetch_daily_sentiment(trade_date)
except Exception as e:
logger.warning(f"Sentiment assessment failed, using fallback: {e}")
return {
"status": "error",
"profit_effect": None,
"limit_up_count": None,
"limit_down_count": None,
"broken_board_rate": None,
"consecutive_board_rate": None,
"profit_effect_score": None,
}
+89
View File
@@ -0,0 +1,89 @@
"""Market State Engine — infers MarketState from feature vectors.
Takes computed features from the Feature Registry, produces a
continuous MarketState vector. Does NOT import individual features.
"""
from __future__ import annotations
from datetime import date, datetime
from typing import Any
from loguru import logger
from ashare_dp.domain.state import MarketState
from ashare_dp.features.registry import FeatureRegistry
def infer_market_state(
features: dict[str, dict[str, Any]],
trade_date: date,
timestamp: datetime | None = None,
) -> MarketState:
"""Infer MarketState from feature vectors.
The Feature Registry computes raw features. This engine
transforms them into the unified MarketState domain object.
Args:
features: Dict mapping feature name → feature value dict,
as returned by FeatureRegistry.compute_all().
trade_date: The trading date these features describe.
timestamp: Optional override for the state timestamp.
Returns:
MarketState domain object with all dimensions populated.
"""
if timestamp is None:
timestamp = datetime.now()
try:
state = MarketState.from_features(features, trade_date, timestamp)
except Exception as e:
logger.error(f"MarketState inference failed: {e}")
state = MarketState(
timestamp=timestamp,
trade_date=trade_date,
confidence=0.0,
quality=0.0,
)
return state
def state_to_label(state: MarketState) -> str:
"""Derive a human-readable regime label from the state vector.
This is for DISPLAY ONLY. Downstream engines use the vector
directly, never the label.
Classification logic:
- trend > 0.65, fear < 0.35 → "趋势"
- trend > 0.65, fear >= 0.35 → "波动上升"
- trend 0.350.65 → "震荡"
- trend < 0.35, fear > 0.65 → "恐慌"
- trend low, fear low → "低迷"
- trend improving, fear decreasing → "复苏"
"""
t = state.trend
f = state.fear
l = state.liquidity
if t > 0.65 and f < 0.35:
if l > 0.7:
return "趋势"
return "趋势(缩量)"
elif t > 0.65 and f >= 0.35:
return "波动上升"
elif t > 0.45:
if f < 0.4:
return "震荡偏强"
return "震荡"
elif t > 0.25:
return "震荡偏弱"
elif f > 0.65:
return "恐慌"
elif t < 0.25 and f < 0.35:
return "低迷"
else:
return "复苏"
-221
View File
@@ -1,221 +0,0 @@
"""Scheduler job implementations."""
from __future__ import annotations
from datetime import date, datetime
from loguru import logger
from ashare_dp.core.calendar import BEIJING_TZ
from ashare_dp.data.eod import EODPipeline
from ashare_dp.storage.repository import KLineRepository
async def eod_pull_job():
"""End-of-day data pull job.
Triggered at 15:05 Beijing time on trading days.
Pulls daily, minute, weekly, and monthly data for all active stocks.
"""
now = datetime.now(BEIJING_TZ)
today = now.date()
# Verify today is a weekday (trading day check done by scheduler)
if now.weekday() >= 5:
logger.info("EOD: Weekend, skipping")
return
logger.info(f"EOD job starting for {today.isoformat()}...")
pipeline = EODPipeline()
try:
symbols = pipeline.get_active_symbols()
logger.info(f"EOD: {len(symbols)} active stocks")
except Exception as e:
logger.error(f"EOD: Failed to get stock list: {e}")
return
# Pull daily data
try:
pipeline.pull_daily(symbols, today)
except Exception as e:
logger.error(f"EOD daily pull failed: {e}")
# Pull minute data (1m, 5m, 15m, 30m, 1h)
try:
pipeline.pull_minute(symbols, today)
except Exception as e:
logger.error(f"EOD minute pull failed: {e}")
# Pull weekly/monthly
try:
pipeline.pull_weekly_monthly(symbols, today)
except Exception as e:
logger.error(f"EOD weekly/monthly pull failed: {e}")
logger.info(f"EOD job completed for {today.isoformat()}")
async def health_check_job():
"""Daily health check: report database statistics."""
logger.info("Health check running...")
repo = KLineRepository()
try:
for freq_res in [repo.get_date_range(f) for f in [Freq.d1, Freq.h1, Freq.m5]]:
pass
logger.info(
f"Health check: DB OK, "
f"1d records={repo.count_records(Freq.d1)}, "
f"1h records={repo.count_records(Freq.h1)}"
)
except Exception as e:
logger.error(f"Health check failed: {e}")
# Import at bottom to avoid circular
from ashare_dp.core.models import Freq
async def ema52_screening_job():
"""EOD EMA52 screening: find stocks near EMA52 for 1d and 1w.
Triggered after EOD data pull. Scans all stocks, computes EMA52
from close prices, and saves results to ema52_screening table.
"""
from ashare_dp.config import Settings
import numpy as np
import pandas as pd
from ashare_dp.storage.database import get_db
settings = Settings()
threshold = settings.ema52_threshold
db = get_db()
# Get all ts_codes and names
stocks = db.query(
"SELECT ts_code, name FROM stock_info ORDER BY ts_code"
)
symbols = [row[0] for row in stocks]
symbol_names = {row[0]: row[1] for row in stocks}
logger.info(
f"EMA52 screening: {len(symbols)} stocks, "
f"threshold=±{threshold * 100:.1f}%"
)
for freq, freq_label, bars_needed in [
(Freq.d1, "1d", 100),
(Freq.w1, "1w", 60),
]:
try:
from ashare_dp.storage.partitioning import partition_glob
glob = partition_glob(freq)
# Get last N bars per stock using ROW_NUMBER()
sql = f"""
WITH ranked AS (
SELECT *, ROW_NUMBER() OVER (
PARTITION BY ts_code ORDER BY trade_time DESC
) as rn
FROM read_parquet('{glob}', hive_partitioning=true, union_by_name=true)
)
SELECT * FROM ranked WHERE rn <= {bars_needed}
ORDER BY ts_code, trade_time ASC
"""
df = db.conn.execute(sql).fetchdf()
if df.empty:
logger.warning(f"EMA52: no data for {freq_label}")
continue
except Exception as e:
logger.error(f"EMA52: failed to read {freq_label}: {e}")
continue
df["trade_time"] = pd.to_datetime(df["trade_time"])
df = df.sort_values(["ts_code", "trade_time"])
# Get last N bars per stock, compute EMA52
results = []
for ts_code, group in df.groupby("ts_code"):
if ts_code not in symbol_names:
continue
group = group.tail(bars_needed)
if len(group) < 26:
continue
closes = group["close"].astype(float).values
# Compute EMA52
ema_values = _compute_ema(closes, 52)
if len(ema_values) == 0:
continue
latest_close = closes[-1]
latest_ema = ema_values[-1]
if latest_ema <= 0:
continue
distance = (latest_close - latest_ema) / latest_ema
# Compute daily amount in 亿 CNY
latest_amount = float(group["amount"].iloc[-1])
amount_yi = latest_amount / 100_000_000 # 元 → 亿
if abs(distance) <= threshold:
results.append({
"trade_date": group["trade_date"].iloc[-1],
"ts_code": ts_code,
"name": symbol_names[ts_code],
"freq": freq_label,
"close_price": round(latest_close, 2),
"ema52": round(latest_ema, 2),
"distance_pct": round(distance * 100, 2),
"amount": round(amount_yi, 2),
})
if results:
# Sort by amount DESC (largest turnover first)
results.sort(key=lambda r: r.get("amount", 0) or 0, reverse=True)
# Upsert: delete old then insert
try:
db.conn.execute(
"DELETE FROM ema52_screening WHERE freq = ?",
[freq_label],
)
db.conn.executemany(
"""INSERT INTO ema52_screening
(trade_date, ts_code, name, freq, close_price, ema52, distance_pct, amount)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT (trade_date, ts_code, freq) DO UPDATE SET
close_price=excluded.close_price,
ema52=excluded.ema52,
distance_pct=excluded.distance_pct,
amount=excluded.amount,
name=excluded.name,
updated_at=now()""",
[(r["trade_date"], r["ts_code"], r["name"],
r["freq"], r["close_price"], r["ema52"], r["distance_pct"], r["amount"])
for r in results],
)
except Exception as e:
logger.error(f"EMA52: failed to save {freq_label}: {e}")
continue
near = [r for r in results if abs(r["distance_pct"]) <= threshold * 100]
logger.info(
f"EMA52 {freq_label}: {len(results)} near "
f"{threshold * 100:.1f}%), {len(near)} within threshold"
)
logger.info("EMA52 screening complete")
def _compute_ema(values: "np.ndarray", period: int) -> "np.ndarray":
"""Compute EMA for a 1D array of values."""
import numpy as np
if len(values) < period:
return np.array([])
alpha = 2.0 / (period + 1)
ema = np.zeros(len(values))
ema[period - 1] = np.mean(values[:period])
for i in range(period, len(values)):
ema[i] = alpha * values[i] + (1 - alpha) * ema[i - 1]
return ema
+19
View File
@@ -0,0 +1,19 @@
"""SIGNAL INTELLIGENCE — Signal × MarketState → Expectancy. The moat.
Signal detection, storage, and historical expectancy queries.
"""
from ashare_dp.signals.detectors import (
detect_ema52_signals, detect_vegas_signals,
detect_chan_signals, detect_orb_signals,
detect_gap_signals, detect_nr7_signals, detect_ib_signals,
run_ema52_screening,
)
from ashare_dp.signals.store import store_signals, count_signals
from ashare_dp.signals.expectancy import get_expectancy
__all__ = [
"detect_ema52_signals", "detect_vegas_signals",
"detect_chan_signals", "detect_orb_signals",
"run_ema52_screening",
"store_signals", "count_signals", "get_expectancy",
]
+696
View File
@@ -0,0 +1,696 @@
"""Signal Detector — finds trading signals in historical K-line data.
Phase 1: EMA52 crossover detection with forward outcome computation.
Single DuckDB scan — detects, captures context, computes outcomes.
"""
from __future__ import annotations
from datetime import date, timedelta
import duckdb
import pandas as pd
from loguru import logger
from ashare_dp.data.store.database import analytics_conn, kline_glob
from ashare_dp.domain.signal import SignalType
def detect_ema52_signals(
end_date: date | None = None,
lookback_days: int = 500,
) -> pd.DataFrame:
"""Detect EMA52 cross signals from daily K-line data.
Single DuckDB query that:
1. Computes EMA52 per stock
2. Detects cross-up and cross-down points
3. Computes forward 5d/10d/20d returns
4. Computes max favorable/adverse excursion
5. Captures daily market breadth at signal time
Args:
end_date: Latest date to scan (default: latest available).
lookback_days: Calendar days to look back.
Returns:
DataFrame with columns matching SignalInstance fields.
"""
parquet_glob = kline_glob()
conn = analytics_conn()
if end_date is None:
row = conn.execute(
f"SELECT MAX(trade_date) FROM read_parquet('{parquet_glob}', "
f"hive_partitioning=true, union_by_name=true)"
).fetchone()
if not row or not row[0]:
conn.close()
return pd.DataFrame()
end_date = date.fromisoformat(str(row[0])[:10])
start_date = end_date - timedelta(days=lookback_days)
try:
sql = f"""
WITH normalized AS (
SELECT
ts_code,
trade_date, trade_time, close, volume, open, high, low
FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true)
WHERE trade_date >= $start_date AND trade_date <= $end_date
),
-- EMA52 per stock
with_ema AS (
SELECT *,
-- EMA52: alpha = 2/(52+1) ≈ 0.03774
-- Use recursive approximation via window
CASE
WHEN ROW_NUMBER() OVER (PARTITION BY ts_code ORDER BY trade_time) = 1
THEN close
ELSE 0.03774 * close + (1 - 0.03774) * LAG(close) OVER (PARTITION BY ts_code ORDER BY trade_time)
END AS ema52_seed
FROM normalized
),
-- Better EMA52: pandas-style using cumulative product isn't easy in SQL.
-- Simplified: use a 52-day SMA as close-enough proxy for Phase 1.
with_sma AS (
SELECT *,
AVG(close) OVER (
PARTITION BY ts_code ORDER BY trade_time
ROWS BETWEEN 51 PRECEDING AND CURRENT ROW
) AS sma52,
ROW_NUMBER() OVER (PARTITION BY ts_code ORDER BY trade_time) as rn
FROM normalized
),
-- Cross detection: signal when close crosses SMA52
with_cross AS (
SELECT *,
LAG(close) OVER (PARTITION BY ts_code ORDER BY trade_time) AS prev_close,
LAG(sma52) OVER (PARTITION BY ts_code ORDER BY trade_time) AS prev_sma52,
-- Daily market breadth at this date
AVG(CASE WHEN close > open THEN 1.0 ELSE 0.0 END) OVER (PARTITION BY trade_date) AS daily_advance_ratio,
SUM(volume) OVER (PARTITION BY trade_date) AS daily_total_volume,
AVG((high - low) / NULLIF(close, 0)) OVER (PARTITION BY trade_date) AS daily_range
FROM with_sma
WHERE rn > 52 -- need SMA to be valid
),
signals AS (
SELECT
ts_code,
trade_date,
trade_time,
close AS signal_price,
sma52,
daily_advance_ratio,
daily_range,
-- Cross detection
CASE
WHEN prev_close <= prev_sma52 AND close > sma52
THEN 'ema52_cross_up'
WHEN prev_close >= prev_sma52 AND close < sma52
THEN 'ema52_cross_down'
ELSE NULL
END AS signal_type,
rn
FROM with_cross
),
-- Forward returns (join with future data)
signal_with_future AS (
SELECT
s.ts_code, s.trade_date, s.signal_price, s.sma52,
s.daily_advance_ratio, s.daily_range, s.signal_type, s.rn,
-- 5d forward
f5.close AS close_5d,
(f5.close - s.signal_price) / NULLIF(s.signal_price, 0) AS ret_5d,
-- 10d forward
f10.close AS close_10d,
(f10.close - s.signal_price) / NULLIF(s.signal_price, 0) AS ret_10d,
-- 20d forward
f20.close AS close_20d,
(f20.close - s.signal_price) / NULLIF(s.signal_price, 0) AS ret_20d
FROM signals s
LEFT JOIN with_sma f5 ON s.ts_code = f5.ts_code AND f5.rn = s.rn + 5
LEFT JOIN with_sma f10 ON s.ts_code = f10.ts_code AND f10.rn = s.rn + 10
LEFT JOIN with_sma f20 ON s.ts_code = f20.ts_code AND f20.rn = s.rn + 20
WHERE s.signal_type IS NOT NULL
),
-- Max drawdown within 5d window
with_mdd AS (
SELECT sf.*,
-- 5d max drawdown: need to check each day between signal and 5d later
-- Simplified: use the min close in the window
MIN(w.close) OVER (
PARTITION BY sf.ts_code, sf.trade_date
ORDER BY w.trade_time
ROWS BETWEEN CURRENT ROW AND 4 FOLLOWING
) AS min_close_5d
FROM signal_with_future sf
LEFT JOIN with_sma w
ON sf.ts_code = w.ts_code
AND w.rn BETWEEN sf.rn AND sf.rn + 5
)
SELECT DISTINCT
signal_type,
ts_code,
trade_date,
signal_price,
daily_advance_ratio,
daily_range,
ret_5d,
ret_10d,
ret_20d,
-- Max drawdown within 5d
CASE WHEN min_close_5d IS NOT NULL AND signal_price > 0
THEN (min_close_5d - signal_price) / signal_price
ELSE NULL END AS max_dd_5d,
-- Max return within 5d (approximate)
CASE WHEN close_5d IS NOT NULL AND signal_price > 0
THEN (close_5d - signal_price) / signal_price
ELSE NULL END AS max_return_5d
FROM with_mdd
WHERE signal_type IS NOT NULL
ORDER BY trade_date, ts_code
"""
result = conn.execute(
sql.replace("$start_date", f"'{start_date.strftime('%Y-%m-%d')}'")
.replace("$end_date", f"'{end_date.strftime('%Y-%m-%d')}'")
).fetchdf()
except Exception as e:
logger.error(f"Signal detection failed: {e}")
conn.close()
return pd.DataFrame()
finally:
conn.close()
if result.empty:
return result
# Clean up
result = result.drop_duplicates(subset=["signal_type", "ts_code", "trade_date"])
logger.info(
f"Detected {len(result)} EMA52 signals "
f"({len(result[result['signal_type']=='ema52_cross_up'])} cross-up, "
f"{len(result[result['signal_type']=='ema52_cross_down'])} cross-down)"
)
return result
# ═══════════════════════════════════════════════════
# Vegas Channel Detection
# ═══════════════════════════════════════════════════
def detect_vegas_signals(end_date=None, lookback_days=500) -> "pd.DataFrame":
"""Detect Vegas Channel signals from daily K-line.
Vegas Channel: EMA12, EMA50, EMA144.
LONG signal: price > EMA144, EMA12 > EMA50 > EMA144 (bullish tunnel),
price pulls back to near EMA50.
SHORT signal: inverse.
"""
import math as _math
from datetime import date as _date, timedelta as _td
parquet_glob = kline_glob()
conn = analytics_conn()
if end_date is None:
row = conn.execute(
f"SELECT MAX(trade_date) FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true)"
).fetchone()
if not row or not row[0]:
conn.close()
return pd.DataFrame()
end_date = _date.fromisoformat(str(row[0])[:10])
start_date = end_date - _td(days=lookback_days)
try:
sql = f"""
WITH normalized AS (
SELECT ts_code, trade_date, trade_time, close, volume, high, low, open
FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true)
WHERE trade_date >= $start AND trade_date <= $end
),
with_ema AS (
SELECT *,
AVG(close) OVER (PARTITION BY ts_code ORDER BY trade_time ROWS BETWEEN 11 PRECEDING AND CURRENT ROW) AS ema12,
AVG(close) OVER (PARTITION BY ts_code ORDER BY trade_time ROWS BETWEEN 49 PRECEDING AND CURRENT ROW) AS ema50,
AVG(close) OVER (PARTITION BY ts_code ORDER BY trade_time ROWS BETWEEN 143 PRECEDING AND CURRENT ROW) AS ema144,
ROW_NUMBER() OVER (PARTITION BY ts_code ORDER BY trade_time) as rn,
LAG(close) OVER (PARTITION BY ts_code ORDER BY trade_time) AS prev_close
FROM normalized
),
with_signal AS (
SELECT *,
-- Vegas LONG: bullish tunnel (ema12>ema50>ema144) + price near EMA50 (within 3%)
CASE WHEN rn > 144
AND ema12 > ema50 AND ema50 > ema144
AND close > ema144
AND ABS(close - ema50) / NULLIF(ema50, 0) < 0.03
AND prev_close <= ema50
THEN 'vegas_long' ELSE NULL END AS signal_type_up,
-- Vegas SHORT
CASE WHEN rn > 144
AND ema12 < ema50 AND ema50 < ema144
AND close < ema144
AND ABS(close - ema50) / NULLIF(ema50, 0) < 0.03
AND prev_close >= ema50
THEN 'vegas_short' ELSE NULL END AS signal_type_down
FROM with_ema
)
SELECT
COALESCE(signal_type_up, signal_type_down) AS signal_type,
ts_code, trade_date, close AS signal_price,
AVG(CASE WHEN close > open THEN 1.0 ELSE 0.0 END) OVER (PARTITION BY trade_date) AS daily_advance_ratio,
AVG((high - low) / NULLIF(close, 0)) OVER (PARTITION BY trade_date) AS daily_range,
rn
FROM with_signal
WHERE signal_type_up IS NOT NULL OR signal_type_down IS NOT NULL
"""
result = conn.execute(
sql.replace("$start", f"'{start_date.strftime('%Y-%m-%d')}'")
.replace("$end", f"'{end_date.strftime('%Y-%m-%d')}'")
).fetchdf()
finally:
conn.close()
if result.empty:
return result
result = result.drop_duplicates(subset=["signal_type", "ts_code", "trade_date"])
_compute_forward_returns(result, parquet_glob)
logger.info(f"Detected {len(result)} Vegas signals")
return result
# ═══════════════════════════════════════════════════
# Chan Theory Detection (MACD divergence)
# ═══════════════════════════════════════════════════
def detect_chan_signals(end_date=None, lookback_days=500) -> "pd.DataFrame":
"""Detect Chan Theory 1buy/1sell signals via MACD divergence.
1buy: price makes lower low but MACD makes higher low (bottom divergence).
1sell: price makes higher high but MACD makes lower high (top divergence).
"""
from datetime import date as _date, timedelta as _td
parquet_glob = kline_glob()
conn = analytics_conn()
if end_date is None:
row = conn.execute(
f"SELECT MAX(trade_date) FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true)"
).fetchone()
if not row or not row[0]:
conn.close()
return pd.DataFrame()
end_date = _date.fromisoformat(str(row[0])[:10])
start_date = end_date - _td(days=lookback_days)
try:
sql = f"""
WITH normalized AS (
SELECT ts_code, trade_date, trade_time, close, volume, high, low, open
FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true)
WHERE trade_date >= $start AND trade_date <= $end
),
with_macd AS (
SELECT *,
AVG(close) OVER (PARTITION BY ts_code ORDER BY trade_time ROWS BETWEEN 11 PRECEDING AND CURRENT ROW)
- AVG(close) OVER (PARTITION BY ts_code ORDER BY trade_time ROWS BETWEEN 25 PRECEDING AND CURRENT ROW) AS macd,
ROW_NUMBER() OVER (PARTITION BY ts_code ORDER BY trade_time) as rn
FROM normalized
),
with_signal AS (
SELECT *,
AVG(macd) OVER (PARTITION BY ts_code ORDER BY trade_time ROWS BETWEEN 8 PRECEDING AND CURRENT ROW) AS macd_signal
FROM with_macd
),
with_divergence AS (
SELECT *,
MIN(close) OVER (PARTITION BY ts_code ORDER BY trade_time ROWS BETWEEN 59 PRECEDING AND CURRENT ROW) AS price_60low,
MAX(close) OVER (PARTITION BY ts_code ORDER BY trade_time ROWS BETWEEN 59 PRECEDING AND CURRENT ROW) AS price_60high,
MIN(macd) OVER (PARTITION BY ts_code ORDER BY trade_time ROWS BETWEEN 59 PRECEDING AND CURRENT ROW) AS macd_60low,
MAX(macd) OVER (PARTITION BY ts_code ORDER BY trade_time ROWS BETWEEN 59 PRECEDING AND CURRENT ROW) AS macd_60high,
LAG(macd, 20) OVER (PARTITION BY ts_code ORDER BY trade_time) AS macd_20ago
FROM with_signal
WHERE rn > 60
)
SELECT
CASE
WHEN close <= price_60low * 1.02 AND macd > macd_60low * 1.1 AND macd > macd_20ago
THEN 'chan_1buy'
WHEN close >= price_60high * 0.98 AND macd < macd_60high * 0.9 AND macd < macd_20ago
THEN 'chan_1sell'
END AS signal_type,
ts_code, trade_date, close AS signal_price,
AVG(CASE WHEN close > open THEN 1.0 ELSE 0.0 END) OVER (PARTITION BY trade_date) AS daily_advance_ratio,
AVG((high - low) / NULLIF(close, 0)) OVER (PARTITION BY trade_date) AS daily_range
FROM with_divergence
WHERE (close <= price_60low * 1.02 AND macd > macd_60low * 1.1 AND macd > macd_20ago)
OR (close >= price_60high * 0.98 AND macd < macd_60high * 0.9 AND macd < macd_20ago)
"""
result = conn.execute(
sql.replace("$start", f"'{start_date.strftime('%Y-%m-%d')}'")
.replace("$end", f"'{end_date.strftime('%Y-%m-%d')}'")
).fetchdf()
finally:
conn.close()
if result.empty:
return result
result = result.drop_duplicates(subset=["signal_type", "ts_code", "trade_date"])
_compute_forward_returns(result, parquet_glob)
logger.info(f"Detected {len(result)} Chan signals")
return result
# ═══════════════════════════════════════════════════
# ORB Detection (Opening Range Breakout — daily)
# ═══════════════════════════════════════════════════
def detect_orb_signals(end_date=None, lookback_days=500) -> "pd.DataFrame":
"""Detect daily ORB signals: close breaks above yesterday's high or below low.
Simple but effective — breakouts often lead to trend days.
"""
from datetime import date as _date, timedelta as _td
parquet_glob = kline_glob()
conn = analytics_conn()
if end_date is None:
row = conn.execute(
f"SELECT MAX(trade_date) FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true)"
).fetchone()
if not row or not row[0]:
conn.close()
return pd.DataFrame()
end_date = _date.fromisoformat(str(row[0])[:10])
start_date = end_date - _td(days=lookback_days)
try:
sql = f"""
WITH normalized AS (
SELECT ts_code, trade_date, trade_time, close, volume, high, low, open
FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true)
WHERE trade_date >= $start AND trade_date <= $end
),
with_prev AS (
SELECT *,
LAG(high) OVER (PARTITION BY ts_code ORDER BY trade_time) AS prev_high,
LAG(low) OVER (PARTITION BY ts_code ORDER BY trade_time) AS prev_low,
LAG(close) OVER (PARTITION BY ts_code ORDER BY trade_time) AS prev_close
FROM normalized
)
SELECT
CASE
WHEN close > prev_high AND prev_close <= prev_high THEN 'orb_up'
WHEN close < prev_low AND prev_close >= prev_low THEN 'orb_down'
END AS signal_type,
ts_code, trade_date, close AS signal_price,
AVG(CASE WHEN close > open THEN 1.0 ELSE 0.0 END) OVER (PARTITION BY trade_date) AS daily_advance_ratio,
AVG((high - low) / NULLIF(close, 0)) OVER (PARTITION BY trade_date) AS daily_range
FROM with_prev
WHERE (close > prev_high AND prev_close <= prev_high)
OR (close < prev_low AND prev_close >= prev_low)
"""
result = conn.execute(
sql.replace("$start", f"'{start_date.strftime('%Y-%m-%d')}'")
.replace("$end", f"'{end_date.strftime('%Y-%m-%d')}'")
).fetchdf()
finally:
conn.close()
if result.empty:
return result
result = result.drop_duplicates(subset=["signal_type", "ts_code", "trade_date"])
_compute_forward_returns(result, parquet_glob)
logger.info(f"Detected {len(result)} ORB signals")
return result
# ══════════════════════════════════════════
# Shared Forward Returns Computation
# ══════════════════════════════════════════
def _compute_forward_returns(df: "pd.DataFrame", parquet_glob: str):
"""Join forward returns (5d, 10d, 20d) onto a signal DataFrame. Mutates in place."""
conn = analytics_conn()
try:
for days, col_name in [(5, "ret_5d"), (10, "ret_10d"), (20, "ret_20d")]:
sql = f"""
WITH future AS (
SELECT ts_code, trade_date, close,
LEAD(close, {days}) OVER (PARTITION BY ts_code ORDER BY trade_date) AS future_close,
MIN(close) OVER (PARTITION BY ts_code ORDER BY trade_date ROWS BETWEEN CURRENT ROW AND {days} FOLLOWING) AS min_close
FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true)
)
SELECT ts_code, trade_date,
(future_close - close) / NULLIF(close, 0) AS ret,
(min_close - close) / NULLIF(close, 0) AS mdd
FROM future
"""
fwd = conn.execute(sql).fetchdf()
fwd["trade_date"] = pd.to_datetime(fwd["trade_date"])
df["trade_date"] = pd.to_datetime(df["trade_date"])
# Merge forward returns
df_temp = df.merge(fwd, on=["ts_code", "trade_date"], how="left")
if col_name not in df.columns:
df[col_name] = df_temp["ret"] if "ret" in df_temp.columns else None
if col_name == "ret_5d":
if "max_dd_5d" not in df.columns:
df["max_dd_5d"] = df_temp["mdd"] if "mdd" in df_temp.columns else None
if "max_return_5d" not in df.columns:
df["max_return_5d"] = df_temp["ret"] if "ret" in df_temp.columns else None
finally:
conn.close()
# ══════════════════════════════════════════
# EMA52 Screening — CLI + scheduler shared
# ══════════════════════════════════════════
def run_ema52_screening(
threshold: float | None = None,
freqs: list | None = None,
) -> dict:
"""Run EMA52 screening. Shared by CLI and scheduler — no duplicated logic."""
import numpy as np
import pandas as pd
from ashare_dp.config import Settings
from ashare_dp.core.models import Freq
from ashare_dp.data.store.database import get_db
from ashare_dp.data.store.partitioning import partition_glob
settings = Settings()
if threshold is None:
threshold = settings.ema52_threshold
if freqs is None:
freqs = [(Freq.d1, "1d", 100), (Freq.w1, "1w", 60)]
with get_db(read_only=True) as db:
stocks = db.query("SELECT ts_code, name FROM stock_info ORDER BY ts_code")
symbol_names = {row[0]: row[1] for row in stocks}
logger.info(f"EMA52 screening: {len(symbol_names)} stocks, threshold=±{threshold*100:.1f}%")
results_by_freq = {}
for freq, freq_label, bars_needed in freqs:
try:
glob = partition_glob(freq)
sql = f"""
WITH ranked AS (
SELECT *, ROW_NUMBER() OVER (PARTITION BY ts_code ORDER BY trade_time DESC) as rn
FROM read_parquet('{glob}', hive_partitioning=true, union_by_name=true)
)
SELECT * FROM ranked WHERE rn <= {bars_needed} ORDER BY ts_code, trade_time ASC
"""
import duckdb
c = duckdb.connect()
df = c.execute(sql).fetchdf() if True else None
c.close()
if df.empty:
logger.warning(f"EMA52: no data for {freq_label}")
continue
except Exception as e:
logger.error(f"EMA52: read failed {freq_label}: {e}")
continue
df["trade_time"] = pd.to_datetime(df["trade_time"])
df = df.sort_values(["ts_code", "trade_time"])
found = []
for ts_code, group in df.groupby("ts_code"):
if ts_code not in symbol_names:
continue
group = group.tail(bars_needed)
if len(group) < 26:
continue
closes = group["close"].astype(float).values
ema = _ema(closes, 52)
if len(ema) == 0:
continue
latest_close, latest_ema = closes[-1], ema[-1]
if latest_ema <= 0:
continue
dist = (latest_close - latest_ema) / latest_ema
if abs(dist) <= threshold:
found.append({
"trade_date": group["trade_date"].iloc[-1],
"ts_code": ts_code, "name": symbol_names[ts_code],
"freq": freq_label,
"close_price": round(latest_close, 2),
"ema52": round(latest_ema, 2),
"distance_pct": round(dist * 100, 2),
"amount": round(float(group["amount"].iloc[-1]) / 1e8, 2),
})
if found:
found.sort(key=lambda r: r.get("amount", 0) or 0, reverse=True)
with get_db(read_only=False) as wdb:
wdb.conn.execute("DELETE FROM ema52_screening WHERE freq = ?", [freq_label])
wdb.conn.executemany(
"""INSERT INTO ema52_screening
(trade_date, ts_code, name, freq, close_price, ema52, distance_pct, amount)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT (trade_date, ts_code, freq) DO UPDATE SET
close_price=excluded.close_price, ema52=excluded.ema52,
distance_pct=excluded.distance_pct, amount=excluded.amount,
name=excluded.name, updated_at=now()""",
[(r["trade_date"], r["ts_code"], r["name"], r["freq"], r["close_price"],
r["ema52"], r["distance_pct"], r["amount"]) for r in found],
)
logger.info(f"EMA52 {freq_label}: {len(found)} near (±{threshold*100:.1f}%)")
results_by_freq[freq_label] = len(found)
logger.info("EMA52 screening complete")
return results_by_freq
# ══════════════════════════════════════════
# Gap / NR7 / Inside Bar Detection
# ══════════════════════════════════════════
def detect_gap_signals(end_date=None, lookback_days=500) -> "pd.DataFrame":
"""Detect gap-up and gap-down signals."""
from datetime import date as _date, timedelta as _td
parquet_glob = kline_glob()
conn = analytics_conn()
if end_date is None:
r = conn.execute(f"SELECT MAX(trade_date) FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true)").fetchone()
end_date = _date.fromisoformat(str(r[0])[:10]) if r and r[0] else _date.today()
start_date = end_date - _td(days=lookback_days)
try:
sql = f"""
WITH normalized AS (
SELECT ts_code, trade_date, close, open, high, low
FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true)
WHERE trade_date >= $s AND trade_date <= $e
),
prev AS (
SELECT *, LAG(high) OVER (PARTITION BY ts_code ORDER BY trade_date) AS prev_high,
LAG(low) OVER (PARTITION BY ts_code ORDER BY trade_date) AS prev_low
FROM normalized
)
SELECT CASE WHEN open > prev_high THEN 'gap_up' WHEN open < prev_low THEN 'gap_down' END AS signal_type,
ts_code, trade_date, close AS signal_price
FROM prev WHERE (open > prev_high) OR (open < prev_low)
"""
df = conn.execute(sql.replace("$s", f"'{start_date}'").replace("$e", f"'{end_date}'")).fetchdf()
finally:
conn.close()
if not df.empty:
df = df.drop_duplicates(subset=["signal_type", "ts_code", "trade_date"])
logger.info(f"Detected {len(df)} Gap signals")
return df
def detect_nr7_signals(end_date=None, lookback_days=500) -> "pd.DataFrame":
"""Detect NR7: narrowest range in 7 days."""
from datetime import date as _date, timedelta as _td
parquet_glob = kline_glob()
conn = analytics_conn()
if end_date is None:
r = conn.execute(f"SELECT MAX(trade_date) FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true)").fetchone()
end_date = _date.fromisoformat(str(r[0])[:10]) if r and r[0] else _date.today()
start_date = end_date - _td(days=lookback_days)
try:
sql = f"""
WITH ranges AS (
SELECT ts_code, trade_date, close, (high - low) AS day_range,
ROW_NUMBER() OVER (PARTITION BY ts_code ORDER BY trade_date) AS rn
FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true)
WHERE trade_date >= $s AND trade_date <= $e
),
window_min AS (
SELECT *,
MIN(day_range) OVER (PARTITION BY ts_code ORDER BY trade_date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS min_range_7,
COUNT(*) OVER (PARTITION BY ts_code ORDER BY trade_date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS window_count
FROM ranges
)
SELECT 'nr7' AS signal_type, ts_code, trade_date, close AS signal_price
FROM window_min WHERE day_range = min_range_7 AND window_count = 7 AND rn >= 7
"""
df = conn.execute(sql.replace("$s", f"'{start_date}'").replace("$e", f"'{end_date}'")).fetchdf()
finally:
conn.close()
if not df.empty:
df = df.drop_duplicates(subset=["ts_code", "trade_date"])
logger.info(f"Detected {len(df)} NR7 signals")
return df
def detect_ib_signals(end_date=None, lookback_days=500) -> "pd.DataFrame":
"""Detect Inside Bar: today's range inside yesterday's."""
from datetime import date as _date, timedelta as _td
parquet_glob = kline_glob()
conn = analytics_conn()
if end_date is None:
r = conn.execute(f"SELECT MAX(trade_date) FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true)").fetchone()
end_date = _date.fromisoformat(str(r[0])[:10]) if r and r[0] else _date.today()
start_date = end_date - _td(days=lookback_days)
try:
sql = f"""
WITH normalized AS (
SELECT ts_code, trade_date, close, open, high, low
FROM read_parquet('{parquet_glob}', hive_partitioning=true, union_by_name=true)
WHERE trade_date >= $s AND trade_date <= $e
),
prev AS (
SELECT *, LAG(high) OVER (PARTITION BY ts_code ORDER BY trade_date) AS prev_high,
LAG(low) OVER (PARTITION BY ts_code ORDER BY trade_date) AS prev_low,
LAG(close) OVER (PARTITION BY ts_code ORDER BY trade_date) AS prev_close
FROM normalized
)
SELECT CASE WHEN close > (high+low)/2.0 THEN 'ib_long' ELSE 'ib_short' END AS signal_type,
ts_code, trade_date, close AS signal_price
FROM prev WHERE high < prev_high AND low > prev_low AND prev_high IS NOT NULL
"""
df = conn.execute(sql.replace("$s", f"'{start_date}'").replace("$e", f"'{end_date}'")).fetchdf()
finally:
conn.close()
if not df.empty:
df = df.drop_duplicates(subset=["ts_code", "trade_date"])
logger.info(f"Detected {len(df)} Inside Bar signals")
return df
def _ema(values, period: int):
"""EMA for 1D array."""
import numpy as np
if len(values) < period:
return np.array([])
a = 2.0 / (period + 1)
ema = np.zeros(len(values))
ema[period - 1] = np.mean(values[:period])
for i in range(period, len(values)):
ema[i] = a * values[i] + (1 - a) * ema[i - 1]
return ema
+176
View File
@@ -0,0 +1,176 @@
"""Expectancy Engine — Signal × MarketState → historical expectancy.
Queries signal_instance table for signals in similar market contexts.
This is the moat. Phase 1: working with EMA52 signals.
Phase 2: all signal types.
"""
from __future__ import annotations
import math
from datetime import date
from loguru import logger
from ashare_dp.domain.context import Expectancy
from ashare_dp.domain.state import MarketState
from ashare_dp.data.store.database import get_db
# Signal types to try in priority order when querying general expectancy
_SIGNAL_PRIORITY = [
"ema52_cross_up", "ema52_cross_down",
"vegas_long", "chan_1buy", "orb_up",
"vegas_short", "chan_1sell", "orb_down",
"gap_up", "nr7", "ib_long",
"gap_down", "ib_short",
]
def get_expectancy(state: MarketState, signal_type: str | None = None) -> Expectancy:
"""The single entry point for expectancy queries.
If signal_type given: query that signal's historical stats.
If None: try signals in priority order, return first with real samples.
Falls back to heuristic estimate when no historical data matches.
All fallback logic lives HERE — callers never chain fallbacks themselves.
"""
if signal_type:
return query_expectancy(state, signal_type=signal_type)
for st in _SIGNAL_PRIORITY:
exp = query_expectancy(state, signal_type=st)
if exp.sample_count > 0:
return exp
return estimate_expectancy(state, signal_type="generic")
def get_all_expectancies(state: MarketState) -> list[Expectancy]:
"""Query ALL signal types and return ranked by win_rate desc.
Used by the dashboard Expectancy comparison table.
Signal types with 0 samples get heuristic estimates (marked by sample_count=0).
"""
results = []
for st in _SIGNAL_PRIORITY:
exp = query_expectancy(state, signal_type=st, min_samples=1)
results.append(exp)
# Sort: real samples first (by win_rate desc), then heuristics last
results.sort(key=lambda e: (e.sample_count > 0, e.win_rate), reverse=True)
return results
def query_expectancy(
state: MarketState,
signal_type: str = "ema52_cross_up",
min_samples: int = 5,
) -> Expectancy:
"""Query historical expectancy for a signal type in current market state.
Matches signals with similar market context (state vector cosine similarity
or simple threshold matching in Phase 1).
Args:
state: Current MarketState vector.
signal_type: Signal type to query (e.g., "ema52_cross_up").
min_samples: Minimum historical samples required for valid result.
Returns:
Expectancy with real historical stats, or heuristic fallback if
insufficient samples.
"""
try:
with get_db(read_only=True) as db:
# Query all signals of this type with outcomes
rows = db.query(
"""
SELECT
signal_type, ts_code, trade_date, signal_price,
state_breadth, state_volatility,
return_5d, return_10d, return_20d,
max_return_5d, max_drawdown_5d
FROM signal_instance
WHERE signal_type = ?
AND outcome_known = TRUE
AND return_5d IS NOT NULL
ORDER BY trade_date DESC
LIMIT 5000
""",
(signal_type,),
)
except Exception as e:
logger.warning(f"Expectancy query failed: {e}")
return estimate_expectancy(state, signal_type)
if len(rows) < min_samples:
logger.info(f"Only {len(rows)} samples for {signal_type}, using heuristic")
return estimate_expectancy(state, signal_type)
# Filter by similar market state (simple threshold match in Phase 1)
target_breadth = state.breadth
similar = []
for row in rows:
hist_breadth = float(row[4] or 0.5)
# Simple similarity: breadth within ±0.15
if abs(hist_breadth - target_breadth) < 0.15:
similar.append(row)
if len(similar) < max(min_samples, 1):
# Not enough similar contexts — use all available samples
similar = rows
# Compute expectancy stats
# Columns: 0=signal_type, 1=ts_code, 2=trade_date, 3=signal_price,
# 4=state_breadth, 5=state_volatility, 6=return_5d, 7=return_10d,
# 8=return_20d, 9=max_return_5d, 10=max_drawdown_5d
returns_5d = [float(r[6]) for r in similar if r[6] is not None]
returns_10d = [float(r[7]) for r in similar if r[7] is not None]
returns_20d = [float(r[8]) for r in similar if r[8] is not None]
drawdowns = [float(r[10]) for r in similar if r[10] is not None]
if not returns_5d:
return estimate_expectancy(state, signal_type)
wins = sum(1 for r in returns_5d if r > 0)
win_rate = wins / len(returns_5d)
avg_return = sum(returns_5d) / len(returns_5d)
max_dd = min(drawdowns) if drawdowns else avg_return * -0.5
avg_hold = 5.0 # Phase 1: fixed 5d horizon
confidence = min(len(similar) / 200, 1.0) # scale with sample size
return Expectancy(
signal_type=signal_type,
win_rate=round(win_rate, 4),
avg_return=round(avg_return, 4),
max_drawdown=round(max_dd, 4),
avg_holding_days=avg_hold,
sample_count=len(similar),
confidence=round(confidence, 4),
similar_states=len(similar),
)
def estimate_expectancy(state: MarketState, signal_type: str = "generic") -> Expectancy:
"""Heuristic fallback when insufficient historical data."""
t = state.trend
f = state.fear
l = state.liquidity
base_win = 0.50 + max(t - 0.5, 0) * 0.20 - f * 0.15 + max(l - 0.5, 0) * 0.10
win_rate = min(max(base_win, 0.25), 0.80)
avg_return = t * 0.08 - f * 0.04
max_dd = -(f * 0.04 + (1 - l) * 0.02)
avg_hold = 2.0 + t * 4.0 - f * 2.0
return Expectancy(
signal_type=signal_type,
win_rate=round(win_rate, 4),
avg_return=round(avg_return, 4),
max_drawdown=round(max_dd, 4),
avg_holding_days=round(avg_hold, 1),
sample_count=0,
confidence=0.0,
similar_states=0,
)
+72
View File
@@ -0,0 +1,72 @@
"""Signal Instance Store — CRUD for signal_instance table.
Separate from detection logic. Detection finds signals,
store persists them. Both used by backfill pipeline.
"""
from __future__ import annotations
import pandas as pd
from loguru import logger
from ashare_dp.data.store.database import get_db
def store_signals(df: pd.DataFrame) -> int:
"""Store detected signals in signal_instance table.
Uses ON CONFLICT DO NOTHING to safely handle re-runs.
Returns number of rows stored.
"""
if df.empty:
return 0
with get_db(read_only=False) as db:
stored = 0
for _, row in df.iterrows():
try:
db.execute(
"""
INSERT INTO signal_instance
(signal_type, ts_code, trade_date, signal_price,
state_breadth, state_volatility,
return_5d, return_10d, return_20d,
max_return_5d, max_drawdown_5d,
outcome_known)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, TRUE)
ON CONFLICT (signal_type, ts_code, trade_date) DO NOTHING
""",
(
row["signal_type"],
row["ts_code"],
row["trade_date"],
float(row["signal_price"]),
float(row["daily_advance_ratio"]) if pd.notna(row.get("daily_advance_ratio")) else 0.5,
float(row["daily_range"]) if pd.notna(row.get("daily_range")) else 0.02,
float(row["ret_5d"]) if pd.notna(row.get("ret_5d")) else None,
float(row["ret_10d"]) if pd.notna(row.get("ret_10d")) else None,
float(row["ret_20d"]) if pd.notna(row.get("ret_20d")) else None,
float(row["max_return_5d"]) if pd.notna(row.get("max_return_5d")) else None,
float(row["max_dd_5d"]) if pd.notna(row.get("max_dd_5d")) else None,
),
)
stored += 1
except Exception as e:
logger.debug(f"Signal store failed for {row.get('ts_code')}: {e}")
logger.info(f"Stored {stored} signal instances")
return stored
def count_signals(signal_type: str | None = None) -> int:
"""Count stored signal instances, optionally filtered by type."""
with get_db(read_only=True) as db:
if signal_type:
rows = db.query(
"SELECT COUNT(*) FROM signal_instance WHERE signal_type = ?",
(signal_type,),
)
else:
rows = db.query("SELECT COUNT(*) FROM signal_instance")
return int(rows[0][0]) if rows else 0
View File
-61
View File
@@ -1,61 +0,0 @@
"""Database schema: DDL statements for DuckDB tables and views."""
from __future__ import annotations
# DDL for persistent tables
DDL_STATEMENTS = [
"""
CREATE TABLE IF NOT EXISTS stock_info (
ts_code VARCHAR(9) PRIMARY KEY,
symbol VARCHAR(6) NOT NULL,
name VARCHAR(40) NOT NULL,
exchange VARCHAR(2) NOT NULL,
area VARCHAR(20),
industry VARCHAR(40),
list_date DATE,
delist_date DATE,
market VARCHAR(10),
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
""",
"""
CREATE INDEX IF NOT EXISTS idx_stock_symbol ON stock_info(symbol)
""",
"""
CREATE INDEX IF NOT EXISTS idx_stock_exchange ON stock_info(exchange)
""",
"""
CREATE TABLE IF NOT EXISTS trading_calendar (
trade_date DATE PRIMARY KEY,
is_trading_day BOOLEAN NOT NULL DEFAULT TRUE,
week_day TINYINT NOT NULL,
year SMALLINT NOT NULL,
month TINYINT NOT NULL
)
""",
"""
CREATE SEQUENCE IF NOT EXISTS seq_ema52_id
""",
"""
CREATE TABLE IF NOT EXISTS ema52_screening (
id BIGINT PRIMARY KEY DEFAULT nextval('seq_ema52_id'),
trade_date DATE NOT NULL,
ts_code VARCHAR(9) NOT NULL,
name VARCHAR(40),
freq VARCHAR(3) NOT NULL,
close_price DOUBLE NOT NULL,
ema52 DOUBLE NOT NULL,
distance_pct DOUBLE NOT NULL,
amount DOUBLE,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
UNIQUE(trade_date, ts_code, freq)
)
""",
"""
CREATE INDEX IF NOT EXISTS idx_ema52_date ON ema52_screening(trade_date)
""",
# Migration: add amount column if upgrading from older schema
"""
ALTER TABLE ema52_screening ADD COLUMN IF NOT EXISTS amount DOUBLE
""",
]