v10: Trading OS — 5-subsystem architecture + Signal Intelligence + Dashboard Command Center

Architecture: - Restructure into 5 subsystems: data/, features/, market/, signals/, execution/, apps/ - Unified ts_code conversion in core/codes.py (idempotent, kills 4 duplicate copies) - analytics_conn() + kline_glob() — zero hardcoded DB/Parquet paths - Fixed double-suffix bug (.SZ.SZ) in backfill pipeline root cause Signal Intelligence (the moat): - 14 signal types: EMA52, Vegas, Chan, ORB, Gap, NR7, Inside Bar - 640K+ historical signal instances across 8 backfilled types - Multi-signal Expectancy Engine with breadth-similarity matching - Signal backfill CLI: ashare-dp backfill signals Market Intelligence: - 8 engines: State, Leadership, Opportunity, Flow, Sentiment, Memory, Knowledge Graph, Recommendations - Real limit-up/down sentiment via akshare (108 ZT, 19 DT, 52 broken board) - Knowledge Graph: 8 themes × 30+ concepts with keyword matching - Money-flow stock recommendations with entry/stop/target trade plans Dashboard Command Center: - Decision-first layout: COMMAND → WHERE → WHY → RISK → EXPECTANCY - Multi-signal Expectancy comparison table (8 types ranked by WR) - Theme Map visualization with rotation detection - Intraday Replay infrastructure (30min state snapshots) - RECOMMENDATIONS card with actionable trade plans Trading Memory: - trade_log table + POST/GET/PUT API for trade recording - Performance stats aggregation Code Quality: - 0 hardcoded DB paths, 0 REPLACE hacks, 0 dead ts_code copies - EMA52 screening deduplicated (CLI + scheduler share one function) - read_parquet_sql() helper for 28 duplicate patterns - 6 bugs fixed from code review (NR7 window, theme matching, column indices, etc.) Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-06 12:11:42 +08:00
parent ac5d538a04
commit cc95bbb638
83 changed files with 6328 additions and 728 deletions
@@ -1,94 +1,92 @@
-# CLAUDE.md - A-Share Data Platform
+# CLAUDE.md — A-Share Data Platform (Trading OS)

 ## Project Overview

-A-share (Chinese stock market) data platform providing 9 K-line frequencies via Parquet + DuckDB storage with REST and WebSocket APIs. Greenfield project, currently at v0.1.0.
+A-share trading operating system built on Parquet + DuckDB with REST/WebSocket APIs. Architecture follows a 5-subsystem design: Data Platform → Market Intelligence → Signal Intelligence → Execution Intelligence → Presentation. Dashboard at `/dashboard` renders a Trading Command Center.

 ## Tech Stack

- **Data source**: `akshare` (wraps East Money and Sina APIs)
- **Storage**: Parquet (Zstd compression, Hive-partitioned `year=YYYY/month=MM/day=DD/`)
- **Query engine**: DuckDB (embedded OLAP, `read_parquet` with `hive_partitioning=true, union_by_name=true`)
- **Web**: FastAPI + uvicorn
- **CLI**: Typer
- **Scheduler**: APScheduler 3.x (AsyncIOScheduler)
- **Config**: pydantic-settings (reads `.env`)
- **Logging**: loguru
+- **Data**: `akshare` (East Money / Sina), Parquet (Zstd), DuckDB (analytics)
+- **API**: FastAPI + uvicorn, single `/api/v1/dashboard/state` endpoint + K-line/stock/calendar REST
+- **CLI**: Typer (`ashare-dp backfill daily|minute|industry|signals ...`)
+- **Scheduler**: APScheduler (EOD 15:05, EMA52 15:10)
+- **Config**: pydantic-settings (`.env`)

-## Project Structure
+## Project Structure (v10 — 5 Subsystems)

 ```
 src/ashare_dp/
-├── config.py          # Pydantic Settings (all config via env vars)
-├── core/
-│   ├── models.py      # Freq enum with storage_dir property, freq groupings
-│   ├── calendar.py    # Trading calendar, market state, Beijing TZ
-│   └── exceptions.py  # AShareDPError hierarchy
-├── data/
-│   ├── akshare_client.py  # AKShare wrapper: dual backend, retry, rate limit
-│   ├── backfill.py        # Historical backfill (daily/weekly/monthly + minute)
-│   ├── eod.py             # End-of-day batch pull (all stocks, all freqs)
-│   └── realtime.py        # Background async spot poller → WebSocket broadcast
-├── storage/
-│   ├── database.py        # DuckDB singleton (get_db)
-│   ├── schema.py          # DDL: stock_info, trading_calendar
-│   ├── repository.py      # KLineRepository: write_klines, read_klines, get_latest
-│   └── partitioning.py    # Hive partition path builder
-├── api/
-│   ├── app.py             # FastAPI factory + lifespan + embedded docs HTML
-│   ├── deps.py            # FastAPI DI (get_repo)
-│   ├── routers/           # stocks, kline, realtime, calendar routers
-│   └── websocket/
-│       ├── manager.py     # ConnectionManager: subscribe/broadcast with async lock
-│       └── handlers.py    # WS message dispatch (/ws/realtime)
-├── scheduler/
-│   ├── scheduler.py       # APScheduler setup (EOD 15:05, health check 08:00)
-│   └── jobs.py            # Job implementations
-└── cli/
-    ├── main.py            # Typer root: ashare-dp {backfill, serve, query, version}
-    ├── backfill_cmd.py
-    ├── serve_cmd.py
-    └── query_cmd.py
+├── config.py           # Settings
+├── core/               # Shared kernel
+│   ├── models.py       # Freq enum, INDEX_CODES, INDEX_SINA_SYMBOLS
+│   ├── codes.py        # ★ Single ts_code conversion module (IDEMPOTENT)
+│   ├── calendar.py     # Trading calendar, market state, Beijing TZ
+│   └── exceptions.py
+├── domain/             # Ontology — shared contracts
+│   ├── state.py        # MarketState (7-dim continuous vector)
+│   ├── context.py      # TradingContext, Playbook, Expectancy, Opportunity
+│   ├── events.py       # RiskEvent
+│   ├── features.py     # FeatureDefinition
+│   ├── leadership.py   # LeaderState enum
+│   └── signal.py       # SignalType, SignalInstance
+├── data/               # ═══ DATA PLATFORM ═══
+│   ├── sources/        # akshare_client, index, industry
+│   ├── pipelines/      # backfill, eod, realtime
+│   └── store/          # database (get_db, analytics_conn, kline_glob), repository, partitioning, schema
+├── features/           # Feature Store (6 registered features, all use analytics_conn + kline_glob)
+├── market/             # ═══ MARKET INTELLIGENCE ═══
+│   ├── state.py        # infer_market_state
+│   ├── leadership.py   # assess_leaders (lifecycle per industry)
+│   ├── opportunity.py  # rank_opportunities
+│   ├── flow.py         # compute_flow (money flow graph)
+│   ├── sentiment.py    # Phase 2 placeholder
+│   └── memory.py       # StateStore (state_snapshot table)
+├── signals/            # ═══ SIGNAL INTELLIGENCE ═══ (the moat)
+│   ├── detectors.py    # EMA52 cross detection + shared screening logic
+│   ├── store.py        # signal_instance CRUD (to be extracted from detectors)
+│   └── expectancy.py   # get_expectancy(state) — single entry, fallback chain internal
+├── execution/          # ═══ EXECUTION INTELLIGENCE ═══
+│   ├── playbook.py     # build_playbook (State → strategies/bias/holding)
+│   ├── risk.py         # RiskRule engine + evaluate_risks
+│   └── brief.py        # build_brief + brief_to_api_dict (single serialization point)
+└── apps/               # ═══ PRESENTATION ═══
+    ├── api/            # app.py, routers/, websocket/, dashboard/
+    ├── cli/            # main.py + backfill/serve/query/eod/screening commands
+    └── scheduler/      # scheduler.py, jobs.py
 ```

-## Critical Design Decisions
-
-### Freq.storage_dir property
-macOS APFS is case-insensitive, so `kline_1m` and `kline_1M` collide. The `Freq.M1` (monthly) uses `storage_dir = "1mon"` to disambiguate. Always use `freq.storage_dir` for filesystem paths, never `freq.value`.
-
-### Batched writes to avoid file overwrites
-`write_klines` groups all stocks for a day into a single `data.parquet` file. The backfill and EOD pipelines collect all DataFrames first, then write once per day/freq. Never write per-stock-per-day files.
-
-### Dual backend with auto-fallback
-`AKShareClient._resolve_backend()` probes East Money reachability once and caches the result. If unreachable (geo-blocked outside China), falls back to Sina. `get_stock_list()` and `get_trading_calendar()` are Sina-only.
-
-### SQL parameterization
-`read_klines()` and `get_latest()` use DuckDB parameterized queries (`$1`, `$2`) for user-supplied values (ts_code, dates). Do NOT use f-string interpolation for user input.
-
-### 2h derivation
-2-hour K-lines are derived on-the-fly from 1h data via `_read_2h()` using pandas resampling. No Parquet storage for 2h.
-
 ## Key Conventions

- Stock codes: `ts_code` format is `"000001.SZ"` (6-digit code + exchange suffix). Internal API calls use 6-digit numeric strings.
- Exchange mapping: codes starting with `6`/`9` → SH, `4`/`8`/`92` → BJ, rest → SZ
- Backend-specific column normalization: Sina returns English columns (`date`, `open`, etc.), EM returns Chinese. `_normalize_hist_df` and `_normalize_min_df` handle both.
- Proxy: env vars cleared + `urllib.request.getproxies` monkey-patched at module import time in `akshare_client.py`
- All async state in `ConnectionManager` is protected by `asyncio.Lock`
+### ts_code conversion (CRITICAL)
+**Always use `from ashare_dp.core.codes import to_ts_code`** — the single idempotent implementation. Never write local `_code_to_ts_code()` copies. `to_ts_code()` is safe to call on any format: bare codes, already-formatted ts_codes, Sina symbols, even legacy corrupted `.SZ.SZ` values.
+
+### Database connections
+- **DuckDB tables** (stock_info, trading_calendar, signal_instance): use `get_db()` context manager from `data.store.database`
+- **Analytics queries** (features, engines): use `analytics_conn()` for raw DuckDB connection — the single sanctioned way. Never hardcode `"data/duckdb/ashare.db"`
+- **Parquet globs**: use `kline_glob()` or `partition_glob(freq)` — never hardcode paths
+
+### Architecture boundaries
+- `data/` knows nothing about trading
+- `features/` computes features, never classifies regimes
+- `market/` infers state, knows nothing about signals
+- `signals/` queries historical expectancy, knows nothing about execution
+- `execution/` maps state to strategies, assembles TradingBrief
+- `apps/` only renders, never reasons
+
+### MarketState is a continuous vector (not enum)
+7 dimensions: trend, fear, liquidity, rotation, participation, volatility, breadth. Each 0.0–1.0. Display labels derived downstream only.
+
+### API Response
+Single endpoint produces all dashboard data: `GET /api/v1/dashboard/state`. Response versioned (`"version": "1.0"`). Serialization in `execution/brief.py::brief_to_api_dict()` — the single serialization point. Router only orchestrates engine calls.

 ## Running

 ```bash
 pip install -e ".[dev]"
-ashare-dp backfill init        # First time: create tables, load stock list
-ashare-dp backfill daily       # Backfill all daily/weekly/monthly history
-ashare-dp serve start          # Start API + scheduler + realtime poller
-ashare-dp query stats          # Check data status
-```
-
-## Tests
-
-```bash
-pytest
-ruff check src/
+ashare-dp backfill init        # Schema + stock list + trading calendar
+ashare-dp backfill daily       # Daily/weekly/monthly + indices
+ashare-dp backfill industry    # Industry classifications
+ashare-dp backfill signals     # EMA52 signal detection + store
+ashare-dp serve start          # API + scheduler + realtime
+open http://localhost:8000/dashboard
 ```