Initial commit: A-Share Data Platform v0.1.0

Parquet + DuckDB storage with REST/WebSocket APIs for Chinese A-share market data. Supports 9 K-line frequencies with dual backend (East Money / Sina) and auto-fallback. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-18 16:50:17 +08:00
parent a11e1ceee1
commit 21378c4f6d
40 changed files with 3737 additions and 213 deletions
@@ -0,0 +1,94 @@
+# CLAUDE.md - A-Share Data Platform
+
+## Project Overview
+
+A-share (Chinese stock market) data platform providing 9 K-line frequencies via Parquet + DuckDB storage with REST and WebSocket APIs. Greenfield project, currently at v0.1.0.
+
+## Tech Stack
+
+- **Data source**: `akshare` (wraps East Money and Sina APIs)
+- **Storage**: Parquet (Zstd compression, Hive-partitioned `year=YYYY/month=MM/day=DD/`)
+- **Query engine**: DuckDB (embedded OLAP, `read_parquet` with `hive_partitioning=true, union_by_name=true`)
+- **Web**: FastAPI + uvicorn
+- **CLI**: Typer
+- **Scheduler**: APScheduler 3.x (AsyncIOScheduler)
+- **Config**: pydantic-settings (reads `.env`)
+- **Logging**: loguru
+
+## Project Structure
+
+```
+src/ashare_dp/
+├── config.py          # Pydantic Settings (all config via env vars)
+├── core/
+│   ├── models.py      # Freq enum with storage_dir property, freq groupings
+│   ├── calendar.py    # Trading calendar, market state, Beijing TZ
+│   └── exceptions.py  # AShareDPError hierarchy
+├── data/
+│   ├── akshare_client.py  # AKShare wrapper: dual backend, retry, rate limit
+│   ├── backfill.py        # Historical backfill (daily/weekly/monthly + minute)
+│   ├── eod.py             # End-of-day batch pull (all stocks, all freqs)
+│   └── realtime.py        # Background async spot poller → WebSocket broadcast
+├── storage/
+│   ├── database.py        # DuckDB singleton (get_db)
+│   ├── schema.py          # DDL: stock_info, trading_calendar
+│   ├── repository.py      # KLineRepository: write_klines, read_klines, get_latest
+│   └── partitioning.py    # Hive partition path builder
+├── api/
+│   ├── app.py             # FastAPI factory + lifespan + embedded docs HTML
+│   ├── deps.py            # FastAPI DI (get_repo)
+│   ├── routers/           # stocks, kline, realtime, calendar routers
+│   └── websocket/
+│       ├── manager.py     # ConnectionManager: subscribe/broadcast with async lock
+│       └── handlers.py    # WS message dispatch (/ws/realtime)
+├── scheduler/
+│   ├── scheduler.py       # APScheduler setup (EOD 15:05, health check 08:00)
+│   └── jobs.py            # Job implementations
+└── cli/
+    ├── main.py            # Typer root: ashare-dp {backfill, serve, query, version}
+    ├── backfill_cmd.py
+    ├── serve_cmd.py
+    └── query_cmd.py
+```
+
+## Critical Design Decisions
+
+### Freq.storage_dir property
+macOS APFS is case-insensitive, so `kline_1m` and `kline_1M` collide. The `Freq.M1` (monthly) uses `storage_dir = "1mon"` to disambiguate. Always use `freq.storage_dir` for filesystem paths, never `freq.value`.
+
+### Batched writes to avoid file overwrites
+`write_klines` groups all stocks for a day into a single `data.parquet` file. The backfill and EOD pipelines collect all DataFrames first, then write once per day/freq. Never write per-stock-per-day files.
+
+### Dual backend with auto-fallback
+`AKShareClient._resolve_backend()` probes East Money reachability once and caches the result. If unreachable (geo-blocked outside China), falls back to Sina. `get_stock_list()` and `get_trading_calendar()` are Sina-only.
+
+### SQL parameterization
+`read_klines()` and `get_latest()` use DuckDB parameterized queries (`$1`, `$2`) for user-supplied values (ts_code, dates). Do NOT use f-string interpolation for user input.
+
+### 2h derivation
+2-hour K-lines are derived on-the-fly from 1h data via `_read_2h()` using pandas resampling. No Parquet storage for 2h.
+
+## Key Conventions
+
+- Stock codes: `ts_code` format is `"000001.SZ"` (6-digit code + exchange suffix). Internal API calls use 6-digit numeric strings.
+- Exchange mapping: codes starting with `6`/`9` → SH, `4`/`8`/`92` → BJ, rest → SZ
+- Backend-specific column normalization: Sina returns English columns (`date`, `open`, etc.), EM returns Chinese. `_normalize_hist_df` and `_normalize_min_df` handle both.
+- Proxy: env vars cleared + `urllib.request.getproxies` monkey-patched at module import time in `akshare_client.py`
+- All async state in `ConnectionManager` is protected by `asyncio.Lock`
+
+## Running
+
+```bash
+pip install -e ".[dev]"
+ashare-dp backfill init        # First time: create tables, load stock list
+ashare-dp backfill daily       # Backfill all daily/weekly/monthly history
+ashare-dp serve start          # Start API + scheduler + realtime poller
+ashare-dp query stats          # Check data status
+```
+
+## Tests
+
+```bash
+pytest
+ruff check src/
+```