jackyu66gitandClaude Opus 4.6
21378c4f6d
Parquet + DuckDB storage with REST/WebSocket APIs for Chinese A-share market data. Supports 9 K-line frequencies with dual backend (East Money / Sina) and auto-fallback. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
95 lines
4.5 KiB
Markdown
95 lines
4.5 KiB
Markdown
# CLAUDE.md - A-Share Data Platform
|
|
|
|
## Project Overview
|
|
|
|
A-share (Chinese stock market) data platform providing 9 K-line frequencies via Parquet + DuckDB storage with REST and WebSocket APIs. Greenfield project, currently at v0.1.0.
|
|
|
|
## Tech Stack
|
|
|
|
- **Data source**: `akshare` (wraps East Money and Sina APIs)
|
|
- **Storage**: Parquet (Zstd compression, Hive-partitioned `year=YYYY/month=MM/day=DD/`)
|
|
- **Query engine**: DuckDB (embedded OLAP, `read_parquet` with `hive_partitioning=true, union_by_name=true`)
|
|
- **Web**: FastAPI + uvicorn
|
|
- **CLI**: Typer
|
|
- **Scheduler**: APScheduler 3.x (AsyncIOScheduler)
|
|
- **Config**: pydantic-settings (reads `.env`)
|
|
- **Logging**: loguru
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
src/ashare_dp/
|
|
├── config.py # Pydantic Settings (all config via env vars)
|
|
├── core/
|
|
│ ├── models.py # Freq enum with storage_dir property, freq groupings
|
|
│ ├── calendar.py # Trading calendar, market state, Beijing TZ
|
|
│ └── exceptions.py # AShareDPError hierarchy
|
|
├── data/
|
|
│ ├── akshare_client.py # AKShare wrapper: dual backend, retry, rate limit
|
|
│ ├── backfill.py # Historical backfill (daily/weekly/monthly + minute)
|
|
│ ├── eod.py # End-of-day batch pull (all stocks, all freqs)
|
|
│ └── realtime.py # Background async spot poller → WebSocket broadcast
|
|
├── storage/
|
|
│ ├── database.py # DuckDB singleton (get_db)
|
|
│ ├── schema.py # DDL: stock_info, trading_calendar
|
|
│ ├── repository.py # KLineRepository: write_klines, read_klines, get_latest
|
|
│ └── partitioning.py # Hive partition path builder
|
|
├── api/
|
|
│ ├── app.py # FastAPI factory + lifespan + embedded docs HTML
|
|
│ ├── deps.py # FastAPI DI (get_repo)
|
|
│ ├── routers/ # stocks, kline, realtime, calendar routers
|
|
│ └── websocket/
|
|
│ ├── manager.py # ConnectionManager: subscribe/broadcast with async lock
|
|
│ └── handlers.py # WS message dispatch (/ws/realtime)
|
|
├── scheduler/
|
|
│ ├── scheduler.py # APScheduler setup (EOD 15:05, health check 08:00)
|
|
│ └── jobs.py # Job implementations
|
|
└── cli/
|
|
├── main.py # Typer root: ashare-dp {backfill, serve, query, version}
|
|
├── backfill_cmd.py
|
|
├── serve_cmd.py
|
|
└── query_cmd.py
|
|
```
|
|
|
|
## Critical Design Decisions
|
|
|
|
### Freq.storage_dir property
|
|
macOS APFS is case-insensitive, so `kline_1m` and `kline_1M` collide. The `Freq.M1` (monthly) uses `storage_dir = "1mon"` to disambiguate. Always use `freq.storage_dir` for filesystem paths, never `freq.value`.
|
|
|
|
### Batched writes to avoid file overwrites
|
|
`write_klines` groups all stocks for a day into a single `data.parquet` file. The backfill and EOD pipelines collect all DataFrames first, then write once per day/freq. Never write per-stock-per-day files.
|
|
|
|
### Dual backend with auto-fallback
|
|
`AKShareClient._resolve_backend()` probes East Money reachability once and caches the result. If unreachable (geo-blocked outside China), falls back to Sina. `get_stock_list()` and `get_trading_calendar()` are Sina-only.
|
|
|
|
### SQL parameterization
|
|
`read_klines()` and `get_latest()` use DuckDB parameterized queries (`$1`, `$2`) for user-supplied values (ts_code, dates). Do NOT use f-string interpolation for user input.
|
|
|
|
### 2h derivation
|
|
2-hour K-lines are derived on-the-fly from 1h data via `_read_2h()` using pandas resampling. No Parquet storage for 2h.
|
|
|
|
## Key Conventions
|
|
|
|
- Stock codes: `ts_code` format is `"000001.SZ"` (6-digit code + exchange suffix). Internal API calls use 6-digit numeric strings.
|
|
- Exchange mapping: codes starting with `6`/`9` → SH, `4`/`8`/`92` → BJ, rest → SZ
|
|
- Backend-specific column normalization: Sina returns English columns (`date`, `open`, etc.), EM returns Chinese. `_normalize_hist_df` and `_normalize_min_df` handle both.
|
|
- Proxy: env vars cleared + `urllib.request.getproxies` monkey-patched at module import time in `akshare_client.py`
|
|
- All async state in `ConnectionManager` is protected by `asyncio.Lock`
|
|
|
|
## Running
|
|
|
|
```bash
|
|
pip install -e ".[dev]"
|
|
ashare-dp backfill init # First time: create tables, load stock list
|
|
ashare-dp backfill daily # Backfill all daily/weekly/monthly history
|
|
ashare-dp serve start # Start API + scheduler + realtime poller
|
|
ashare-dp query stats # Check data status
|
|
```
|
|
|
|
## Tests
|
|
|
|
```bash
|
|
pytest
|
|
ruff check src/
|
|
```
|