A_Share_DP/CLAUDE.md

CLAUDE.md - A-Share Data Platform

Project Overview

A-share (Chinese stock market) data platform providing 9 K-line frequencies via Parquet + DuckDB storage with REST and WebSocket APIs. Greenfield project, currently at v0.1.0.

Tech Stack

• Data source: akshare (wraps East Money and Sina APIs)
• Storage: Parquet (Zstd compression, Hive-partitioned year=YYYY/month=MM/day=DD/)
• Query engine: DuckDB (embedded OLAP, read_parquet with hive_partitioning=true, union_by_name=true)
• Web: FastAPI + uvicorn
• CLI: Typer
• Scheduler: APScheduler 3.x (AsyncIOScheduler)
• Config: pydantic-settings (reads .env)
• Logging: loguru

Project Structure

src/ashare_dp/
├── config.py          # Pydantic Settings (all config via env vars)
├── core/
│   ├── models.py      # Freq enum with storage_dir property, freq groupings
│   ├── calendar.py    # Trading calendar, market state, Beijing TZ
│   └── exceptions.py  # AShareDPError hierarchy
├── data/
│   ├── akshare_client.py  # AKShare wrapper: dual backend, retry, rate limit
│   ├── backfill.py        # Historical backfill (daily/weekly/monthly + minute)
│   ├── eod.py             # End-of-day batch pull (all stocks, all freqs)
│   └── realtime.py        # Background async spot poller → WebSocket broadcast
├── storage/
│   ├── database.py        # DuckDB singleton (get_db)
│   ├── schema.py          # DDL: stock_info, trading_calendar
│   ├── repository.py      # KLineRepository: write_klines, read_klines, get_latest
│   └── partitioning.py    # Hive partition path builder
├── api/
│   ├── app.py             # FastAPI factory + lifespan + embedded docs HTML
│   ├── deps.py            # FastAPI DI (get_repo)
│   ├── routers/           # stocks, kline, realtime, calendar routers
│   └── websocket/
│       ├── manager.py     # ConnectionManager: subscribe/broadcast with async lock
│       └── handlers.py    # WS message dispatch (/ws/realtime)
├── scheduler/
│   ├── scheduler.py       # APScheduler setup (EOD 15:05, health check 08:00)
│   └── jobs.py            # Job implementations
└── cli/
    ├── main.py            # Typer root: ashare-dp {backfill, serve, query, version}
    ├── backfill_cmd.py
    ├── serve_cmd.py
    └── query_cmd.py

Critical Design Decisions

Freq.storage_dir property

macOS APFS is case-insensitive, so kline_1m and kline_1M collide. The Freq.M1 (monthly) uses storage_dir = "1mon" to disambiguate. Always use freq.storage_dir for filesystem paths, never freq.value.

Batched writes to avoid file overwrites

write_klines groups all stocks for a day into a single data.parquet file. The backfill and EOD pipelines collect all DataFrames first, then write once per day/freq. Never write per-stock-per-day files.

Dual backend with auto-fallback

AKShareClient._resolve_backend() probes East Money reachability once and caches the result. If unreachable (geo-blocked outside China), falls back to Sina. get_stock_list() and get_trading_calendar() are Sina-only.

SQL parameterization

read_klines() and get_latest() use DuckDB parameterized queries ($1, $2) for user-supplied values (ts_code, dates). Do NOT use f-string interpolation for user input.

2h derivation

2-hour K-lines are derived on-the-fly from 1h data via _read_2h() using pandas resampling. No Parquet storage for 2h.

Key Conventions

• Stock codes: ts_code format is "000001.SZ" (6-digit code + exchange suffix). Internal API calls use 6-digit numeric strings.
• Exchange mapping: codes starting with 6/9 → SH, 4/8/92 → BJ, rest → SZ
• Backend-specific column normalization: Sina returns English columns (date, open, etc.), EM returns Chinese. _normalize_hist_df and _normalize_min_df handle both.
• Proxy: env vars cleared + urllib.request.getproxies monkey-patched at module import time in akshare_client.py
• All async state in ConnectionManager is protected by asyncio.Lock

Running

pip install -e ".[dev]"
ashare-dp backfill init        # First time: create tables, load stock list
ashare-dp backfill daily       # Backfill all daily/weekly/monthly history
ashare-dp serve start          # Start API + scheduler + realtime poller
ashare-dp query stats          # Check data status

Tests

pytest
ruff check src/