添加本地数据源，以后就可以直接用本地数据了
This commit is contained in:
jackyu66git
@@ -10,6 +10,15 @@ RUN pip install -r /app/requirements.txt
|
||||
|
||||
COPY app /app/app
|
||||
|
||||
ENV DATA_DIR=/data \
|
||||
EXCHANGE=binance \
|
||||
SYMBOLS=BTC/USDT:USDT,ETH/USDT:USDT \
|
||||
TIMEFRAMES=1m,1h,1d,1w,1M \
|
||||
START_FROM=2022-01-01 \
|
||||
POLL_FACTOR=0.5
|
||||
|
||||
VOLUME ["/data"]
|
||||
|
||||
EXPOSE 9000
|
||||
|
||||
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "9000"]
|
||||
|
||||
+127
-34
@@ -1,48 +1,141 @@
|
||||
# Local Data Service (REST + WebSocket)
|
||||
# Local Data Service（REST + WebSocket）
|
||||
|
||||
一键部署、跨平台的本地行情数据服务。默认抓取 Binance 永续合约 `BTC/USDT:USDT, ETH/USDT:USDT` 的 `1m/5m/15m/1h` K 线，增量写入本地 Parquet 并通过 WebSocket 推送。
|
||||
本服务基于 FastAPI + ccxt，自动拉取交易所行情、写入本地 Parquet，同时提供 REST 和 WebSocket 数据访问。
|
||||
自带时间周期聚合能力：只需抓取 `1m / 1h / 1d / 1w / 1M` 等基础周期，即可自动生成 `2m/3m/.../30m`、`2h/3h/.../16h` 等衍生周期。
|
||||
|
||||
## 快速开始（方式B：已安装 Docker）
|
||||
---
|
||||
|
||||
## 1. 环境准备
|
||||
|
||||
### 1.1 依赖
|
||||
- Python ≥ 3.10（本地运行方式需要）
|
||||
- `pip install -r requirements.txt`（包含 `fastapi`, `uvicorn`, `ccxt`, `pandas`, `pyarrow`, `technical` 等）
|
||||
- 或者直接使用仓库内的 `docker-compose.yml`
|
||||
|
||||
### 1.2 关键环境变量
|
||||
| 变量 | 说明 | 默认 |
|
||||
| --- | --- | --- |
|
||||
| `DATA_DIR` | 本地 Parquet 存储目录 | `/data` |
|
||||
| `EXCHANGE` | 交易所标识（目前支持 binance） | `binance` |
|
||||
| `SYMBOLS` | 逗号分隔的交易对列表 | `BTC/USDT:USDT,ETH/USDT:USDT` |
|
||||
| `TIMEFRAMES` | 基础抓取周期，逗号分隔 | `1m,1h,1d,1w,1M` |
|
||||
| `START_FROM` | 首次启动回补的起始 UTC 时间（ISO 字符串或毫秒时间戳） | `2022-01-01` |
|
||||
| `POLL_FACTOR` | 拉取间隔因子，实际间隔 = 周期毫秒 × factor | `0.5` |
|
||||
| `BACKOFF_BASE / BACKOFF_MAX` | 异常重试的指数退避参数 | `2.0 / 30.0` |
|
||||
|
||||
> 衍生周期列表由程序自动推导，无需手动写入 `TIMEFRAMES`。
|
||||
|
||||
---
|
||||
|
||||
## 2. 启动与关闭
|
||||
|
||||
### 2.1 Docker 方式
|
||||
```bash
|
||||
cd user_data/Chan/datasvc
|
||||
docker compose up -d
|
||||
docker compose up -d # 启动
|
||||
docker compose logs -f # 查看日志
|
||||
docker compose down # 关闭
|
||||
```
|
||||
|
||||
- REST: http://localhost:9000/api/candles?symbol=BTC/USDT:USDT&tf=1m
|
||||
- WS: ws://localhost:9000/ws?symbol=BTC/USDT:USDT&tf=1m&since=1690000000000
|
||||
- Swagger: http://localhost:9000/docs
|
||||
|
||||
## 环境变量（docker-compose.yml）
|
||||
- EXCHANGE: 交易所，默认 binance
|
||||
- SYMBOLS: 逗号分隔交易对
|
||||
- TIMEFRAMES: 逗号分隔周期
|
||||
- START_DAYS: 首次启动回补最近 N 天
|
||||
- POLL_FACTOR: 轮询因子，间隔=周期毫秒*factor
|
||||
- DATA_DIR: 容器内数据目录（已映射到 `./data`）
|
||||
|
||||
## 数据位置
|
||||
- 本地缓存：`user_data/Chan/datasvc/data/{timeframe}/{symbol}.parquet`
|
||||
|
||||
## 常用命令
|
||||
### 2.2 本地运行（无 Docker）
|
||||
```bash
|
||||
docker compose logs -f
|
||||
export DATA_DIR=./data
|
||||
export SYMBOLS="BTC/USDT:USDT"
|
||||
export TIMEFRAMES="1m,1h,1d"
|
||||
|
||||
docker compose down
|
||||
cd /Users/jack/Project/freqtrade
|
||||
uvicorn user_data.Chan.datasvc.app.main:app --reload
|
||||
```
|
||||
|
||||
## 接口说明
|
||||
- GET /api/candles
|
||||
- 参数：symbol, tf, start(ms), end(ms)
|
||||
- 返回：[{timestamp, open, high, low, close, volume}]
|
||||
- WS /ws
|
||||
- 参数：symbol, tf, since(ms)
|
||||
- 消息：
|
||||
- snapshot: 初始快照数组
|
||||
- upsert: 单根K线增量（尾部修正）
|
||||
关闭时 Ctrl+C 即可，服务会自动取消后台抓取任务并释放资源。
|
||||
|
||||
## 注意
|
||||
- 默认未带交易所 API Key，仅公共行情。
|
||||
- 如需更多交易对/周期，修改 `docker-compose.yml` 后重启。
|
||||
---
|
||||
|
||||
## 3. 数据存储与聚合
|
||||
|
||||
### 3.1 基础周期
|
||||
只会为 `TIMEFRAMES` 声明的基础周期创建抓取任务（例如 `1m / 1h / 1d`）。
|
||||
|
||||
### 3.2 衍生周期
|
||||
启动后自动维护以下聚合：
|
||||
|
||||
| 基础周期 | 自动生成 |
|
||||
| --- | --- |
|
||||
| `1m` | `2m, 3m, 4m, 5m, 10m, 15m, 20m, 25m, 30m` |
|
||||
| `1h` | `2h, 3h, 4h, 6h, 8h, 12h, 16h` |
|
||||
| `1d` | `2d, 3d, 4d, 5d, 6d` |
|
||||
| `1w` | `2w` |
|
||||
| `1M` | `2M, 3M, 6M` |
|
||||
|
||||
聚合过程通过 `technical.util.resample_to_interval` 完成，写入同一 Parquet 数据目录。
|
||||
所有周期都可以被 REST/WS 访问。
|
||||
|
||||
### 3.3 数据目录
|
||||
```
|
||||
{DATA_DIR}/{timeframe}/{symbol}.parquet
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. 接口调用
|
||||
|
||||
### 4.1 健康检查
|
||||
```
|
||||
GET /health
|
||||
```
|
||||
返回运行状态、基础/衍生周期列表、各抓取任务的最新进度与错误计数，便于监控。
|
||||
|
||||
### 4.2 REST API
|
||||
```
|
||||
GET /api/candles?symbol=BTC/USDT:USDT&tf=2h&start=1700000000000&end=1700003600000
|
||||
```
|
||||
参数说明：
|
||||
- `symbol`：交易对（必须在 `SYMBOLS` 列表中）
|
||||
- `tf`：时间周期（支持基础或衍生）
|
||||
- `start` / `end`：毫秒时间戳，可选
|
||||
|
||||
返回示例：
|
||||
```json
|
||||
[
|
||||
{"timestamp": 1700000000000, "open": 36000.0, "high": 36120.0, "low": 35980.0, "close": 36050.0, "volume": 125.4},
|
||||
...
|
||||
]
|
||||
```
|
||||
|
||||
### 4.3 WebSocket
|
||||
```
|
||||
ws://localhost:8000/ws?symbol=ETH/USDT:USDT&tf=15m&since=1700000000000
|
||||
```
|
||||
- 首次连接：收到 `snapshot` 消息（快照数组）
|
||||
- 后续增量：收到 `upsert` 消息（最新几根K线），以及周期性 `ping`
|
||||
|
||||
消息示例：
|
||||
```json
|
||||
{"topic":"candles.ETH/USDT:USDT.15m","type":"snapshot","data":[{"t":1700000000000,"o":2000.0,"h":2005.0,"l":1995.0,"c":2002.5,"v":312.7}, ...]}
|
||||
{"topic":"candles.ETH/USDT:USDT.15m","type":"upsert","data":{"t":1700000900000,"o":2002.5,"h":2006.0,"l":2000.0,"c":2004.0,"v":120.8}}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 停机与维护
|
||||
|
||||
- **正常关闭**：`docker compose down` 或 Ctrl+C。服务会等待所有抓取任务结束并关闭 `ccxt` 客户端。
|
||||
- **异常恢复**：若网络异常，服务会自动指数退避重试；可通过 `/health` 的 `consecutive_errors` 与 `last_error` 排查。
|
||||
- **数据清理**：直接删除 `DATA_DIR` 下对应的 Parquet 文件即可，下次启动会重新回补。
|
||||
|
||||
---
|
||||
|
||||
## 6. 常见问题
|
||||
|
||||
1. **缺少 `technical` 模块**
|
||||
聚合周期会跳过，并在日志中提示；先执行 `pip install technical` 再重启。
|
||||
|
||||
2. **接收不到某个周期的数据**
|
||||
确认该周期在 `TIMEFRAMES` 或自动聚合列表中；若是衍生周期，需要确保对应基础周期已在运行。
|
||||
|
||||
3. **如何新增交易对/周期**
|
||||
修改环境变量或 docker-compose 配置后，重启服务即可；Parquet 文件会按需生成。
|
||||
|
||||
---
|
||||
|
||||
欢迎结合自身策略或可视化前端直接消费本地数据服务。若要集成到其他项目，可直接引用 `/api/candles` 的 JSON 响应或订阅 `/ws` 的实时推送。
|
||||
|
||||
+361
-46
@@ -1,15 +1,24 @@
|
||||
import os
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
from contextlib import suppress
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Dict, List, Optional
|
||||
from typing import Dict, List, Optional, Tuple, Union
|
||||
|
||||
import ccxt
|
||||
import ccxt.async_support as ccxt_async
|
||||
import pandas as pd
|
||||
from fastapi import FastAPI, WebSocket, WebSocketDisconnect, Query
|
||||
from fastapi import FastAPI, WebSocket, WebSocketDisconnect, Query, HTTPException, status
|
||||
from fastapi.responses import JSONResponse
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
try:
|
||||
from technical.util import resample_to_interval # type: ignore
|
||||
except ImportError: # pragma: no cover - 环境缺失依赖时自动降级
|
||||
resample_to_interval = None # type: ignore
|
||||
|
||||
from .storage import (
|
||||
ensure_storage,
|
||||
read_candles,
|
||||
@@ -18,12 +27,86 @@ from .storage import (
|
||||
)
|
||||
|
||||
|
||||
LOG_LEVEL = os.environ.get("LOG_LEVEL", "INFO").upper()
|
||||
logging.basicConfig(
|
||||
level=LOG_LEVEL,
|
||||
format="%(asctime)s %(levelname)s [%(name)s] %(message)s",
|
||||
)
|
||||
logger = logging.getLogger("datasvc")
|
||||
RESAMPLE_AVAILABLE = resample_to_interval is not None
|
||||
RESAMPLE_WARNING_EMITTED = False
|
||||
|
||||
AGGREGATION_PLAN: Dict[str, List[str]] = {
|
||||
"1m": ["2m", "3m", "4m", "5m", "10m", "15m", "20m", "25m", "30m"],
|
||||
"1h": ["2h", "3h", "4h", "6h", "8h", "12h", "16h"],
|
||||
"1d": ["2d", "3d", "4d", "5d", "6d"],
|
||||
"1w": ["2w"],
|
||||
"1M": ["2M", "3M", "6M"],
|
||||
}
|
||||
|
||||
CandleRow = List[Union[int, float]]
|
||||
|
||||
|
||||
def _split_env_list(value: str) -> List[str]:
|
||||
return [item.strip() for item in value.split(",") if item.strip()]
|
||||
|
||||
|
||||
def _unique_preserve(values: List[str]) -> List[str]:
|
||||
seen = set()
|
||||
ordered: List[str] = []
|
||||
for item in values:
|
||||
if item not in seen:
|
||||
ordered.append(item)
|
||||
seen.add(item)
|
||||
return ordered
|
||||
|
||||
|
||||
def timeframe_to_minutes(tf: str) -> Optional[int]:
|
||||
if not tf:
|
||||
return None
|
||||
unit = tf[-1]
|
||||
try:
|
||||
value = int(tf[:-1])
|
||||
except ValueError:
|
||||
return None
|
||||
multiplier = {
|
||||
"m": 1,
|
||||
"h": 60,
|
||||
"d": 1440,
|
||||
"w": 10080,
|
||||
"M": 43200, # 30 天近似
|
||||
}.get(unit)
|
||||
if multiplier is None:
|
||||
return None
|
||||
return value * multiplier
|
||||
|
||||
DATA_DIR = os.environ.get("DATA_DIR", "/data")
|
||||
EXCHANGE = os.environ.get("EXCHANGE", "binance")
|
||||
SYMBOLS = [s.strip() for s in os.environ.get("SYMBOLS", "BTC/USDT:USDT,ETH/USDT:USDT").split(",") if s.strip()]
|
||||
TIMEFRAMES = [t.strip() for t in os.environ.get("TIMEFRAMES", "1m,5m,15m,1h").split(",") if t.strip()]
|
||||
START_FROM = os.environ.get("START_FROM", "2025-01-01") # 首次启动拉取起始日期（UTC）
|
||||
SYMBOLS = _split_env_list(os.environ.get("SYMBOLS", "BTC/USDT:USDT,ETH/USDT:USDT"))
|
||||
if not SYMBOLS:
|
||||
SYMBOLS = ["BTC/USDT:USDT"]
|
||||
|
||||
_default_timeframes = ["1m", "1h", "1d", "1w", "1M"]
|
||||
requested_timeframes = _split_env_list(os.environ.get("TIMEFRAMES", ",".join(_default_timeframes)))
|
||||
if not requested_timeframes:
|
||||
requested_timeframes = _default_timeframes
|
||||
|
||||
FETCH_TIMEFRAMES = _unique_preserve(requested_timeframes)
|
||||
AVAILABLE_TIMEFRAMES = list(FETCH_TIMEFRAMES)
|
||||
for base_tf in FETCH_TIMEFRAMES:
|
||||
for derived_tf in AGGREGATION_PLAN.get(base_tf, []):
|
||||
if derived_tf not in AVAILABLE_TIMEFRAMES:
|
||||
AVAILABLE_TIMEFRAMES.append(derived_tf)
|
||||
DERIVED_TIMEFRAMES = [tf for tf in AVAILABLE_TIMEFRAMES if tf not in FETCH_TIMEFRAMES]
|
||||
AGGREGATION_TARGETS = {tf: AGGREGATION_PLAN.get(tf, []) for tf in FETCH_TIMEFRAMES}
|
||||
|
||||
START_FROM = os.environ.get("START_FROM", "2022-01-01") # 首次启动拉取起始日期（UTC）
|
||||
POLL_FACTOR = float(os.environ.get("POLL_FACTOR", "0.5")) # 轮询间隔 = tf_ms * factor
|
||||
BACKOFF_BASE = float(os.environ.get("BACKOFF_BASE", "2.0"))
|
||||
BACKOFF_MAX = float(os.environ.get("BACKOFF_MAX", "30.0"))
|
||||
|
||||
VALID_SYMBOLS = set(SYMBOLS)
|
||||
VALID_TIMEFRAMES = set(AVAILABLE_TIMEFRAMES)
|
||||
|
||||
ensure_storage(DATA_DIR)
|
||||
|
||||
@@ -38,18 +121,24 @@ app.add_middleware(
|
||||
|
||||
|
||||
def tf_to_ms(tf: str) -> int:
|
||||
table = {
|
||||
"1m": 60_000,
|
||||
"3m": 3 * 60_000,
|
||||
"5m": 5 * 60_000,
|
||||
"15m": 15 * 60_000,
|
||||
"30m": 30 * 60_000,
|
||||
"1h": 60 * 60_000,
|
||||
"2h": 2 * 60 * 60_000,
|
||||
"4h": 4 * 60 * 60_000,
|
||||
"1d": 24 * 60 * 60_000,
|
||||
}
|
||||
return table.get(tf, 60_000)
|
||||
minutes = timeframe_to_minutes(tf)
|
||||
if minutes is None:
|
||||
logger.warning("无法解析时间周期，默认使用 60 秒", extra={"timeframe": tf})
|
||||
return 60_000
|
||||
return minutes * 60_000
|
||||
|
||||
|
||||
def ensure_symbol_timeframe(symbol: str, timeframe: str) -> None:
|
||||
if symbol not in VALID_SYMBOLS:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=f"symbol 必须为 {sorted(VALID_SYMBOLS)} 之一。",
|
||||
)
|
||||
if timeframe not in VALID_TIMEFRAMES:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=f"tf 必须为 {sorted(VALID_TIMEFRAMES)} 之一。",
|
||||
)
|
||||
|
||||
|
||||
def parse_start_from_ms(val: str) -> int:
|
||||
@@ -63,10 +152,10 @@ def parse_start_from_ms(val: str) -> int:
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
dt = datetime.fromisoformat(val) # 允许 '2025-01-01' 或 '2025-01-01T00:00:00'
|
||||
dt = datetime.fromisoformat(val) # 允许 '2022-01-01' 或 '2022-01-01T00:00:00'
|
||||
except Exception:
|
||||
# 回退到固定日期
|
||||
dt = datetime(2025, 1, 1)
|
||||
dt = datetime(2022, 1, 1)
|
||||
return int(dt.timestamp() * 1000)
|
||||
|
||||
|
||||
@@ -104,52 +193,272 @@ class Hub:
|
||||
|
||||
hub = Hub()
|
||||
|
||||
fetch_tasks: List[asyncio.Task] = []
|
||||
|
||||
|
||||
def resample_and_store(symbol: str, base_timeframe: str, derived_timeframes: List[str]) -> List[Tuple[str, List[CandleRow]]]:
|
||||
if not RESAMPLE_AVAILABLE or not derived_timeframes:
|
||||
return []
|
||||
base_df = read_candles(DATA_DIR, symbol, base_timeframe, None, None)
|
||||
if base_df.empty:
|
||||
return []
|
||||
base_df = base_df.copy()
|
||||
if "date" not in base_df.columns:
|
||||
base_df["date"] = pd.to_datetime(base_df["timestamp"], unit="ms", utc=True)
|
||||
base_df = base_df.sort_values("timestamp")
|
||||
|
||||
updates: List[Tuple[str, List[List[float]]]] = []
|
||||
for target_tf in derived_timeframes:
|
||||
minutes = timeframe_to_minutes(target_tf)
|
||||
if minutes is None:
|
||||
logger.warning("无法解析聚合周期", extra={"target_timeframe": target_tf})
|
||||
continue
|
||||
try:
|
||||
derived_df = resample_to_interval(base_df, minutes) # type: ignore[misc]
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"聚合周期计算失败",
|
||||
extra={"symbol": symbol, "base_timeframe": base_timeframe, "target_timeframe": target_tf},
|
||||
)
|
||||
continue
|
||||
if derived_df is None or derived_df.empty:
|
||||
continue
|
||||
derived_df = derived_df.copy()
|
||||
if "timestamp" not in derived_df.columns:
|
||||
if "date" in derived_df.columns:
|
||||
dates = pd.to_datetime(derived_df["date"], utc=True, errors="coerce")
|
||||
derived_df["timestamp"] = (dates.view("int64") // 1_000_000)
|
||||
elif isinstance(derived_df.index, pd.DatetimeIndex):
|
||||
idx = derived_df.index
|
||||
if idx.tz is None:
|
||||
idx = idx.tz_localize("UTC")
|
||||
else:
|
||||
idx = idx.tz_convert("UTC")
|
||||
derived_df["timestamp"] = (idx.view("int64") // 1_000_000)
|
||||
if "timestamp" not in derived_df.columns:
|
||||
logger.warning(
|
||||
"聚合结果缺少 timestamp 列，已跳过",
|
||||
extra={"target_timeframe": target_tf},
|
||||
)
|
||||
continue
|
||||
derived_df = derived_df.dropna(subset=["timestamp", "open", "high", "low", "close", "volume"])
|
||||
if derived_df.empty:
|
||||
continue
|
||||
derived_df["timestamp"] = derived_df["timestamp"].astype("int64")
|
||||
derived_df = derived_df.sort_values("timestamp")
|
||||
last_ts = get_last_timestamp(DATA_DIR, symbol, target_tf)
|
||||
if last_ts is not None:
|
||||
derived_df = derived_df[derived_df["timestamp"] > last_ts]
|
||||
if derived_df.empty:
|
||||
continue
|
||||
numpy_rows = derived_df[["timestamp", "open", "high", "low", "close", "volume"]].to_numpy()
|
||||
records: List[CandleRow] = []
|
||||
for ts, o, h, l, c, v in numpy_rows:
|
||||
records.append(
|
||||
[
|
||||
int(ts),
|
||||
float(o),
|
||||
float(h),
|
||||
float(l),
|
||||
float(c),
|
||||
float(v),
|
||||
]
|
||||
)
|
||||
if not records:
|
||||
continue
|
||||
upsert_candles(DATA_DIR, symbol, target_tf, records)
|
||||
updates.append((target_tf, records[-3:] if len(records) > 3 else records))
|
||||
return updates
|
||||
|
||||
|
||||
@dataclass
|
||||
class FetchState:
|
||||
symbol: str
|
||||
timeframe: str
|
||||
started_at: datetime = field(default_factory=datetime.utcnow)
|
||||
last_fetch_at: Optional[datetime] = None
|
||||
last_candle_ts: Optional[int] = None
|
||||
consecutive_errors: int = 0
|
||||
last_error: Optional[str] = None
|
||||
|
||||
def to_payload(self) -> dict:
|
||||
def serialize_dt(dt: Optional[datetime]) -> Optional[str]:
|
||||
if not dt:
|
||||
return None
|
||||
return dt.replace(microsecond=0).isoformat() + "Z"
|
||||
|
||||
return {
|
||||
"symbol": self.symbol,
|
||||
"timeframe": self.timeframe,
|
||||
"started_at": serialize_dt(self.started_at),
|
||||
"last_fetch_at": serialize_dt(self.last_fetch_at),
|
||||
"last_candle_ts": self.last_candle_ts,
|
||||
"consecutive_errors": self.consecutive_errors,
|
||||
"last_error": self.last_error,
|
||||
}
|
||||
|
||||
|
||||
fetch_states: Dict[Tuple[str, str], FetchState] = {}
|
||||
|
||||
|
||||
def build_exchange():
|
||||
if EXCHANGE.lower() == "binance":
|
||||
return ccxt.binance({"enableRateLimit": True})
|
||||
return ccxt_async.binance({"enableRateLimit": True})
|
||||
raise RuntimeError(f"Unsupported EXCHANGE: {EXCHANGE}")
|
||||
|
||||
|
||||
async def fetch_loop(symbol: str, timeframe: str):
|
||||
"""持续增量抓取并广播。"""
|
||||
derived_timeframes = AGGREGATION_TARGETS.get(timeframe, [])
|
||||
global RESAMPLE_WARNING_EMITTED
|
||||
if derived_timeframes and not RESAMPLE_AVAILABLE and not RESAMPLE_WARNING_EMITTED:
|
||||
logger.warning(
|
||||
"缺少 technical.util.resample_to_interval 模块，聚合时间周期生成已跳过",
|
||||
extra={"timeframe": timeframe},
|
||||
)
|
||||
RESAMPLE_WARNING_EMITTED = True
|
||||
|
||||
exchange = build_exchange()
|
||||
tf_ms = tf_to_ms(timeframe)
|
||||
start_since = parse_start_from_ms(START_FROM)
|
||||
last_ts = get_last_timestamp(DATA_DIR, symbol, timeframe)
|
||||
since = max(start_since, (last_ts + tf_ms) if last_ts else start_since)
|
||||
backoff = 1.0
|
||||
state_key = (symbol, timeframe)
|
||||
fetch_states[state_key] = FetchState(symbol=symbol, timeframe=timeframe, last_candle_ts=last_ts)
|
||||
|
||||
while True:
|
||||
try:
|
||||
candles = exchange.fetch_ohlcv(symbol, timeframe, since=since, limit=1000)
|
||||
if candles:
|
||||
upsert_candles(DATA_DIR, symbol, timeframe, candles)
|
||||
for row in candles[-3:]:
|
||||
payload = {
|
||||
"topic": f"candles.{symbol}.{timeframe}",
|
||||
"type": "upsert",
|
||||
"data": {
|
||||
"t": row[0],
|
||||
"o": row[1],
|
||||
"h": row[2],
|
||||
"l": row[3],
|
||||
"c": row[4],
|
||||
"v": row[5],
|
||||
},
|
||||
}
|
||||
await hub.publish(symbol, timeframe, payload)
|
||||
since = candles[-1][0] + tf_ms
|
||||
await asyncio.sleep(max(1.0, tf_ms * POLL_FACTOR / 1000.0))
|
||||
except Exception:
|
||||
await asyncio.sleep(3.0)
|
||||
logger.info("启动拉取任务", extra={"symbol": symbol, "timeframe": timeframe})
|
||||
|
||||
try:
|
||||
while True:
|
||||
try:
|
||||
candles = await exchange.fetch_ohlcv(symbol, timeframe, since=since, limit=1000)
|
||||
if candles:
|
||||
upsert_candles(DATA_DIR, symbol, timeframe, candles)
|
||||
derived_updates: List[Tuple[str, List[CandleRow]]] = []
|
||||
if derived_timeframes and RESAMPLE_AVAILABLE:
|
||||
derived_updates = await asyncio.to_thread(
|
||||
resample_and_store,
|
||||
symbol,
|
||||
timeframe,
|
||||
derived_timeframes,
|
||||
)
|
||||
for row in candles[-3:]:
|
||||
payload = {
|
||||
"topic": f"candles.{symbol}.{timeframe}",
|
||||
"type": "upsert",
|
||||
"data": {
|
||||
"t": row[0],
|
||||
"o": row[1],
|
||||
"h": row[2],
|
||||
"l": row[3],
|
||||
"c": row[4],
|
||||
"v": row[5],
|
||||
},
|
||||
}
|
||||
await hub.publish(symbol, timeframe, payload)
|
||||
for target_tf, rows in derived_updates:
|
||||
if not rows:
|
||||
continue
|
||||
for row in rows:
|
||||
ts = int(row[0])
|
||||
o, h, l, c, v = map(float, row[1:])
|
||||
payload = {
|
||||
"topic": f"candles.{symbol}.{target_tf}",
|
||||
"type": "upsert",
|
||||
"data": {
|
||||
"t": ts,
|
||||
"o": o,
|
||||
"h": h,
|
||||
"l": l,
|
||||
"c": c,
|
||||
"v": v,
|
||||
},
|
||||
}
|
||||
await hub.publish(symbol, target_tf, payload)
|
||||
since = candles[-1][0] + tf_ms
|
||||
backoff = 1.0
|
||||
state = fetch_states[state_key]
|
||||
state.last_fetch_at = datetime.utcnow()
|
||||
state.last_candle_ts = candles[-1][0]
|
||||
state.consecutive_errors = 0
|
||||
state.last_error = None
|
||||
await asyncio.sleep(max(1.0, tf_ms * POLL_FACTOR / 1000.0))
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except (ccxt.NetworkError, ccxt.ExchangeNotAvailable, ccxt.RequestTimeout) as exc:
|
||||
logger.warning(
|
||||
"网络异常，准备重试",
|
||||
extra={"symbol": symbol, "timeframe": timeframe, "error": str(exc)},
|
||||
)
|
||||
state = fetch_states[state_key]
|
||||
state.last_error = str(exc)
|
||||
state.consecutive_errors += 1
|
||||
backoff = min(backoff * BACKOFF_BASE, BACKOFF_MAX)
|
||||
await asyncio.sleep(backoff)
|
||||
except Exception as exc:
|
||||
logger.exception(
|
||||
"抓取循环发生异常，重建客户端后重试",
|
||||
extra={"symbol": symbol, "timeframe": timeframe},
|
||||
)
|
||||
state = fetch_states[state_key]
|
||||
state.last_error = str(exc)
|
||||
state.consecutive_errors += 1
|
||||
await asyncio.sleep(backoff)
|
||||
with suppress(Exception):
|
||||
await exchange.close()
|
||||
exchange = build_exchange()
|
||||
backoff = min(backoff * BACKOFF_BASE, BACKOFF_MAX)
|
||||
except asyncio.CancelledError:
|
||||
logger.info("取消拉取任务", extra={"symbol": symbol, "timeframe": timeframe})
|
||||
state = fetch_states.get(state_key)
|
||||
if state:
|
||||
state.last_error = "cancelled"
|
||||
raise
|
||||
finally:
|
||||
with suppress(Exception):
|
||||
await exchange.close()
|
||||
logger.info("拉取任务退出", extra={"symbol": symbol, "timeframe": timeframe})
|
||||
|
||||
|
||||
@app.on_event("startup")
|
||||
async def on_start():
|
||||
ensure_storage(DATA_DIR)
|
||||
fetch_tasks.clear()
|
||||
for s in SYMBOLS:
|
||||
for tf in TIMEFRAMES:
|
||||
asyncio.create_task(fetch_loop(s, tf))
|
||||
for tf in FETCH_TIMEFRAMES:
|
||||
task = asyncio.create_task(fetch_loop(s, tf), name=f"fetch::{s}::{tf}")
|
||||
fetch_tasks.append(task)
|
||||
|
||||
|
||||
@app.on_event("shutdown")
|
||||
async def on_shutdown():
|
||||
if not fetch_tasks:
|
||||
return
|
||||
logger.info("正在停止拉取任务")
|
||||
tasks = list(fetch_tasks)
|
||||
for task in tasks:
|
||||
task.cancel()
|
||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
for result in results:
|
||||
if isinstance(result, Exception) and not isinstance(result, asyncio.CancelledError):
|
||||
logger.warning("任务停止时出现异常：%s", result)
|
||||
fetch_tasks.clear()
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
async def health():
|
||||
now = datetime.utcnow().replace(microsecond=0).isoformat() + "Z"
|
||||
return {
|
||||
"status": "ok",
|
||||
"time": now,
|
||||
"exchange": EXCHANGE,
|
||||
"symbols": SYMBOLS,
|
||||
"base_timeframes": FETCH_TIMEFRAMES,
|
||||
"derived_timeframes": DERIVED_TIMEFRAMES,
|
||||
"timeframes": AVAILABLE_TIMEFRAMES,
|
||||
"tasks": [state.to_payload() for state in fetch_states.values()],
|
||||
}
|
||||
|
||||
|
||||
@app.get("/api/candles")
|
||||
@@ -160,6 +469,7 @@ def api_candles(
|
||||
end: Optional[int] = Query(None, description="结束时间戳(ms)"),
|
||||
):
|
||||
try:
|
||||
ensure_symbol_timeframe(symbol, tf)
|
||||
df = read_candles(DATA_DIR, symbol, tf, start, end)
|
||||
records = df.to_dict("records") if not df.empty else []
|
||||
return JSONResponse(records)
|
||||
@@ -169,6 +479,9 @@ def api_candles(
|
||||
|
||||
@app.websocket("/ws")
|
||||
async def ws_endpoint(websocket: WebSocket, symbol: str, tf: str, since: Optional[int] = None):
|
||||
if symbol not in VALID_SYMBOLS or tf not in VALID_TIMEFRAMES:
|
||||
await websocket.close(code=status.WS_1008_POLICY_VIOLATION, reason="invalid symbol/timeframe")
|
||||
return
|
||||
await hub.subscribe(websocket, symbol, tf)
|
||||
try:
|
||||
snap = read_candles(DATA_DIR, symbol, tf, since, None)
|
||||
@@ -201,7 +514,9 @@ def root():
|
||||
"service": "Local Data Service",
|
||||
"exchange": EXCHANGE,
|
||||
"symbols": SYMBOLS,
|
||||
"timeframes": TIMEFRAMES,
|
||||
"base_timeframes": FETCH_TIMEFRAMES,
|
||||
"derived_timeframes": DERIVED_TIMEFRAMES,
|
||||
"timeframes": AVAILABLE_TIMEFRAMES,
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -7,8 +7,8 @@ services:
|
||||
environment:
|
||||
- EXCHANGE=binance
|
||||
- SYMBOLS=BTC/USDT:USDT,ETH/USDT:USDT
|
||||
- TIMEFRAMES=1m,5m,15m,1h
|
||||
- START_FROM=2025-01-01
|
||||
- TIMEFRAMES=1m,1h,1d,1w,1M
|
||||
- START_FROM=2022-01-01
|
||||
- POLL_FACTOR=0.5
|
||||
- DATA_DIR=/data
|
||||
- TZ=Asia/Shanghai
|
||||
|
||||
@@ -4,4 +4,5 @@ ccxt==4.4.27
|
||||
pandas==2.2.2
|
||||
pyarrow==16.1.0
|
||||
orjson==3.10.3
|
||||
technical==1.5.0
|
||||
|
||||
|
||||
Reference in New Issue
Block a user