Compare commits
93
Commits
7f393b93ed
...
chan
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2b1b500108 | ||
|
|
c22cd48f36 | ||
|
|
b301ad22c9 | ||
|
|
2cd50f1e01 | ||
|
|
6b72ba226b | ||
|
|
080ff7d20e | ||
|
|
3bbd92784c | ||
|
|
77056a8318 | ||
|
|
f14ef540c1 | ||
|
|
7585491481 | ||
|
|
1ee4c3d4e1 | ||
|
|
e21f1103c5 | ||
|
|
84ee59ecfb | ||
|
|
93ed315a4b | ||
|
|
ed741c16b8 | ||
|
|
380d6d61cd | ||
|
|
7292ef5dd4 | ||
|
|
5695d8e983 | ||
|
|
78708ddeee | ||
|
|
e137d92b50 | ||
|
|
9efbae6ade | ||
|
|
be3602f5ba | ||
|
|
409dc2f020 | ||
|
|
30592889aa | ||
|
|
6d93975e67 | ||
|
|
1ffc4fbb18 | ||
|
|
17fe965a7c | ||
|
|
881cc5e77d | ||
|
|
5ffbadb0b2 | ||
|
|
668ebe1e44 | ||
|
|
f62b4f62a5 | ||
|
|
c513455998 | ||
|
|
7435cef61f | ||
|
|
9562b8c598 | ||
|
|
d797adc77e | ||
|
|
040810d48d | ||
|
|
d884de2c36 | ||
|
|
b6ed5c8b50 | ||
|
|
999ad924ad | ||
|
|
15f4aa088e | ||
|
|
3e3d579e35 | ||
|
|
1642b1bbfc | ||
|
|
335d891478 | ||
|
|
af029bc26e | ||
|
|
cef894376c | ||
|
|
be442783a1 | ||
|
|
f11c6507b0 | ||
|
|
bfdb2f2e2a | ||
|
|
06928d1d5f | ||
|
|
431e776905 | ||
|
|
360e4e1c41 | ||
|
|
f716ddd149 | ||
|
|
c57adc8cb0 | ||
|
|
97457b0518 | ||
|
|
0745e73b4f | ||
|
|
545fcc7c77 | ||
|
|
34a8f37b39 | ||
|
|
0a0fd2f682 | ||
|
|
c8a0062707 | ||
|
|
f5f456c523 | ||
|
|
d4fbe9d905 | ||
|
|
ef584b8d49 | ||
|
|
2b9a837a58 | ||
|
|
ef2a85dd8c | ||
|
|
27a4360ce8 | ||
|
|
7c13781f90 | ||
|
|
d13a0f85bc | ||
|
|
550eafdd9d | ||
|
|
99ae072cc6 | ||
|
|
d37bfcb26b | ||
|
|
2a59dcb9dc | ||
|
|
4a4f8727fd | ||
|
|
6c843639d5 | ||
|
|
9f4d7fec73 | ||
|
|
d2fcb27d01 | ||
|
|
0b4d7693b8 | ||
|
|
3b14bba247 | ||
|
|
75fbf4167b | ||
|
|
54792fe015 | ||
|
|
95f2b614a2 | ||
|
|
1661bbbac9 | ||
|
|
631d97e493 | ||
|
|
01c4a4ab4d | ||
|
|
28075173af | ||
|
|
181bca303f | ||
|
|
6259f7380b | ||
|
|
32c18f0201 | ||
|
|
9b72173285 | ||
|
|
7e339d2a54 | ||
|
|
66061f79a1 | ||
|
|
b286cb0137 | ||
|
|
a243edd2d3 | ||
|
|
206b27fe72 |
+38
@@ -15,8 +15,46 @@ __pycache__/
|
||||
*.sqlite-shm
|
||||
*.sqlite-wal
|
||||
|
||||
# Office documents kept alongside the repo but not part of it.
|
||||
# "~$" files are Excel's lock files, recreated every time a workbook is opened.
|
||||
*.xlsx
|
||||
*.xls
|
||||
~$*
|
||||
|
||||
# Local data
|
||||
data/
|
||||
|
||||
# Exchange history fetched by research/live/*.py. 40MB and re-fetchable from
|
||||
# the venue, so it stays local; the small result CSVs it feeds are committed.
|
||||
research/live/cache/
|
||||
|
||||
# Scratch outputs from short shakedown runs, superseded by the real collection.
|
||||
research/out/archive/
|
||||
|
||||
# Per-trade simulation dumps from research/step*.py. 70MB+ and regenerable by
|
||||
# rerunning the step; the summaries they feed live in HANDOFF.md.
|
||||
research/out/*.feather
|
||||
|
||||
# Virtualenvs. venv writes its own .gitignore since 3.11, but only for the
|
||||
# directory it creates — declare it here so other layouts are covered too.
|
||||
.venv/
|
||||
venv/
|
||||
|
||||
# Local tooling
|
||||
.gstack/
|
||||
research/out/*.jsonl.gz
|
||||
research/out/penetration.csv
|
||||
research/out/shadow_*.csv
|
||||
research/out/run_meta_*.json
|
||||
|
||||
# Telegram 凭据。**不要提交**
|
||||
research/live/deploy/tg.env
|
||||
research/.tg.env
|
||||
|
||||
# 生产状态与信号总线。刻意放在仓库外(LIVE_HOME / BUS_DIR),这几条只防
|
||||
# 有人把它们指回仓库里:里面是日亏损累计与已处理信号键,被 git clean
|
||||
# 清掉等于两道闸静默失忆
|
||||
/live/deploy/live.env
|
||||
live_state.json
|
||||
live_trades.jsonl
|
||||
signals_live.jsonl
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
# Chan — 缠论分析引擎
|
||||
|
||||
把 OHLCV K 线拆成缠论结构(K 线单元 → 合并 K 线 → 分型 → 笔 → 线段 → 中枢 → 买卖点),
|
||||
配一个 TradingView Charting Library 的 Web 界面,外加一套验证信号有效性的回测脚本。
|
||||
|
||||
标的不限:加密永续(ccxt)与 A 股(akshare)都走同一条分析链路。
|
||||
|
||||
```
|
||||
chanlun/ 缠论引擎,纯 pandas/numpy,无外部指标库依赖
|
||||
web/ Flask API + 图表界面
|
||||
research/ 信号有效性验证脚本(step1 ~ step30)
|
||||
data/ 本地 K 线(freqtrade 的 feather 格式)
|
||||
```
|
||||
|
||||
## 安装
|
||||
|
||||
需要 Python ≥ 3.11(pandas 3.x / numpy 2.x 的要求,不是本项目代码的限制)。
|
||||
|
||||
```bash
|
||||
python -m venv .venv
|
||||
.venv/bin/pip install -r requirements.txt # 核心运行时,8 个包
|
||||
.venv/bin/pip install -r requirements-dev.txt # 另加测试与 research/ 所需
|
||||
```
|
||||
|
||||
## 跑 Web
|
||||
|
||||
```bash
|
||||
cd web
|
||||
../.venv/bin/python app.py # 默认 http://0.0.0.0:8128
|
||||
```
|
||||
|
||||
从仓库根跑 `.venv/bin/python web/app.py` 也可以——Python 会把脚本所在目录放进 `sys.path`。
|
||||
但**不能用 `python -m web.app`**,也不能 `import web.app`:`web/` 内部是无前缀导入
|
||||
(`import config`、`from api.analyze import bp`),`-m` 方式下 `sys.path` 里是仓库根而不是
|
||||
`web/`,会 `ModuleNotFoundError: No module named 'config'`。
|
||||
|
||||
配置全部走环境变量,见 `web/config.py`:
|
||||
|
||||
| 变量 | 默认值 | 用途 |
|
||||
|------|--------|------|
|
||||
| `FLASK_HOST` / `FLASK_PORT` | `0.0.0.0` / `8128` | 监听地址 |
|
||||
| `DATA_SERVICE_URL` | `https://provider.jackyu66.com` | 行情 REST 源 |
|
||||
| `DATA_SERVICE_WS_URL` | `wss://jackyu66.com/ws` | 行情 WebSocket 源 |
|
||||
| `ASHARE_DP_URL` | `http://103.179.242.166:8000` | A 股数据源 |
|
||||
| `CHAN_HTTP_PROXY` | 未设置则不走代理 | ccxt / HTTP 代理 |
|
||||
| `MACD_FACTOR` / `MACD_SMOOTH` | `1` / `1` | MACD 周期倍数,默认 12/26/9 |
|
||||
|
||||
主要接口:`GET /api/analyze` 返回某标的某周期的完整缠论结构,`/api/klines/recent`
|
||||
取最新 K 线,`/api/trend_filter` 与 `/api/trend_detail` 做多周期趋势筛选,
|
||||
`/api/symbols`、`/api/search_stock`、`/api/sectors` 等负责标的检索。页面在 `/` 与 `/chan_tv`。
|
||||
|
||||
## 作为库使用
|
||||
|
||||
```python
|
||||
import pandas as pd
|
||||
from chanlun import TF_DF
|
||||
|
||||
# 需要 date/open/high/low/close/volume 六列,date 为 datetime
|
||||
df = pd.read_feather("data/binance/futures/BTC_USDT_USDT-1h-futures.feather")
|
||||
|
||||
tf = TF_DF(df, interval=1, timeframe="1h")
|
||||
|
||||
len(tf.klu_list) # K 线单元
|
||||
len(tf.klc_list) # 合并 K 线(处理包含关系后)
|
||||
len(tf.bi_list) # 笔
|
||||
len(tf.seg_list) # 线段
|
||||
len(tf.zs_list) # 中枢
|
||||
tf.bi_list[-1].dir # Chan_BI_DIR.DOWN
|
||||
tf.chanmacd # MACD 结构分析(背驰判定用)
|
||||
```
|
||||
|
||||
**`interval` 的单位是分钟**,对传入的 df 做重采样;`interval=1` 是特例,表示原样使用、
|
||||
不重采样。所以拿 1h 的 feather 要传 `interval=1`,拿 1m 数据想看 1h 才传 `interval=60`:
|
||||
|
||||
```python
|
||||
df1m = pd.read_feather("data/binance/futures/BTC_USDT_USDT-1m-futures.feather")
|
||||
TF_DF(df1m, interval=5, timeframe="5m")
|
||||
TF_DF(df1m, interval=60, timeframe="1h")
|
||||
```
|
||||
|
||||
传错不会报错,只会静默给出错误周期的结构——1h 数据配 `interval=4` 相当于按 4 分钟
|
||||
重采样,结果与 `interval=1` 完全相同。
|
||||
|
||||
## 分析流程
|
||||
|
||||
`TF_DF.init_TF_DF()` 按顺序做这几步,每步的实现在 `chanlun/pipeline/builders/` 下同名文件:
|
||||
|
||||
1. `resample_to_interval` — 重采样(`interval != 1` 时)
|
||||
2. `add_indicators` — 追加 33 列指标(MACD / BBANDS / EMA / RSI / ATR 等)
|
||||
3. `cal_kl_data` → `klu_list` — K 线单元
|
||||
4. `get_klc_list` → `klc_list` — 按包含关系合并 K 线,并标记分型
|
||||
5. `cal_bi_list` → `bi_list` — 笔
|
||||
6. `cal_bi_zs_list_pure` → `bi_zs_list` — 笔中枢
|
||||
7. `get_seg_list` → `seg_list` — 线段
|
||||
8. `get_zs_list` / `get_big_zs_list` — 中枢与大级别中枢
|
||||
9. `ChanMACD(klu_list)` — MACD 段 / 柱堆结构,供背驰判定
|
||||
|
||||
## 数据
|
||||
|
||||
`data/<交易所>/futures/<SYMBOL>-<周期>-futures.feather`,即 freqtrade 的下载格式,
|
||||
如 `data/binance/futures/BTC_USDT_USDT-1h-futures.feather`。
|
||||
`research/lib/data.py` 负责定位:`BTC/USDT:USDT` + `1h` 会解析到上面这个路径,
|
||||
找不到本地文件则回落到远端拉取。
|
||||
|
||||
## 测试
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m pytest chanlun/tests web/tests -q
|
||||
```
|
||||
|
||||
`chanlun/tests/test_ta_compat.py` 有个**需要注意的陷阱**:它把 `chanlun/indicators/ta.py`
|
||||
的输出逐 bar 钉在 TA-Lib 上,但 **TA-Lib 不存在时会静默跳过**。也就是说改了 `ta.py`
|
||||
之后在没装 TA-Lib 的环境里跑,测试会显示通过,其实一项都没验证。改动那个文件时请先装:
|
||||
|
||||
```bash
|
||||
sudo apt-get install -y libta-lib0 ta-lib-dev
|
||||
.venv/bin/pip install TA-Lib technical
|
||||
```
|
||||
|
||||
## 已知问题
|
||||
|
||||
- **`web/DEPLOY_GUIDE.md` 已失效**:它引用的 `deploy_venv.sh`、`stop_venv.sh`、
|
||||
`status_venv.sh` 等 6 个脚本都在 `7f393b9` 精简提交里删掉了,目前没有部署脚本。
|
||||
两个 systemd unit 文件(`web/chanlun-web*.service`)仍可参考,但它们用 gunicorn
|
||||
且写死端口 8123,与 `config.py` 默认的 8128 不一致,gunicorn 也不在依赖清单里。
|
||||
- **`web/README.txt` 已过时**:它说的 `web/requirements.txt` 不存在,依赖清单在仓库根目录。
|
||||
- **四个零引用的死文件**:`chanlun/analysis/` 下的 `ChanPY.py`、`ChanLun_Classifier.py`、
|
||||
`Find_Trend.py`、`ChanHeng.py` 全项目无人引用。`ChanPY.py` 依赖未安装的外部 chan.py 库,
|
||||
另外三个需要 matplotlib / mplfinance / xgboost / scikit-learn——这些都**不在**依赖清单里,
|
||||
是有意为之。要用得自行安装。
|
||||
@@ -14,12 +14,10 @@ from chanlun.core.ChanSBI import ChanSBI
|
||||
from chanlun.core.ChanSEG import ChanSEG
|
||||
from chanlun.core.ChanZS import ChanZS
|
||||
from chanlun.core.ChanBSP import ChanBSP
|
||||
import talib.abstract as ta
|
||||
import pandas as pd
|
||||
import matplotlib.pyplot as plt
|
||||
from matplotlib.dates import DateFormatter, date2num
|
||||
import matplotlib.patches as patches
|
||||
from technical.util import resample_to_interval
|
||||
from decimal import Decimal
|
||||
from chanlun.pipeline.orchestrator import ChanLun
|
||||
import xgboost as xgb
|
||||
|
||||
@@ -2,7 +2,7 @@ import ccxt
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
import mplfinance as mpf
|
||||
from talib import MACD, SMA
|
||||
from chanlun.indicators import ta
|
||||
from datetime import datetime, timedelta
|
||||
import logging
|
||||
import datetime as dt
|
||||
@@ -249,8 +249,9 @@ def analyze_higher_timeframe(df_30m):
|
||||
# 8. Back-divergence detection (enhanced)
|
||||
def detect_back_divergence(df, strokes, higher_trend):
|
||||
try:
|
||||
macd, signal, hist = MACD(df['Close'], fastperiod=12, slowperiod=26, signalperiod=9)
|
||||
sma20 = SMA(df['Close'], timeperiod=20)
|
||||
macd_df = ta.MACD(df['Close'], fastperiod=12, slowperiod=26, signalperiod=9)
|
||||
macd, hist = macd_df['macd'], macd_df['macdhist']
|
||||
sma20 = ta.SMA(df['Close'], timeperiod=20)
|
||||
df['macd'] = macd
|
||||
df['hist'] = hist
|
||||
df['sma20'] = sma20
|
||||
|
||||
@@ -0,0 +1,384 @@
|
||||
"""快速三类买卖点(引擎内称第四类,B4/S4)。
|
||||
|
||||
原本长在 `research/lib/fast_bsp3.py`,现移入引擎作为唯一实现,研究脚本改为转发导入,
|
||||
这样回测口径与 web 图表永远一致。
|
||||
|
||||
命名说明:缠论原文里没有「第四类买卖点」,但 B4/S4 也不只是「B3/S3 提前几根」——
|
||||
两者的选样口径不同,统计性质符号相反,故单独立类。形态条件是实时可判的:
|
||||
|
||||
中枢已成 -> 收盘突破 zg -> 收盘未跌回中枢 -> 重新上行
|
||||
|
||||
最后一步发生的当根就能下单,滞后约 2 根,而引擎 B3/S3 要等 pullback_bi.sure_time,
|
||||
滞后 9~10 根。但差别不止滞后:
|
||||
|
||||
判据 引擎用笔端点事后判(回拉笔低点 >= zg),本函数用收盘价实时判
|
||||
方向 引擎由离开笔方向决定,本函数由收盘从哪一侧突破决定
|
||||
口径 627 个中枢里引擎发 625 个信号(几乎不筛),本函数只认 212 个(34%);
|
||||
被拒的多数是「中枢确认时价格早已离开、此后再没回来」的历史区间
|
||||
|
||||
step30 同条件对拍(同一套 pure 笔中枢、同一组过滤器、同样的 1.5/3.0/48 出场):
|
||||
|
||||
原始 PF +大级别同向 +同向+阶梯 胜率 t值
|
||||
引擎 B3/S3 0.66 0.66 0.71 27.4% -18.76
|
||||
本函数 B4/S4 1.59 1.85 2.26 47.1% +10.09
|
||||
|
||||
同一组过滤器对 B4 有效、对 B3 无效,滞后差解释不了这一点(入场后移 1~4 根只是
|
||||
PF 3.18->2.53 的平滑衰减)。且 SL1.5/TP3.0 下随机入场胜率约 33%,引擎那 27.4%
|
||||
低于随机——它选中的是一批系统性反向的样本,不是「晚了所以差」。
|
||||
|
||||
require_touch 控制是否强求回抽碰到中枢边界。step32/33 的实测表明强求反而更差:
|
||||
这等于排除掉「突破后一去不回头」的强势段,而那正是缠论里最强的趋势形态。
|
||||
故默认 False(笔数 +21%、滞后 -1 根、PF 2.26→2.37)。
|
||||
|
||||
判据本身(收盘越过前一根极值)没有独立预测力:裸用 15 万笔样本 PF 0.95,
|
||||
把中枢换成「近20根高点」这类伪阻力位后 PF 0.90。alpha 全部来自中枢结构,
|
||||
判据只负责在这个已知价位上确认动能恢复。它也不能用于识别笔端点——
|
||||
入场前的回抽极值命中笔端点±2根的比例 19.3%,低于随机基准 22%。
|
||||
|
||||
全部判定只使用当根及之前的数据,无未来函数。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from chanlun.core.ChanEnum import Chan_FX_TYPE
|
||||
|
||||
# 中枢的「可用时刻」取第几笔的确认时间。
|
||||
# -1 = 最后一笔(历史默认)。中枢每吸收一根K线,最后一笔就可能后移,
|
||||
# 于是 available_ts 跟着漂——这是右边缘重画的根因(见 HANDOFF §5.41)。
|
||||
# 2 = 第三笔。三笔重叠即中枢成立,此后不再变,理论上更早也更稳。
|
||||
# ⛔ step47 已判定 bis[2] 作废(毛 R 0.933 → −0.000,见 HANDOFF §5.42)。
|
||||
# 开关保留只为可复现那次 A/B,**不要改默认值**。
|
||||
import os as _os
|
||||
|
||||
AVAIL_BI_INDEX = -1
|
||||
|
||||
|
||||
def _resolve_avail_bi(avail_bi: int | None) -> int:
|
||||
"""优先级:显式入参 > 环境变量 CHAN_AVAIL_BI > 模块默认。
|
||||
|
||||
环境变量在调用时读取而非 import 时——ProcessPoolExecutor 在 fork 启动方式下
|
||||
子进程会继承已 import 的模块,import 时读就固化成父进程的值了。
|
||||
"""
|
||||
if avail_bi is not None:
|
||||
return avail_bi
|
||||
return int(_os.environ.get("CHAN_AVAIL_BI", AVAIL_BI_INDEX))
|
||||
|
||||
|
||||
def timestamps_ms(src: pd.DataFrame) -> np.ndarray:
|
||||
"""取毫秒时间戳。研究侧的 df 自带 timestamp,web 侧的不一定,故按 date 回退。
|
||||
|
||||
回退写法不能用 `date.astype("int64") // 10**6`:该值单位取决于列精度,
|
||||
对毫秒精度的列会把时间戳砸平。
|
||||
"""
|
||||
if "timestamp" in src.columns:
|
||||
return src["timestamp"].to_numpy()
|
||||
d = pd.to_datetime(src["date"])
|
||||
if getattr(d.dt, "tz", None) is None:
|
||||
d = d.dt.tz_localize("UTC")
|
||||
return (d.dt.tz_convert("UTC").dt.tz_localize(None)
|
||||
.astype("datetime64[ms]").astype("int64").to_numpy())
|
||||
|
||||
|
||||
def ensure_timestamp(df: pd.DataFrame) -> pd.DataFrame:
|
||||
"""保证 df 带 timestamp 列,缺失时补一份副本,不改动调用方的对象。"""
|
||||
if "timestamp" in df.columns:
|
||||
return df
|
||||
out = df.copy()
|
||||
out["timestamp"] = timestamps_ms(out)
|
||||
return out
|
||||
|
||||
|
||||
def build_htf_zones(df_htf: pd.DataFrame, tf: str, chan=None, avail_bi: int | None = None) -> pd.DataFrame:
|
||||
"""算 pure 笔中枢,返回带生效时间的区间表。
|
||||
|
||||
available_ts —— 该中枢最早可被使用的时间戳(其确认时刻)。
|
||||
传入已构建好的 chan 可避免重复跑一遍 pipeline(大数据集上省一半时间)。
|
||||
"""
|
||||
if chan is None:
|
||||
from chanlun import TF_DF
|
||||
|
||||
chan = TF_DF(df_htf, 1, tf)
|
||||
zs_list = chan.cal_bi_zs_list_pure(chan.bi_list)
|
||||
# 用引擎自己的 dataframe 对齐,避免调用方传入的 df 与引擎内部行数不一致
|
||||
src = chan.dataframe if getattr(chan, "dataframe", None) is not None else df_htf
|
||||
return zones_from_zs_list(zs_list, src, avail_bi=avail_bi)
|
||||
|
||||
|
||||
def zones_from_zs_list(zs_list, src: pd.DataFrame, avail_bi: int | None = None) -> pd.DataFrame:
|
||||
"""把已算好的 pure 笔中枢转成区间表。
|
||||
|
||||
调用方手工跑过 cal_bi_zs_list_pure 时走这里,免得再算一遍(web 的 analyze_chan
|
||||
就是这种用法)。
|
||||
"""
|
||||
ts_of = dict(zip(src["date"].dt.strftime("%Y-%m-%d %H:%M:%S"), timestamps_ms(src)))
|
||||
# 中枢可用时刻取第几笔。在循环外解析一次,别让每个中枢都去读一遍环境变量。
|
||||
i = _resolve_avail_bi(avail_bi)
|
||||
|
||||
rows = []
|
||||
for zs in zs_list:
|
||||
bis = getattr(zs, "bi_list", [])
|
||||
if not bis:
|
||||
continue
|
||||
# 笔数不够时退回最后一笔(i=2 需要至少 3 笔才成立)
|
||||
key_bi = bis[i] if (i == -1 or len(bis) > i) else bis[-1]
|
||||
sure_key = str(getattr(key_bi, "sure_time", "") or "")
|
||||
end_key = str(getattr(key_bi, "end_time", "") or "")
|
||||
avail = ts_of.get(sure_key) or ts_of.get(end_key)
|
||||
if avail is None:
|
||||
continue
|
||||
start_key = str(bis[0].start_time)
|
||||
rows.append({
|
||||
"zg": float(zs.zg), "zd": float(zs.zd),
|
||||
"gg": float(getattr(zs, "gg", zs.zg)), "dd": float(getattr(zs, "dd", zs.zd)),
|
||||
"start_ts": ts_of.get(start_key, avail),
|
||||
"available_ts": int(avail),
|
||||
})
|
||||
out = pd.DataFrame(rows)
|
||||
return out.sort_values("available_ts").reset_index(drop=True) if not out.empty else out
|
||||
|
||||
|
||||
def add_zone_ladder(zones: pd.DataFrame) -> pd.DataFrame:
|
||||
"""标注每个中枢相对前一个中枢是否同向推进(缠论「趋势 vs 盘整」)。
|
||||
|
||||
z_above / z_below 分别对应向上、向下推进。买信号要求 z_above、卖信号要求 z_below
|
||||
时,30m/2h 的 PF 从 2.72 升到 3.41。
|
||||
"""
|
||||
out = zones.copy()
|
||||
if out.empty:
|
||||
out["z_above"], out["z_below"] = pd.Series(dtype=bool), pd.Series(dtype=bool)
|
||||
return out
|
||||
pg, pdn = out["zg"].shift(), out["zd"].shift()
|
||||
out["z_above"] = (out["zd"] > pg).fillna(False)
|
||||
out["z_below"] = (out["zg"] < pdn).fillna(False)
|
||||
return out
|
||||
|
||||
|
||||
def htf_fx_timeline(chan_htf, df_htf: pd.DataFrame | None = None) -> pd.DataFrame:
|
||||
"""把大级别分型压成一条按确认时间排序的时间线。
|
||||
|
||||
confirm_ts 是该分型最早可被使用的时刻。timestamp 是K线开盘时刻,而分型要等这根K线
|
||||
收盘才算数,所以整体后移一个大级别周期;否则小级别会提前一整根大级别K线拿到信号。
|
||||
|
||||
只取方向与确认时刻——同向过滤用不到背驰强度,省掉 MACD 面积计算。
|
||||
"""
|
||||
src = chan_htf.dataframe if getattr(chan_htf, "dataframe", None) is not None else df_htf
|
||||
if src is None or len(src) == 0:
|
||||
return pd.DataFrame(columns=["confirm_ts", "fx_ts", "direction", "price"])
|
||||
ts = timestamps_ms(src)
|
||||
idx_of = {t: i for i, t in enumerate(src["date"].dt.strftime("%Y-%m-%d %H:%M:%S"))}
|
||||
period = int(np.median(np.diff(ts))) if len(ts) > 1 else 0
|
||||
|
||||
rows = []
|
||||
for klc in getattr(chan_htf, "klc_list", []):
|
||||
if klc.fx not in (Chan_FX_TYPE.TOP, Chan_FX_TYPE.BOTTOM):
|
||||
continue
|
||||
if klc.next is None or klc.next.end_klu is None:
|
||||
continue
|
||||
e_key, c_key = str(klc.end_time), str(klc.next.end_klu.time)
|
||||
if e_key not in idx_of or c_key not in idx_of:
|
||||
continue
|
||||
fx_idx, confirm_idx = idx_of[e_key], idx_of[c_key]
|
||||
if confirm_idx <= fx_idx:
|
||||
continue
|
||||
d = 1 if klc.fx == Chan_FX_TYPE.BOTTOM else -1
|
||||
rows.append({
|
||||
"confirm_ts": int(ts[confirm_idx]) + period,
|
||||
"fx_ts": int(ts[fx_idx]),
|
||||
"direction": d,
|
||||
"price": float(klc.low if d == 1 else klc.high),
|
||||
})
|
||||
out = pd.DataFrame(rows)
|
||||
if out.empty:
|
||||
return pd.DataFrame(columns=["confirm_ts", "fx_ts", "direction", "price"])
|
||||
return out.sort_values("confirm_ts").reset_index(drop=True)
|
||||
|
||||
|
||||
def attach_htf_agree(sig: pd.DataFrame, df_ltf: pd.DataFrame, tl: pd.DataFrame) -> pd.DataFrame:
|
||||
"""给每个小级别信号挂上「入场时刻之前最近的大级别分型」是否同向。
|
||||
|
||||
产出 htf_dir(+1 底 / -1 顶)与 htf_agree(1 同向 / 0 反向 / NaN 无可用分型)。
|
||||
"""
|
||||
out = sig.copy()
|
||||
if sig.empty or tl.empty:
|
||||
out["htf_dir"] = np.nan
|
||||
out["htf_agree"] = np.nan
|
||||
return out
|
||||
|
||||
ts_ltf = timestamps_ms(df_ltf)
|
||||
entry_ts = ts_ltf[out["entry_idx"].to_numpy().astype(int)]
|
||||
k = np.searchsorted(tl["confirm_ts"].to_numpy(), entry_ts, side="right") - 1
|
||||
valid = k >= 0
|
||||
k_safe = np.clip(k, 0, len(tl) - 1)
|
||||
|
||||
fx_dir = tl["direction"].to_numpy()[k_safe].astype(float)
|
||||
out["htf_dir"] = np.where(valid, fx_dir, np.nan)
|
||||
out["htf_agree"] = np.where(
|
||||
valid, (fx_dir == out["direction"].to_numpy()).astype(float), np.nan
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def attach_zone_ladder(sig: pd.DataFrame, zones: pd.DataFrame) -> pd.DataFrame:
|
||||
"""按信号方向取该中枢的阶梯标记:买看 z_above、卖看 z_below。
|
||||
|
||||
zone_i 是 find_fast_bsp3 内 enumerate 出的位置序号,故用 iloc 定位。
|
||||
"""
|
||||
out = sig.copy()
|
||||
if sig.empty:
|
||||
out["ladder_ok"] = pd.Series(dtype=bool)
|
||||
return out
|
||||
z = zones if "z_above" in zones.columns else add_zone_ladder(zones)
|
||||
above = z["z_above"].to_numpy()
|
||||
below = z["z_below"].to_numpy()
|
||||
zi = out["zone_i"].to_numpy().astype(int)
|
||||
ok = np.where(out["direction"].to_numpy() == 1, above[zi], below[zi])
|
||||
out["ladder_ok"] = ok.astype(bool)
|
||||
return out
|
||||
|
||||
|
||||
def find_fast_bsp3(
|
||||
df: pd.DataFrame,
|
||||
zones: pd.DataFrame,
|
||||
scan: int = 200,
|
||||
pullback_win: int = 30,
|
||||
tol: float = -1.0,
|
||||
max_per_zone: int = 1,
|
||||
diag: dict | None = None,
|
||||
require_touch: bool = False,
|
||||
) -> pd.DataFrame:
|
||||
"""扫描每个中枢,找突破后回抽不回中枢的入场点。
|
||||
|
||||
zones 需含 zg / zd / available_ts,且 available_ts 已是可用时刻。
|
||||
max_per_zone > 1 时,同一中枢在首次入场后继续往后找二次、三次突破回抽,
|
||||
用来检验「趋势里同一中枢反复给机会」是否值得做。
|
||||
|
||||
tol 是「算作回抽中」的边界容差。require_touch=False 时它不再是入场门槛,
|
||||
仅决定哪些K线被视为回抽中而跳过转强判定,故 tol 越大入场越晚。
|
||||
默认负值 = 完全禁用该跳过,突破后每根都检查转强,滞后压到 2.2 根。
|
||||
step34 实测滞后与收益严格单调:9.9根 PF1.88 / 5.8根 2.37 / 2.2根 2.86。
|
||||
|
||||
返回列:
|
||||
entry_idx 实时可下单的K线
|
||||
direction +1 三买 / -1 三卖
|
||||
bo_idx 突破根
|
||||
pb_idx 回抽极值根
|
||||
lag entry_idx - bo_idx
|
||||
depth 回抽深度(相对中枢边界,负值表示曾插入中枢)
|
||||
occ 这是该中枢的第几次入场
|
||||
"""
|
||||
if zones.empty:
|
||||
return pd.DataFrame()
|
||||
|
||||
ts = df["timestamp"].to_numpy()
|
||||
close = df["close"].to_numpy(dtype=float)
|
||||
high = df["high"].to_numpy(dtype=float)
|
||||
low = df["low"].to_numpy(dtype=float)
|
||||
n = len(df)
|
||||
rows = []
|
||||
|
||||
def note(key: str) -> None:
|
||||
if diag is not None:
|
||||
diag[key] = diag.get(key, 0) + 1
|
||||
|
||||
for zone_i, (_, z) in enumerate(zones.iterrows()):
|
||||
note("中枢总数")
|
||||
zg, zd = float(z["zg"]), float(z["zd"])
|
||||
if zg <= zd:
|
||||
note("×无效中枢")
|
||||
continue
|
||||
start = int(np.searchsorted(ts, z["available_ts"], side="left"))
|
||||
if start >= n - 2:
|
||||
note("×中枢太靠后")
|
||||
continue
|
||||
# 允许多次入场时按比例放宽扫描窗口,否则后几次机会会被窗口截断
|
||||
scan_end = min(start + scan * max_per_zone, n)
|
||||
cursor = start
|
||||
|
||||
for occ in range(1, max_per_zone + 1):
|
||||
if cursor >= n - 2:
|
||||
break
|
||||
# 第一步:找突破。要求突破前确实待在中枢内,避免把远处的价格当突破。
|
||||
was_inside = False
|
||||
bo_idx, d = None, 0
|
||||
for j in range(cursor, scan_end):
|
||||
c = close[j]
|
||||
if zd <= c <= zg:
|
||||
was_inside = True
|
||||
continue
|
||||
if not was_inside:
|
||||
continue
|
||||
bo_idx, d = j, (1 if c > zg else -1)
|
||||
break
|
||||
if bo_idx is None:
|
||||
if occ == 1:
|
||||
note("×窗口内未突破")
|
||||
break
|
||||
|
||||
edge = zg if d == 1 else zd
|
||||
# 第二步:突破后监控回抽,回抽不跌回中枢且重新顺势 -> 入场
|
||||
touched = False
|
||||
pb_idx = None
|
||||
pb_ext = None
|
||||
entry_idx = None
|
||||
fell_back = False
|
||||
for j in range(bo_idx + 1, min(bo_idx + pullback_win + 1, n)):
|
||||
# 收盘跌回中枢 -> 突破失效
|
||||
if zd <= close[j] <= zg:
|
||||
fell_back = True
|
||||
break
|
||||
# 回抽触及边界附近(允许 tol 的毛刺)
|
||||
near = (low[j] <= edge * (1 + tol)) if d == 1 else (high[j] >= edge * (1 - tol))
|
||||
if near:
|
||||
touched = True
|
||||
ext = low[j] if d == 1 else high[j]
|
||||
if pb_ext is None or ((ext < pb_ext) if d == 1 else (ext > pb_ext)):
|
||||
pb_ext, pb_idx = ext, j
|
||||
continue
|
||||
# 回抽后重新顺势:收盘创出前一根之上(三买)/ 之下(三卖)
|
||||
# require_touch=False 时不强求回抽碰到中枢边界,
|
||||
# 这样「突破后一去不回头」的强势段也能收进来。
|
||||
if (touched and pb_idx is not None) or not require_touch:
|
||||
go = close[j] > high[j - 1] if d == 1 else close[j] < low[j - 1]
|
||||
if go:
|
||||
entry_idx = j
|
||||
break
|
||||
if entry_idx is None:
|
||||
if occ == 1:
|
||||
note("×突破后跌回中枢" if fell_back
|
||||
else "×回抽未触及边界" if not touched
|
||||
else "×触及边界但未转强")
|
||||
# 这次突破没走成,从突破点之后继续找下一次
|
||||
cursor = bo_idx + 1
|
||||
continue
|
||||
if pb_ext is None:
|
||||
# 未触及边界就转强(require_touch=False):取突破至入场间的实际极值
|
||||
seg = slice(bo_idx + 1, entry_idx + 1)
|
||||
pb_ext = float(low[seg].min() if d == 1 else high[seg].max())
|
||||
pb_idx = int((low[seg].argmin() if d == 1 else high[seg].argmax())
|
||||
+ bo_idx + 1)
|
||||
if occ == 1:
|
||||
note("√成交")
|
||||
|
||||
# 回抽深度:>0 表示未插入中枢,越大表示回抽越浅
|
||||
depth = (pb_ext - zg) / zg if d == 1 else (zd - pb_ext) / zd
|
||||
rows.append({
|
||||
"entry_idx": entry_idx, "direction": d,
|
||||
"bo_idx": bo_idx, "pb_idx": pb_idx,
|
||||
"lag": entry_idx - bo_idx,
|
||||
"depth": depth,
|
||||
"zg": zg, "zd": zd,
|
||||
"width_pct": (zg - zd) / close[bo_idx],
|
||||
"occ": occ,
|
||||
"zone_i": zone_i,
|
||||
})
|
||||
cursor = entry_idx + 1
|
||||
|
||||
out = pd.DataFrame(rows)
|
||||
if out.empty:
|
||||
return out
|
||||
# 相邻中枢可能突破到同一根K线,同一时刻只能有一个仓位,保留最早成型的那个
|
||||
return (out.sort_values(["entry_idx", "occ", "zone_i"])
|
||||
.drop_duplicates("entry_idx", keep="first")
|
||||
.reset_index(drop=True))
|
||||
+52
-21
@@ -15,10 +15,12 @@ class ChanBI():
|
||||
self.sure_time = None
|
||||
self.klc_list = []
|
||||
self.klc_list.append(klc)
|
||||
self._klc_idx = {klc.index}
|
||||
self.end_time = klc.end_time
|
||||
self.start_time = klc.start_time
|
||||
self.macd_hist = 0
|
||||
self.macd_div = 0
|
||||
self._macd_hist = 0
|
||||
self._macd_div = 0
|
||||
self._macd_dirty = False
|
||||
self.seg = None
|
||||
self.height = 0
|
||||
self.width = 0
|
||||
@@ -33,26 +35,57 @@ class ChanBI():
|
||||
def set_seg(self, seg):
|
||||
self.seg = seg
|
||||
self.seg_index = len(seg.bi_list)-1
|
||||
# macd_hist / macd_div 改为惰性。原来 add_klc 每加一根 KLC 就把整笔的
|
||||
# 所有 KLU 重新累加一遍,是 O(k²);而这两个值只有背驰判定(bsp.py)在读,
|
||||
# lean 模式下 bsp 根本不算,等于全程白算。这里只标脏,取值时才算。
|
||||
# 语义不变:klc_list 只增不减(set_start_klc 会重置并同时清脏),
|
||||
# dir 在 klc 累加期间固定,所以延后到读取时算与逐次重算结果相同。
|
||||
@property
|
||||
def macd_hist(self):
|
||||
if self._macd_dirty:
|
||||
self.cal_macdhist()
|
||||
return self._macd_hist
|
||||
|
||||
@macd_hist.setter
|
||||
def macd_hist(self, v):
|
||||
self._macd_hist = v
|
||||
self._macd_dirty = False
|
||||
|
||||
@property
|
||||
def macd_div(self):
|
||||
self.cal_macd_div()
|
||||
return self._macd_div
|
||||
|
||||
@macd_div.setter
|
||||
def macd_div(self, v):
|
||||
self._macd_div = v
|
||||
|
||||
def set_macdhist(self, macd_hist):
|
||||
self.macd_hist = macd_hist
|
||||
def set_macd_div(self, macd_div):
|
||||
self.macd_div = macd_div
|
||||
def cal_macd_div(self):
|
||||
self.macd_div = 0.0
|
||||
self._macd_div = 0.0
|
||||
if self.pre and self.pre.pre:
|
||||
if self.pre.pre.macd_hist == 0:
|
||||
self.macd_div = 0.0
|
||||
self._macd_div = 0.0
|
||||
else:
|
||||
self.macd_div = self.macd_hist / self.pre.pre.macd_hist
|
||||
self._macd_div = self.macd_hist / self.pre.pre.macd_hist
|
||||
#print(self.start_time, self.end_time, self.macd_hist, self.pre.pre.macd_hist, self.macd_div)
|
||||
def cal_macdhist(self):
|
||||
self.macd_hist = 0
|
||||
self._macd_dirty = False
|
||||
self._macd_hist = 0
|
||||
up = self.dir == Chan_BI_DIR.UP
|
||||
acc = 0
|
||||
for klc in self.klc_list:
|
||||
for klu in klc.klu_list:
|
||||
if self.dir == Chan_BI_DIR.UP and klu.macdhist > 0:
|
||||
self.macd_hist += klu.macdhist
|
||||
if self.dir == Chan_BI_DIR.DOWN and klu.macdhist < 0:
|
||||
self.macd_hist -= klu.macdhist
|
||||
h = klu.macdhist
|
||||
if up:
|
||||
if h > 0:
|
||||
acc += h
|
||||
elif h < 0:
|
||||
acc -= h
|
||||
self._macd_hist = acc
|
||||
def check_bi_zs_overlap(self):
|
||||
if self.next and self.next.next:
|
||||
if self.dir == Chan_BI_DIR.UP:
|
||||
@@ -95,6 +128,10 @@ class ChanBI():
|
||||
self.start_klc = klc
|
||||
self.klc_list = []
|
||||
self.klc_list.append(klc)
|
||||
# klc_list 被整个换掉,去重集合与惰性缓存都要跟着重置,
|
||||
# 否则后续 add_klc 会以为旧下标还在里面而漏加
|
||||
self._klc_idx = {klc.index}
|
||||
self._macd_dirty = True
|
||||
self.high = klc.high
|
||||
self.low = klc.low
|
||||
self.dir = ddir
|
||||
@@ -103,20 +140,14 @@ class ChanBI():
|
||||
def set_next(self, bi):
|
||||
self.next = bi
|
||||
def add_klc(self, klc):
|
||||
added = False
|
||||
if len(self.klc_list) > 0:
|
||||
for index in range(0, len(self.klc_list)):
|
||||
if self.klc_list[index].index == klc.index:
|
||||
added = True
|
||||
break
|
||||
if not added:
|
||||
# 去重原本是对 klc_list 线性扫描,配合下面每次全量重算的 macdhist,
|
||||
# 让「往一笔里加 k 根 KLC」变成 O(k²)。改用下标集合,O(1)。
|
||||
if klc.index not in self._klc_idx:
|
||||
self._klc_idx.add(klc.index)
|
||||
self.klc_list.append(klc)
|
||||
#print(self.start_time, klc.start_time)
|
||||
#print(klc.end_time, klc.index)
|
||||
self.end_klc = klc
|
||||
self.end_time = klc.klu_list[-1].time
|
||||
self.cal_macdhist()
|
||||
self.cal_macd_div()
|
||||
self._macd_dirty = True
|
||||
def append_klc_list(self, klc_list):
|
||||
self.klc_list.append(klc_list)
|
||||
def get_decimal(self, value):
|
||||
|
||||
@@ -281,6 +281,10 @@ class Chan_BSP_TYPE(Enum):
|
||||
S1 = auto()
|
||||
S2 = auto()
|
||||
S3 = auto()
|
||||
# 第四类:三类买卖点的低滞后变体,几何位置同 B3/S3,但不等笔确认。
|
||||
# 见 chanlun/analysis/fast_bsp.py
|
||||
B4 = auto()
|
||||
S4 = auto()
|
||||
NONE = auto()
|
||||
"""
|
||||
class Chan_BSP_TYPE(Enum):
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
from chanlun.core.ChanEnum import Chan_BSP_DIR, Chan_BSP_TYPE
|
||||
|
||||
|
||||
class ChanFastBSP():
|
||||
"""第四类买卖点(B4/S4)。
|
||||
|
||||
与 ChanBSP 的区别在于它不挂在笔上:fast_bsp 刻意不等笔确认,入场点是一根具体的
|
||||
K线而非一笔的端点,所以时间与价格直接取自 K 线,没有 bi / klc 可依附。
|
||||
|
||||
htf_agree 与 ladder_ok 是两个独立的过滤标志,不在这里合成——上层(图表或策略)
|
||||
自己决定要不要用、怎么组合。
|
||||
"""
|
||||
|
||||
def __init__(self, time, price, ddir, entry_idx, bo_time=None, pb_time=None,
|
||||
lag=0, depth=0.0, zg=None, zd=None, occ=1,
|
||||
htf_dir=None, htf_agree=None, ladder_ok=None):
|
||||
self.time = time
|
||||
self.price = float(price)
|
||||
self.dir = ddir
|
||||
self.type = Chan_BSP_TYPE.B4 if ddir == Chan_BSP_DIR.BUY else Chan_BSP_TYPE.S4
|
||||
self.entry_idx = int(entry_idx)
|
||||
self.bo_time = bo_time
|
||||
self.pb_time = pb_time
|
||||
self.lag = int(lag)
|
||||
self.depth = float(depth)
|
||||
self.zg = float(zg) if zg is not None else None
|
||||
self.zd = float(zd) if zd is not None else None
|
||||
self.occ = int(occ)
|
||||
self.htf_dir = htf_dir
|
||||
self.htf_agree = htf_agree
|
||||
self.ladder_ok = ladder_ok
|
||||
# 入场即成立,没有「等待确认」这个状态;留此字段是为了与 ChanBSP 的序列化对齐
|
||||
self.is_sure = True
|
||||
self.start_time = time
|
||||
self.end_time = time
|
||||
self.sure_time = time
|
||||
|
||||
def __repr__(self):
|
||||
name = str(self.type).replace('Chan_BSP_TYPE.', '')
|
||||
return f"<ChanFastBSP {name} {self.time} {self.price} lag={self.lag}>"
|
||||
+32
-8
@@ -63,10 +63,11 @@ class ChanKLC():
|
||||
self.bsp = False
|
||||
self.bsp_type = Chan_BSP_TYPE.NONE
|
||||
# EMA状态字典:key为EMA名称,value为 {'pos': Chan_EMA_POS, 'semantic': Chan_EMA_SEMANTIC}
|
||||
self.ema_status = {}
|
||||
self._ema_status = {}
|
||||
self._ema_status_dirty = False
|
||||
# 向后兼容:保留 ema52_status 和 ema52_pos
|
||||
self.ema52_status = 0
|
||||
self.ema52_pos = Chan_EMA_POS.UNKNOWN
|
||||
self._ema52_status = 0
|
||||
self._ema52_pos = Chan_EMA_POS.UNKNOWN
|
||||
self.bb2633upper = klu.bb2633upper
|
||||
self.bb2633lower = klu.bb2633lower
|
||||
self.bb2633middle = klu.bb2633middle
|
||||
@@ -283,21 +284,44 @@ class ChanKLC():
|
||||
'ema156': self.ema156,
|
||||
'ema208': self.ema208,
|
||||
}
|
||||
self.ema_status = {}
|
||||
self._ema_status_dirty = False
|
||||
self._ema_status = {}
|
||||
for name, value in ema_configs.items():
|
||||
# 按 EMA 值的百分比自动计算阈值
|
||||
threshold = abs(value) * self.threshold_pct if value and self.threshold_pct > 0 else 0
|
||||
pos = ChanKLC.cal_ema_pos(self.high, self.low, self.close, value, threshold)
|
||||
semantic = ChanKLC.cal_ema_semantic(pos, self.dir, self.ema_dir)
|
||||
self.ema_status[name] = {
|
||||
self._ema_status[name] = {
|
||||
'pos': pos,
|
||||
'semantic': semantic,
|
||||
'value': value,
|
||||
'threshold': threshold,
|
||||
}
|
||||
# 向后兼容
|
||||
self.ema52_pos = self.ema_status['ema52']['pos']
|
||||
self.ema52_status = ChanKLC.semantic_to_int(self.ema_status['ema52']['semantic'])
|
||||
self._ema52_pos = self._ema_status['ema52']['pos']
|
||||
self._ema52_status = ChanKLC.semantic_to_int(self._ema_status['ema52']['semantic'])
|
||||
|
||||
# 以下三个改成惰性求值。原来 set_end_klu 每次合并 KLU 都会立刻重算一遍,
|
||||
# 实测占 TF_DF 构建的约 25%,而全仓(含前端)没有任何地方读取它的产出。
|
||||
# 保留属性形式是为了任何外部读取仍拿到正确值,只是推迟到真被读时才算。
|
||||
@property
|
||||
def ema_status(self):
|
||||
if self._ema_status_dirty:
|
||||
self.cal_all_ema_status()
|
||||
return self._ema_status
|
||||
|
||||
@property
|
||||
def ema52_pos(self):
|
||||
if self._ema_status_dirty:
|
||||
self.cal_all_ema_status()
|
||||
return self._ema52_pos
|
||||
|
||||
@property
|
||||
def ema52_status(self):
|
||||
if self._ema_status_dirty:
|
||||
self.cal_all_ema_status()
|
||||
return self._ema52_status
|
||||
|
||||
def get_ema_pos(self, ema_name):
|
||||
"""获取指定EMA的客观位置,如 klc.get_ema_pos('ema24')"""
|
||||
if ema_name in self.ema_status:
|
||||
@@ -448,7 +472,7 @@ class ChanKLC():
|
||||
klu.set_klc(self)
|
||||
self.klc_dir = Chan_KLINE_DIR.UP if self.close > self.open else Chan_KLINE_DIR.DOWN
|
||||
self.cal_indicators()
|
||||
self.cal_all_ema_status()
|
||||
self._ema_status_dirty = True
|
||||
if self.open > self.high:
|
||||
self.open = self.high
|
||||
if self.close > self.high:
|
||||
|
||||
+33
-20
@@ -160,27 +160,40 @@ class ChanKLU:
|
||||
return False
|
||||
else:
|
||||
return True
|
||||
# 属性名 ← dataframe 列名。两条赋值路径(逐行 Series / 预取 ndarray)共用这张表,
|
||||
# 免得将来加指标时只改一处、另一处静默漏掉。
|
||||
INDICATOR_FIELDS = (
|
||||
('macd', 'macd'), ('signal', 'macdsignal'), ('macdhist', 'macdhist'),
|
||||
('ema26', 'ema26'), ('ema52', 'ema52'), ('ema24', 'ema24'),
|
||||
('ema104', 'ema104'), ('ema156', 'ema156'), ('ema208', 'ema208'),
|
||||
('ema13', 'ema13'), ('ema7', 'ema7'), ('ema5', 'ema5'),
|
||||
('rsi', 'rsi'), ('volume_ratio', 'volume_ratio'),
|
||||
('bb52upper', 'bb52upper'), ('bb52lower', 'bb52lower'),
|
||||
('bb2633upper', 'bb2633upper'), ('bb2633lower', 'bb2633lower'),
|
||||
('bb2633middle', 'bb2633middle'), ('ma5', 'ma5'),
|
||||
)
|
||||
|
||||
def set_indicators(self, item):
|
||||
self.macd = float(item['macd']) if 'macd' in item and item['macd'] else 0
|
||||
self.signal = float(item['macdsignal']) if 'macdsignal' in item and item['macdsignal'] else 0
|
||||
self.macdhist = float(item['macdhist']) if 'macdhist' in item and item['macdhist'] else 0
|
||||
self.ema26 = float(item['ema26']) if 'ema26' in item and item['ema26'] else 0
|
||||
self.ema52 = float(item['ema52']) if 'ema52' in item and item['ema52'] else 0
|
||||
self.ema24 = float(item['ema24']) if 'ema24' in item and item['ema24'] else 0
|
||||
self.ema104 = float(item['ema104']) if 'ema104' in item and item['ema104'] else 0
|
||||
self.ema156 = float(item['ema156']) if 'ema156' in item and item['ema156'] else 0
|
||||
self.ema208 = float(item['ema208']) if 'ema208' in item and item['ema208'] else 0
|
||||
self.ema13 = float(item['ema13']) if 'ema13' in item and item['ema13'] else 0
|
||||
self.ema7 = float(item['ema7']) if 'ema7' in item and item['ema7'] else 0
|
||||
self.rsi = float(item['rsi']) if 'rsi' in item and item['rsi'] else 0
|
||||
self.volume_ratio = float(item['volume_ratio']) if 'volume_ratio' in item and item['volume_ratio'] else 0
|
||||
self.bb52upper = float(item['bb52upper']) if 'bb52upper' in item and item['bb52upper'] else 0
|
||||
self.bb52lower = float(item['bb52lower']) if 'bb52lower' in item and item['bb52lower'] else 0
|
||||
self.bb2633upper = float(item['bb2633upper']) if 'bb2633upper' in item and item['bb2633upper'] else 0
|
||||
self.bb2633lower = float(item['bb2633lower']) if 'bb2633lower' in item and item['bb2633lower'] else 0
|
||||
self.bb2633middle = float(item['bb2633middle']) if 'bb2633middle' in item and item['bb2633middle'] else 0
|
||||
self.ma5 = float(item['ma5']) if 'ma5' in item and item['ma5'] else 0
|
||||
self.ema5 = float(item['ema5']) if 'ema5' in item and item['ema5'] else 0
|
||||
"""单根赋值。增量追加时每次只有一根,走这条即可。
|
||||
|
||||
原写法是 `float(item[c]) if c in item and item[c] else 0`。其中的真值判断
|
||||
是空转:值为 0.0 时 float(0.0) 仍是 0,值为 NaN 时 NaN 为真值、照样透传。
|
||||
唯一起作用的是「列不存在则填 0」,所以这里只保留那一层。
|
||||
"""
|
||||
for attr, col in self.INDICATOR_FIELDS:
|
||||
v = item[col] if col in item else 0
|
||||
setattr(self, attr, float(v) if v else 0)
|
||||
|
||||
def set_indicators_from(self, cols, i):
|
||||
"""从预取的 {列名: ndarray} 按下标赋值,语义与 set_indicators 相同。
|
||||
|
||||
全量构建时用这条:避免每根 `df.iloc[i]` 构造一个 Series,再在其上做
|
||||
几十次逐键查找——那是 TF_DF 构建 96% 的耗时所在。
|
||||
"""
|
||||
for attr, col in self.INDICATOR_FIELDS:
|
||||
arr = cols.get(col)
|
||||
v = arr[i] if arr is not None else 0
|
||||
setattr(self, attr, float(v) if v else 0)
|
||||
def cal_macd_state(self):
|
||||
# 按定义精简实现:优先级 CROSS0 > 位置(HIGH/HE/RETURN_ZERO) > NEAR0 > UNKNOWN
|
||||
# 首条或缺前一根
|
||||
|
||||
@@ -43,6 +43,29 @@ class ChanSEG():
|
||||
self.macd_hist = macd_hist
|
||||
def set_macd_div(self, macd_div):
|
||||
self.macd_div = macd_div
|
||||
def cal_macdhist(self):
|
||||
# 线段面积 = 同向笔 MACD 柱面积之和(与笔面积口径一致)
|
||||
acc = 0.0
|
||||
seg_dir_name = getattr(self.dir, 'name', None)
|
||||
for bi in self.bi_list:
|
||||
if bi is None:
|
||||
continue
|
||||
if getattr(getattr(bi, 'dir', None), 'name', None) != seg_dir_name:
|
||||
continue
|
||||
acc += float(bi.macd_hist or 0)
|
||||
self.macd_hist = acc
|
||||
return acc
|
||||
def cal_macd_div(self):
|
||||
# 与前一个同向线段比面积:seg.pre 是反向邻段,pre.pre 才是同向
|
||||
self.macd_div = 0.0
|
||||
prev = self.pre.pre if self.pre and self.pre.pre else None
|
||||
if prev is None:
|
||||
return 0.0
|
||||
prev_hist = float(prev.macd_hist or 0)
|
||||
if prev_hist == 0:
|
||||
return 0.0
|
||||
self.macd_div = float(self.macd_hist or 0) / prev_hist
|
||||
return self.macd_div
|
||||
def set_end_bi(self, bi: ChanBI, sure_bi: ChanBI):
|
||||
self.end_bi = bi
|
||||
if bi and bi.is_sure:
|
||||
|
||||
@@ -0,0 +1,196 @@
|
||||
"""Drop-in replacement for the `talib.abstract` calls this project makes.
|
||||
|
||||
Same call signatures, same column names, same NaN warm-up lengths, so call
|
||||
sites only change their import line.
|
||||
|
||||
Only what the codebase actually uses is implemented: SMA, MA, EMA, RSI, ATR,
|
||||
MACD, BBANDS. Numerical agreement with TA-Lib is enforced by
|
||||
`chanlun/tests/test_ta_compat.py`, which skips when talib is absent.
|
||||
|
||||
The warm-up conventions below are TA-Lib's, not the textbook ones, and they
|
||||
differ between functions — getting them wrong shifts every downstream Chan
|
||||
structure by a bar:
|
||||
|
||||
SMA/BBANDS first value at index period-1
|
||||
EMA seeded with the SMA of the first `period` values, at index period-1
|
||||
RSI/ATR Wilder smoothing (alpha = 1/period), first value at index period
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
__all__ = ["SMA", "MA", "EMA", "RSI", "ATR", "MACD", "BBANDS"]
|
||||
|
||||
|
||||
def _series(data, price: str = "close") -> pd.Series:
|
||||
"""Accept the abstract-API shapes: DataFrame, Series, or ndarray."""
|
||||
if isinstance(data, pd.DataFrame):
|
||||
return data[price].astype(float)
|
||||
if isinstance(data, pd.Series):
|
||||
return data.astype(float)
|
||||
return pd.Series(np.asarray(data, dtype=float))
|
||||
|
||||
|
||||
def _recursive(values: np.ndarray, seed: float, start: int, alpha: float, n: int) -> np.ndarray:
|
||||
"""out[start] = seed; out[i] = alpha*values[i] + (1-alpha)*out[i-1].
|
||||
|
||||
Delegates the recursion to pandas' C implementation rather than a Python
|
||||
loop — `research/` runs this over long histories.
|
||||
"""
|
||||
out = np.full(n, np.nan)
|
||||
if start >= n:
|
||||
return out
|
||||
tail = values[start:].astype(float).copy()
|
||||
tail[0] = seed
|
||||
out[start:] = pd.Series(tail).ewm(alpha=alpha, adjust=False).mean().to_numpy()
|
||||
return out
|
||||
|
||||
|
||||
def SMA(data, timeperiod: int = 30, price: str = "close") -> pd.Series:
|
||||
s = _series(data, price)
|
||||
return s.rolling(window=timeperiod, min_periods=timeperiod).mean()
|
||||
|
||||
|
||||
def MA(data, timeperiod: int = 30, matype: int = 0, price: str = "close") -> pd.Series:
|
||||
if matype != 0:
|
||||
raise NotImplementedError(f"MA matype={matype} is not used by this codebase")
|
||||
return SMA(data, timeperiod, price=price)
|
||||
|
||||
|
||||
def _ema(x: np.ndarray, period: int, start: int) -> np.ndarray:
|
||||
"""EMA whose first output lands on `start`, seeded by the SMA of the
|
||||
`period` values ending there.
|
||||
|
||||
`start` is a parameter because MACD needs the fast EMA to begin later than
|
||||
it naturally would; see the note in MACD().
|
||||
"""
|
||||
n = x.size
|
||||
if n <= start or start < period - 1:
|
||||
return np.full(n, np.nan)
|
||||
seed = x[start - period + 1: start + 1].mean()
|
||||
return _recursive(x, seed, start, 2.0 / (period + 1.0), n)
|
||||
|
||||
|
||||
def EMA(data, timeperiod: int = 30, price: str = "close") -> pd.Series:
|
||||
s = _series(data, price)
|
||||
x = s.to_numpy(dtype=float)
|
||||
return pd.Series(_ema(x, timeperiod, timeperiod - 1), index=s.index)
|
||||
|
||||
|
||||
def RSI(data, timeperiod: int = 14, price: str = "close") -> pd.Series:
|
||||
s = _series(data, price)
|
||||
x = s.to_numpy(dtype=float)
|
||||
n = x.size
|
||||
out = np.full(n, np.nan)
|
||||
if n <= timeperiod:
|
||||
return pd.Series(out, index=s.index)
|
||||
|
||||
delta = np.diff(x)
|
||||
gain = np.where(delta > 0.0, delta, 0.0)
|
||||
loss = np.where(delta < 0.0, -delta, 0.0)
|
||||
|
||||
# delta[k] corresponds to bar k+1, so the first `timeperiod` deltas seed bar `timeperiod`.
|
||||
alpha = 1.0 / timeperiod
|
||||
avg_gain = _recursive(gain, gain[:timeperiod].mean(), timeperiod - 1, alpha, n - 1)
|
||||
avg_loss = _recursive(loss, loss[:timeperiod].mean(), timeperiod - 1, alpha, n - 1)
|
||||
|
||||
ag = avg_gain[timeperiod - 1:]
|
||||
al = avg_loss[timeperiod - 1:]
|
||||
with np.errstate(divide="ignore", invalid="ignore"):
|
||||
rsi = np.where(al == 0.0, 100.0, 100.0 - 100.0 / (1.0 + ag / al))
|
||||
out[timeperiod:] = rsi
|
||||
return pd.Series(out, index=s.index)
|
||||
|
||||
|
||||
def ATR(data, timeperiod: int = 14) -> pd.Series:
|
||||
if not isinstance(data, pd.DataFrame):
|
||||
raise TypeError("ATR needs a DataFrame with high/low/close")
|
||||
high = data["high"].to_numpy(dtype=float)
|
||||
low = data["low"].to_numpy(dtype=float)
|
||||
close = data["close"].to_numpy(dtype=float)
|
||||
n = high.size
|
||||
out = np.full(n, np.nan)
|
||||
if n <= timeperiod:
|
||||
return pd.Series(out, index=data.index)
|
||||
|
||||
prev_close = close[:-1]
|
||||
tr = np.maximum.reduce([
|
||||
high[1:] - low[1:],
|
||||
np.abs(high[1:] - prev_close),
|
||||
np.abs(low[1:] - prev_close),
|
||||
])
|
||||
|
||||
# tr[k] is bar k+1; the first `timeperiod` true ranges seed bar `timeperiod`.
|
||||
smoothed = _recursive(tr, tr[:timeperiod].mean(), timeperiod - 1, 1.0 / timeperiod, n - 1)
|
||||
out[timeperiod:] = smoothed[timeperiod - 1:]
|
||||
return pd.Series(out, index=data.index)
|
||||
|
||||
|
||||
def MACD(
|
||||
data,
|
||||
fastperiod: int = 12,
|
||||
slowperiod: int = 26,
|
||||
signalperiod: int = 9,
|
||||
price: str = "close",
|
||||
) -> pd.DataFrame:
|
||||
if slowperiod < fastperiod:
|
||||
fastperiod, slowperiod = slowperiod, fastperiod
|
||||
|
||||
s = _series(data, price)
|
||||
x = s.to_numpy(dtype=float)
|
||||
n = x.size
|
||||
|
||||
macd = np.full(n, np.nan)
|
||||
signal = np.full(n, np.nan)
|
||||
hist = np.full(n, np.nan)
|
||||
empty = pd.DataFrame({"macd": macd, "macdsignal": signal, "macdhist": hist}, index=s.index)
|
||||
|
||||
# Both EMAs emit their first value on the same bar. That makes the slow one
|
||||
# ordinary, but re-seeds the fast one from the SMA of the `fastperiod`
|
||||
# values ending there instead of carrying the recursion forward from bar
|
||||
# fastperiod-1 — the two disagree by ~0.2 on a 100-priced series.
|
||||
macd_start = slowperiod - 1
|
||||
if n <= macd_start:
|
||||
return empty
|
||||
line = _ema(x, fastperiod, macd_start) - _ema(x, slowperiod, macd_start)
|
||||
|
||||
# The signal EMA runs over the MACD line, so everything shifts by another
|
||||
# signalperiod-1 bars, and TA-Lib trims the MACD line to match.
|
||||
valid = line[macd_start:]
|
||||
if valid.size < signalperiod:
|
||||
return empty
|
||||
sig = _recursive(
|
||||
valid, valid[:signalperiod].mean(), signalperiod - 1, 2.0 / (signalperiod + 1.0), valid.size
|
||||
)
|
||||
start = macd_start + signalperiod - 1
|
||||
macd[start:] = line[start:]
|
||||
signal[macd_start:] = sig
|
||||
hist = macd - signal
|
||||
|
||||
return pd.DataFrame({"macd": macd, "macdsignal": signal, "macdhist": hist}, index=s.index)
|
||||
|
||||
|
||||
def BBANDS(
|
||||
data,
|
||||
timeperiod: int = 5,
|
||||
nbdevup: float = 2.0,
|
||||
nbdevdn: float = 2.0,
|
||||
matype: int = 0,
|
||||
price: str = "close",
|
||||
) -> pd.DataFrame:
|
||||
if matype != 0:
|
||||
raise NotImplementedError(f"BBANDS matype={matype} is not used by this codebase")
|
||||
s = _series(data, price)
|
||||
middle = s.rolling(window=timeperiod, min_periods=timeperiod).mean()
|
||||
# TA-Lib uses the population standard deviation.
|
||||
std = s.rolling(window=timeperiod, min_periods=timeperiod).std(ddof=0)
|
||||
return pd.DataFrame(
|
||||
{
|
||||
"upperband": middle + nbdevup * std,
|
||||
"middleband": middle,
|
||||
"lowerband": middle - nbdevdn * std,
|
||||
},
|
||||
index=s.index,
|
||||
)
|
||||
@@ -6,9 +6,7 @@ from decimal import Decimal
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import talib.abstract as ta
|
||||
from pandas import DataFrame
|
||||
from technical.util import resample_to_interval
|
||||
|
||||
from chanlun.core.ChanBI import ChanBI
|
||||
from chanlun.core.ChanBIZS import ChanBIZS
|
||||
@@ -670,13 +668,22 @@ class BiBuilderMixin:
|
||||
return bi_list
|
||||
|
||||
def check_top_fx(self, last_bottom, klc):
|
||||
if (last_bottom.high > klc.pre.low or last_bottom.high > klc.next.low) and (klc.index - last_bottom.index < 100):
|
||||
#严格笔
|
||||
#last_bottom_high = max(last_bottom.high, last_bottom.pre.high, last_bottom.next.high)
|
||||
#缠论原著笔
|
||||
last_bottom_high = last_bottom.high
|
||||
if (last_bottom_high > klc.pre.low or last_bottom_high > klc.next.low) and (klc.index - last_bottom.index < 100):
|
||||
#print(klc.end_time, "check_top_fx False")
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def check_bottom_fx(self, last_top, klc):
|
||||
if (last_top.low < klc.pre.high or last_top.low < klc.next.high) and (klc.index - last_top.index < 100):
|
||||
#严格笔
|
||||
#last_top_low = min(last_top.low, last_top.pre.low, last_top.next.low)
|
||||
#缠论原著笔
|
||||
last_top_low = last_top.low
|
||||
if (last_top_low < klc.pre.high or last_top_low < klc.next.high) and (klc.index - last_top.index < 100):
|
||||
return False
|
||||
return True
|
||||
# 线段内的中枢
|
||||
|
||||
@@ -6,9 +6,7 @@ from decimal import Decimal
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import talib.abstract as ta
|
||||
from pandas import DataFrame
|
||||
from technical.util import resample_to_interval
|
||||
|
||||
from chanlun.core.ChanBI import ChanBI
|
||||
from chanlun.core.ChanBIZS import ChanBIZS
|
||||
|
||||
@@ -0,0 +1,148 @@
|
||||
"""第四类买卖点(B4/S4)接入 TF_DF。
|
||||
|
||||
判定逻辑全在 chanlun/analysis/fast_bsp.py,这里只负责把引擎的中枢/K线喂进去,
|
||||
再把结果包成 ChanFastBSP。
|
||||
|
||||
刻意不在 init_TF_DF 里默认计算:现有构造路径的开销保持不变,由调用方按需触发。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
import pandas as pd
|
||||
|
||||
from chanlun.analysis.fast_bsp import (
|
||||
add_zone_ladder,
|
||||
attach_htf_agree,
|
||||
attach_zone_ladder,
|
||||
ensure_timestamp,
|
||||
find_fast_bsp3,
|
||||
htf_fx_timeline,
|
||||
zones_from_zs_list,
|
||||
)
|
||||
from chanlun.core.ChanEnum import Chan_BSP_DIR
|
||||
from chanlun.core.ChanFastBSP import ChanFastBSP
|
||||
|
||||
# 区间套配对:小级别出中枢与买卖点,大级别只出分型定方向。
|
||||
# 取值来自 research/HANDOFF.md §1.5,是回测里实际用过的组合。
|
||||
FAST_BSP_HTF_PAIR = {
|
||||
'1m': '5m',
|
||||
'5m': '30m',
|
||||
'15m': '1h',
|
||||
'30m': '2h',
|
||||
}
|
||||
|
||||
# 未列入配对表的周期回落到这个倍数
|
||||
FAST_BSP_HTF_FALLBACK_RATIO = 4
|
||||
|
||||
_TF_UNIT_MINUTES = {'m': 1, 'h': 60, 'd': 1440, 'w': 10080}
|
||||
|
||||
|
||||
def timeframe_minutes(tf: str) -> int | None:
|
||||
"""'30m' -> 30,'2h' -> 120。无法解析时返回 None。"""
|
||||
if not tf:
|
||||
return None
|
||||
m = re.fullmatch(r'(\d+)\s*([mhdw])', str(tf).strip().lower())
|
||||
if not m:
|
||||
return None
|
||||
return int(m.group(1)) * _TF_UNIT_MINUTES[m.group(2)]
|
||||
|
||||
|
||||
def resolve_htf(tf: str) -> tuple[str, int] | None:
|
||||
"""给小级别找配套的大级别,返回 (标签, 分钟数)。"""
|
||||
minutes = timeframe_minutes(tf)
|
||||
if minutes is None:
|
||||
return None
|
||||
paired = FAST_BSP_HTF_PAIR.get(str(tf).strip().lower())
|
||||
if paired:
|
||||
return paired, timeframe_minutes(paired)
|
||||
return f'{minutes * FAST_BSP_HTF_FALLBACK_RATIO}m', minutes * FAST_BSP_HTF_FALLBACK_RATIO
|
||||
|
||||
|
||||
class FastBspBuilderMixin:
|
||||
def build_fast_bsp_htf(self, df, timeframe=None):
|
||||
"""对同一份 df 重采样得到大级别,不额外拉数据。
|
||||
|
||||
大级别只用来取分型方向,样本太少就没有过滤意义,故重采样后不足 60 根时放弃。
|
||||
"""
|
||||
tf = timeframe or getattr(self, 'timeframe', None)
|
||||
htf = resolve_htf(tf)
|
||||
ltf_minutes = timeframe_minutes(tf)
|
||||
if htf is None or not ltf_minutes:
|
||||
return None
|
||||
label, minutes = htf
|
||||
if not minutes or len(df) * ltf_minutes < minutes * 60:
|
||||
return None
|
||||
try:
|
||||
from chanlun.pipeline.timeframe import TF_DF
|
||||
|
||||
return TF_DF(df, minutes, label)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def cal_fast_bsp(self, df=None, bi_zs_list=None, htf_chan=None, with_htf=True,
|
||||
timeframe=None, **kw):
|
||||
"""算第四类买卖点,返回 ChanFastBSP 列表。
|
||||
|
||||
bi_zs_list 传入已算好的 pure 笔中枢可免去重复计算。
|
||||
with_htf=False 时跳过大级别构建,只留 ladder_ok 这一个过滤标志。
|
||||
kw 透传给 find_fast_bsp3(scan / pullback_win / tol / require_touch 等)。
|
||||
"""
|
||||
src = df if df is not None else getattr(self, 'dataframe', None)
|
||||
if src is None or len(src) == 0:
|
||||
self.fast_bsp_list = []
|
||||
return self.fast_bsp_list
|
||||
|
||||
src = ensure_timestamp(src)
|
||||
if bi_zs_list is None:
|
||||
bi_zs_list = getattr(self, 'bi_zs_list', None)
|
||||
if not bi_zs_list:
|
||||
bi_zs_list = self.cal_bi_zs_list_pure(self.cal_bi_list(self.get_klc_list(self.cal_kl_data(src))))
|
||||
|
||||
zones = zones_from_zs_list(bi_zs_list, src)
|
||||
if zones.empty:
|
||||
self.fast_bsp_list = []
|
||||
return self.fast_bsp_list
|
||||
|
||||
zones = add_zone_ladder(zones)
|
||||
sig = find_fast_bsp3(src, zones, **kw)
|
||||
if sig.empty:
|
||||
self.fast_bsp_list = []
|
||||
return self.fast_bsp_list
|
||||
|
||||
sig = attach_zone_ladder(sig, zones)
|
||||
|
||||
if with_htf:
|
||||
if htf_chan is None:
|
||||
htf_chan = self.build_fast_bsp_htf(src, timeframe)
|
||||
sig = attach_htf_agree(sig, src, htf_fx_timeline(htf_chan) if htf_chan is not None else pd.DataFrame())
|
||||
else:
|
||||
sig['htf_dir'] = None
|
||||
sig['htf_agree'] = None
|
||||
|
||||
times = src['date'].dt.strftime('%Y-%m-%d %H:%M:%S').to_numpy()
|
||||
close = src['close'].to_numpy(dtype=float)
|
||||
|
||||
out = []
|
||||
for r in sig.itertuples(index=False):
|
||||
entry_idx = int(r.entry_idx)
|
||||
agree = getattr(r, 'htf_agree', None)
|
||||
htf_dir = getattr(r, 'htf_dir', None)
|
||||
out.append(ChanFastBSP(
|
||||
time=times[entry_idx],
|
||||
price=close[entry_idx],
|
||||
ddir=Chan_BSP_DIR.BUY if r.direction == 1 else Chan_BSP_DIR.SELL,
|
||||
entry_idx=entry_idx,
|
||||
bo_time=times[int(r.bo_idx)],
|
||||
pb_time=times[int(r.pb_idx)] if r.pb_idx == r.pb_idx else None,
|
||||
lag=r.lag,
|
||||
depth=r.depth,
|
||||
zg=r.zg,
|
||||
zd=r.zd,
|
||||
occ=r.occ,
|
||||
htf_dir=None if htf_dir is None or htf_dir != htf_dir else int(htf_dir),
|
||||
htf_agree=None if agree is None or agree != agree else bool(agree),
|
||||
ladder_ok=bool(r.ladder_ok),
|
||||
))
|
||||
self.fast_bsp_list = out
|
||||
return out
|
||||
@@ -9,7 +9,7 @@ from datetime import datetime
|
||||
|
||||
import pandas as pd
|
||||
from pandas import DataFrame
|
||||
from technical.util import resample_to_interval
|
||||
from chanlun.pipeline.resample import resample_to_interval
|
||||
|
||||
from chanlun.core.ChanEnum import Chan_FX_TYPE, Chan_KLC_FX, Chan_KLC_STATE
|
||||
from chanlun.core.ChanKLU import ChanKLU
|
||||
|
||||
@@ -6,9 +6,8 @@ from decimal import Decimal
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import talib.abstract as ta
|
||||
from chanlun.indicators import ta
|
||||
from pandas import DataFrame
|
||||
from technical.util import resample_to_interval
|
||||
|
||||
from chanlun.core.ChanBI import ChanBI
|
||||
from chanlun.core.ChanBIZS import ChanBIZS
|
||||
@@ -55,8 +54,14 @@ class IndicatorsBuilderMixin:
|
||||
return None
|
||||
|
||||
def add_indicators(self, df):
|
||||
fast = 26
|
||||
slow = 52
|
||||
"""算指标并一次性挂到 df 上。
|
||||
|
||||
这里不逐列 `df['x'] = ...`:那样每一列都触发一次 BlockManager 插入,
|
||||
30 多列的开销比全部 TA 计算本身还大(2001 行实测 TA 合计 2.5ms,
|
||||
逐列赋值 3.6ms)。增量路径每根都要走一遍,这笔开销是白付的。
|
||||
"""
|
||||
fast = 12
|
||||
slow = 26
|
||||
period = 9
|
||||
macd = ta.MACD(df, fastperiod=fast, slowperiod=slow, signalperiod=period)
|
||||
bb365 = ta.BBANDS(df, timeperiod=365, nbdevup=3.0, nbdevdn=3.0, matype=0)
|
||||
@@ -76,40 +81,43 @@ class IndicatorsBuilderMixin:
|
||||
bbp30 = (df['close'] - bb30['lowerband']) / (bb30['upperband'] - bb30['lowerband'])
|
||||
bbp302 = (df['close'] - bb302['lowerband']) / (bb302['upperband'] - bb302['lowerband'])
|
||||
bbp2633 = (df['close'] - bb2633['lowerband']) / (bb2633['upperband'] - bb2633['lowerband'])
|
||||
df['bb2633upper'] = bb2633['upperband']
|
||||
df['bb2633lower'] = bb2633['lowerband']
|
||||
df['bbp2633'] = bbp2633
|
||||
df['bb2633middle'] = bb2633['middleband']
|
||||
df['atr'] = ta.ATR(df, timeperiod=14)
|
||||
df['bbup365'] = bb365['upperband']
|
||||
df['bblow365'] = bb365['lowerband']
|
||||
df['bbp365'] = bbp365
|
||||
df['bbup120'] = bb120['upperband']
|
||||
df['bblow120'] = bb120['lowerband']
|
||||
df['bbp120'] = bbp120
|
||||
df['bbup30'] = bb30['upperband']
|
||||
df['bblow30'] = bb30['lowerband']
|
||||
df['bbmiddle30'] = bb30_middle # 添加bb30中轨
|
||||
df['bbp30'] = bbp30
|
||||
df['bbup302'] = bb302['upperband']
|
||||
df['bblow302'] = bb302['lowerband']
|
||||
df['bbp302'] = bbp302
|
||||
df['macd'] = macd['macd']
|
||||
df['macdsignal'] = macd['macdsignal']
|
||||
df['macdhist'] = macd['macdhist']
|
||||
df['ema5'] = ta.EMA(df, timeperiod=5)
|
||||
df['ema10'] = ta.EMA(df, timeperiod=10)
|
||||
df['ema24'] = ta.EMA(df, timeperiod=24)
|
||||
df['ema52'] = ta.EMA(df, timeperiod=52)
|
||||
df['ema104'] = ta.EMA(df, timeperiod=104)
|
||||
df['ema156'] = ta.EMA(df, timeperiod=156)
|
||||
df['ema208'] = ta.EMA(df, timeperiod=208)
|
||||
df['ema26'] = ta.EMA(df, timeperiod=26)
|
||||
df['ema13'] = ta.EMA(df, timeperiod=13)
|
||||
df['ema7'] = ta.EMA(df, timeperiod=7)
|
||||
df['rsi'] = ta.RSI(df, timeperiod=14)
|
||||
df['volume_ratio'] = self.cal_volume_ratio(df)
|
||||
return df
|
||||
cols = {
|
||||
'bb2633upper': bb2633['upperband'],
|
||||
'bb2633lower': bb2633['lowerband'],
|
||||
'bbp2633': bbp2633,
|
||||
'bb2633middle': bb2633['middleband'],
|
||||
'atr': ta.ATR(df, timeperiod=14),
|
||||
'bbup365': bb365['upperband'],
|
||||
'bblow365': bb365['lowerband'],
|
||||
'bbp365': bbp365,
|
||||
'bbup120': bb120['upperband'],
|
||||
'bblow120': bb120['lowerband'],
|
||||
'bbp120': bbp120,
|
||||
'bbup30': bb30['upperband'],
|
||||
'bblow30': bb30['lowerband'],
|
||||
'bbmiddle30': bb30_middle,
|
||||
'bbp30': bbp30,
|
||||
'bbup302': bb302['upperband'],
|
||||
'bblow302': bb302['lowerband'],
|
||||
'bbp302': bbp302,
|
||||
'macd': macd['macd'],
|
||||
'macdsignal': macd['macdsignal'],
|
||||
'macdhist': macd['macdhist'],
|
||||
}
|
||||
for _p, _n in ((5, 'ema5'), (10, 'ema10'), (24, 'ema24'), (52, 'ema52'),
|
||||
(104, 'ema104'), (156, 'ema156'), (208, 'ema208'),
|
||||
(26, 'ema26'), (13, 'ema13'), (7, 'ema7')):
|
||||
cols[_n] = ta.EMA(df, timeperiod=_p)
|
||||
cols['rsi'] = ta.RSI(df, timeperiod=14)
|
||||
cols['volume_ratio'] = self.cal_volume_ratio(df)
|
||||
|
||||
# 重复调用(增量路径每根都会)时先摘掉旧列,否则 concat 出重名列。
|
||||
# 摘掉再接回末尾,列序与逐列覆盖的结果一致。
|
||||
new = pd.DataFrame(cols, index=df.index)
|
||||
dup = [c for c in new.columns if c in df.columns]
|
||||
if dup:
|
||||
df = df.drop(columns=dup)
|
||||
return pd.concat([df, new], axis=1)
|
||||
|
||||
def get_ema_state(self, dataframe):
|
||||
klu_list = self.get_klu_list(dataframe)
|
||||
|
||||
@@ -6,9 +6,7 @@ from decimal import Decimal
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import talib.abstract as ta
|
||||
from pandas import DataFrame
|
||||
from technical.util import resample_to_interval
|
||||
|
||||
from chanlun.core.ChanBI import ChanBI
|
||||
from chanlun.core.ChanBIZS import ChanBIZS
|
||||
@@ -130,60 +128,77 @@ class KlineBuilderMixin:
|
||||
return Chan_FX_TYPE.UNKNOWN
|
||||
|
||||
def check_fx_pattern(self, klc):
|
||||
"""给分型前后三根 KLC 的裸 K 打 pattern 标记。
|
||||
|
||||
原本还会把 `klu.to_string()` 拼成一个字符串——那是给下面那行注释掉的
|
||||
print 用的,拼完就丢。它在 cal_bi_list 的内层,2000 根上要跑近三万次
|
||||
f-string + 六万次 enum 格式化,是纯废动作,已删。
|
||||
|
||||
`klu.pattern` 只被 cal_klu_pattern 自己的双 K / 三 K 判定读,
|
||||
不出这个模块,也不进 web 序列化。所以 lean 下整个调用可跳。
|
||||
"""
|
||||
if getattr(self, 'lean', False):
|
||||
return
|
||||
klu_list = klc.pre.klu_list + klc.klu_list + klc.next.klu_list
|
||||
|
||||
self.cal_klu_pattern(klu_list)
|
||||
p = ""
|
||||
for klu in klu_list:
|
||||
p += klu.to_string()
|
||||
#print(p)
|
||||
|
||||
def cal_volume_ratio(self, dataframe, window=10):
|
||||
df = dataframe.copy()
|
||||
# 计算过去N根K线的平均成交量
|
||||
df['avg_volume'] = df['volume'].rolling(window=window).mean()
|
||||
# 计算量比
|
||||
df['volume_ratio'] = df['volume'] / df['avg_volume']
|
||||
# 填充缺失值(前N根K线)
|
||||
df['volume_ratio'] = df['volume_ratio'].fillna(1.0)
|
||||
return df['volume_ratio']
|
||||
"""当根量 / 前 window 根均量。前 window-1 根无基准,填 1.0。
|
||||
|
||||
原写法先 `dataframe.copy()` 再挂两列——为算一列 rolling 复制了整张
|
||||
四十列的表。直接在 Series 上算,结果逐值相同。
|
||||
"""
|
||||
vol = dataframe['volume']
|
||||
return (vol / vol.rolling(window=window).mean()).fillna(1.0).rename('volume_ratio')
|
||||
|
||||
def cal_kl_data(self, dataframe:DataFrame):
|
||||
fields = "time,open,high,low,close,volume"
|
||||
"""按行构造 KLU 链。
|
||||
|
||||
这里刻意不用 `dataframe.iloc[i]`:那会为每一根新建一个几十列的 Series,
|
||||
随后 set_indicators 再在其上做几十次逐键查找。实测这两件事合计占 TF_DF
|
||||
构建耗时的 96%。改为先把用到的列取成 ndarray,循环里只做整数下标访问。
|
||||
"""
|
||||
n = len(dataframe)
|
||||
if n == 0:
|
||||
return []
|
||||
|
||||
times = self._format_times(dataframe['date'])
|
||||
o_a = dataframe['open'].to_numpy(dtype=float)
|
||||
h_a = dataframe['high'].to_numpy(dtype=float)
|
||||
l_a = dataframe['low'].to_numpy(dtype=float)
|
||||
c_a = dataframe['close'].to_numpy(dtype=float)
|
||||
v_a = dataframe['volume'].to_numpy(dtype=float)
|
||||
|
||||
has_ind = 'macd' in dataframe.columns
|
||||
ind_cols = {}
|
||||
if has_ind:
|
||||
for _attr, col in ChanKLU.INDICATOR_FIELDS:
|
||||
if col in dataframe.columns:
|
||||
ind_cols[col] = dataframe[col].to_numpy(dtype=float)
|
||||
|
||||
klu_list = []
|
||||
last_klu = None
|
||||
for i in range(0, len(dataframe)):
|
||||
item = dataframe.iloc[i]
|
||||
date = item['date']
|
||||
o = item['open']
|
||||
h = item['high']
|
||||
l = item['low']
|
||||
c = item['close']
|
||||
v = item['volume']
|
||||
# time_obj = date.fromtimestamp(date)
|
||||
# date = date + timedelta(hours=8)
|
||||
time_str = date.strftime('%Y-%m-%d %H:%M:%S')
|
||||
item_data = [
|
||||
time_str,
|
||||
o,
|
||||
h,
|
||||
l,
|
||||
c,
|
||||
v
|
||||
]
|
||||
# klu = KLU(self.create_item_dict(item_data, GetColumnNameFromFieldList(fields)))
|
||||
klu = ChanKLU(time_str, o, h, l, c, v)
|
||||
# print(klu.time, klu.open, klu.high, klu.low, klu.close, klu.volume)
|
||||
for i in range(n):
|
||||
klu = ChanKLU(times[i], o_a[i], h_a[i], l_a[i], c_a[i], v_a[i])
|
||||
klu.set_idx(i)
|
||||
klu_list.append(klu)
|
||||
if last_klu:
|
||||
last_klu.set_next(klu)
|
||||
klu.set_pre(last_klu)
|
||||
last_klu = klu
|
||||
if 'macd' in item:
|
||||
klu.set_indicators(item)
|
||||
if has_ind:
|
||||
klu.set_indicators_from(ind_cols, i)
|
||||
return klu_list
|
||||
|
||||
@staticmethod
|
||||
def _format_times(col):
|
||||
"""向量化 strftime。非 datetime 列(少见)退回逐个格式化。"""
|
||||
fmt = '%Y-%m-%d %H:%M:%S'
|
||||
try:
|
||||
return col.dt.strftime(fmt).to_numpy()
|
||||
except AttributeError:
|
||||
return np.array([d.strftime(fmt) for d in col], dtype=object)
|
||||
|
||||
def get_kl_data(self, dataframe:DataFrame):
|
||||
return self.cal_kl_data(dataframe)
|
||||
|
||||
@@ -226,35 +241,25 @@ class KlineBuilderMixin:
|
||||
|
||||
def get_klc_list(self, klu_list):
|
||||
klc_list = []
|
||||
last_klu = None
|
||||
# ChanMACD.__init__ 已调用 cal_macd_state,切勿再调一次(会重复堆积 seg/unittf)
|
||||
macd = ChanMACD(klu_list)
|
||||
klu_list = macd.klu_list
|
||||
self._last_chan_macd = macd
|
||||
ema_up_list = []
|
||||
ema_down_list = []
|
||||
ema_up_count = 0
|
||||
ema_down_count = 0
|
||||
# ChanMACD.__init__ 已调用 cal_macd_state,切勿再调一次(会重复堆积 seg/unittf)。
|
||||
# lean 模式跳过整套 MACD 状态机:它只服务于 bsp/背驰/web 展示,笔与中枢不依赖它。
|
||||
if getattr(self, 'lean', False):
|
||||
self._last_chan_macd = None
|
||||
else:
|
||||
macd = ChanMACD(klu_list)
|
||||
klu_list = macd.klu_list
|
||||
self._last_chan_macd = macd
|
||||
|
||||
last_klu = None
|
||||
for klu in klu_list:
|
||||
ema = klu.ema52
|
||||
last_ema = last_klu.ema52 if last_klu else 0
|
||||
if klu.close >= ema:
|
||||
ema_up_count += 1
|
||||
elif klu.close < ema:
|
||||
ema_down_count += 1
|
||||
if last_klu and last_klu.close >= last_ema and klu.close < ema:
|
||||
ema_up_list.append(ema_up_count)
|
||||
#print(last_klu.time, ema_up_count, "UP END")
|
||||
ema_up_count = 0
|
||||
elif last_klu and last_klu.close < last_ema and klu.close >= ema:
|
||||
ema_down_list.append(ema_down_count)
|
||||
#print(last_klu.time, ema_down_count, "DOWN END")
|
||||
ema_down_count = 0
|
||||
self._push_klu_into_klc_list(klc_list, klu, last_klu)
|
||||
last_klu = klu
|
||||
klc_list = self.cal_trend(klc_list)
|
||||
#print(ema52_up_list, ema52_down_list)
|
||||
# cal_trend 只产出 klc.trend,而全仓只有它自己(经 prev_klcs 读自身序列
|
||||
# 状态)、一个 __str__ 和 web 的 klc_trend 图层消费它——笔与中枢不读。
|
||||
# 增量路径的 klc 其 trend 恒为 UNKNOWN 却与批量构建逐字段相同,是实证。
|
||||
# 所以 lean 下可跳;web 走非 lean,图层不受影响。
|
||||
if not getattr(self, 'lean', False):
|
||||
klc_list = self.cal_trend(klc_list)
|
||||
return klc_list
|
||||
|
||||
|
||||
|
||||
@@ -6,9 +6,7 @@ from decimal import Decimal
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import talib.abstract as ta
|
||||
from pandas import DataFrame
|
||||
from technical.util import resample_to_interval
|
||||
|
||||
from chanlun.core.ChanBI import ChanBI
|
||||
from chanlun.core.ChanBIZS import ChanBIZS
|
||||
|
||||
@@ -6,9 +6,7 @@ from decimal import Decimal
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import talib.abstract as ta
|
||||
from pandas import DataFrame
|
||||
from technical.util import resample_to_interval
|
||||
|
||||
from chanlun.core.ChanBI import ChanBI
|
||||
from chanlun.core.ChanBIZS import ChanBIZS
|
||||
|
||||
@@ -17,9 +17,7 @@ from chanlun.core.ChanSBI import ChanSBI
|
||||
from chanlun.core.ChanSEG import ChanSEG
|
||||
from chanlun.core.ChanZS import ChanZS
|
||||
from chanlun.core.ChanBSP import ChanBSP
|
||||
import talib.abstract as ta
|
||||
import pandas as pd
|
||||
from technical.util import resample_to_interval
|
||||
from decimal import Decimal
|
||||
import numpy as np
|
||||
from chanlun.indicators.ChanMACD import ChanMACD
|
||||
@@ -171,6 +169,8 @@ class ChanLun():
|
||||
return self.tf_df.find_second_bsp(bi_list, first_bsp_list)
|
||||
def find_all_bsp(self, bi_list, bi_zs_list):
|
||||
return self.tf_df.find_all_bsp(bi_list, bi_zs_list)
|
||||
def cal_fast_bsp(self, df=None, bi_zs_list=None, htf_chan=None, with_htf=True, timeframe=None, **kw):
|
||||
return self.tf_df.cal_fast_bsp(df, bi_zs_list, htf_chan, with_htf, timeframe, **kw)
|
||||
def get_zs_list(self, bi_list, seg_list):
|
||||
return self.tf_df.get_zs_list(bi_list, seg_list)
|
||||
def cal_bi_zs(self, seg_list):
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
"""OHLCV resampling — replaces `technical.util.resample_to_interval`.
|
||||
|
||||
That was the only symbol this project imported from `technical`, which in turn
|
||||
pulled in the freqtrade dependency chain. Behaviour is preserved exactly,
|
||||
including the left-labelled bins (rows are candle *open* times) and the
|
||||
`dropna()` that drops empty intervals.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pandas as pd
|
||||
|
||||
__all__ = ["TICKER_INTERVAL_MINUTES", "resample_to_interval"]
|
||||
|
||||
TICKER_INTERVAL_MINUTES: dict[str, int] = {
|
||||
"1m": 1,
|
||||
"5m": 5,
|
||||
"15m": 15,
|
||||
"30m": 30,
|
||||
"1h": 60,
|
||||
"60m": 60,
|
||||
"2h": 120,
|
||||
"4h": 240,
|
||||
"6h": 360,
|
||||
"12h": 720,
|
||||
"1d": 1440,
|
||||
"1w": 10080,
|
||||
}
|
||||
|
||||
_OHLC_AGG = {
|
||||
"open": "first",
|
||||
"high": "max",
|
||||
"low": "min",
|
||||
"close": "last",
|
||||
"volume": "sum",
|
||||
}
|
||||
|
||||
|
||||
def resample_to_interval(dataframe: pd.DataFrame, interval: int | str) -> pd.DataFrame:
|
||||
"""Resample OHLCV rows to `interval` minutes (or a timeframe string).
|
||||
|
||||
Merging the result back onto a finer frame requires care to avoid lookahead
|
||||
bias; this function only resamples.
|
||||
"""
|
||||
if isinstance(interval, str):
|
||||
interval = TICKER_INTERVAL_MINUTES[interval]
|
||||
|
||||
df = dataframe.copy()
|
||||
df = df.set_index(pd.DatetimeIndex(df["date"]))
|
||||
df = df.resample(f"{interval}min", label="left").agg(_OHLC_AGG).dropna()
|
||||
df.reset_index(inplace=True)
|
||||
return df
|
||||
@@ -2,9 +2,8 @@ from datetime import timedelta
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import talib.abstract as ta
|
||||
from pandas import DataFrame
|
||||
from technical.util import resample_to_interval
|
||||
from chanlun.pipeline.resample import resample_to_interval
|
||||
|
||||
from chanlun.core.ChanBI import ChanBI
|
||||
from chanlun.core.ChanBIZS import ChanBIZS
|
||||
@@ -31,17 +30,26 @@ from chanlun.core.ChanZS import ChanZS, ChanZS_Big
|
||||
from chanlun.indicators.ChanMACD import ChanMACD
|
||||
from chanlun.pipeline.builders.bi import BiBuilderMixin
|
||||
from chanlun.pipeline.builders.bsp import BspBuilderMixin
|
||||
from chanlun.pipeline.builders.fast_bsp import FastBspBuilderMixin
|
||||
from chanlun.pipeline.builders.incremental import IncrementalBuilderMixin
|
||||
from chanlun.pipeline.builders.indicators import IndicatorsBuilderMixin
|
||||
from chanlun.pipeline.builders.kline import KlineBuilderMixin
|
||||
from chanlun.pipeline.builders.seg import SegBuilderMixin
|
||||
from chanlun.pipeline.builders.zs import ZsBuilderMixin
|
||||
|
||||
class TF_DF(IndicatorsBuilderMixin, KlineBuilderMixin, BiBuilderMixin, SegBuilderMixin, ZsBuilderMixin, BspBuilderMixin, IncrementalBuilderMixin):
|
||||
def __init__(self, df=None, interval=0, timeframe=None):
|
||||
class TF_DF(IndicatorsBuilderMixin, KlineBuilderMixin, BiBuilderMixin, SegBuilderMixin, ZsBuilderMixin, BspBuilderMixin, FastBspBuilderMixin, IncrementalBuilderMixin):
|
||||
def __init__(self, df=None, interval=0, timeframe=None, lean=False):
|
||||
"""lean=True 只构建到中枢,跳过线段/走势中枢/MACD 状态机。
|
||||
|
||||
研究与实盘只吃 bi_list → 中枢 → fast_bsp 这条链;线段、zs、big_zs 和整套
|
||||
MACD 背驰状态机是 web 展示与 bsp_list 才用的。实测这些占全量构建的约四成。
|
||||
注意 lean 下 bsp_list/seg_list/chanmacd 均为空,**不要给 web 用**。
|
||||
"""
|
||||
self.lean = lean
|
||||
if df is not None:
|
||||
self.init_TF_DF(df, interval, timeframe)
|
||||
def init_TF_DF(self, df, interval, timeframe):
|
||||
self.init_TF_DF(df, interval, timeframe, lean=lean)
|
||||
def init_TF_DF(self, df, interval, timeframe, lean=False):
|
||||
self.lean = lean
|
||||
self.timeframe = timeframe
|
||||
self.interval = interval
|
||||
# 检查 DataFrame 是否为空或没有 date 列
|
||||
@@ -62,12 +70,17 @@ class TF_DF(IndicatorsBuilderMixin, KlineBuilderMixin, BiBuilderMixin, SegBuilde
|
||||
self.zs_list = []
|
||||
self.bi_zs_list = []
|
||||
self.bsp_list = []
|
||||
self.fast_bsp_list = []
|
||||
self.seg_list = []
|
||||
self.klc_fx_list = []
|
||||
self.klu_list = self.cal_kl_data(self.dataframe)
|
||||
self.klc_list = self.get_klc_list(self.klu_list)
|
||||
self.bi_list = self.cal_bi_list(self.klc_list)
|
||||
self.bi_zs_list = self.cal_bi_zs_list_pure(self.bi_list)
|
||||
if self.lean:
|
||||
self.big_zs_list = []
|
||||
self.chanmacd = None
|
||||
return
|
||||
self.seg_list = self.get_seg_list(self.bi_list)
|
||||
self.zs_list = self.get_zs_list(self.bi_list, self.seg_list)
|
||||
self.big_zs_list = self.get_big_zs_list(self.zs_list)
|
||||
|
||||
@@ -0,0 +1,198 @@
|
||||
"""Pin chanlun.indicators.ta to TA-Lib's output, bar for bar.
|
||||
|
||||
These indicators feed the Chan structure builders, so a one-bar shift in the
|
||||
warm-up or a different smoothing seed silently changes every downstream
|
||||
bi/seg/zs. Equality against the reference implementation is the only check
|
||||
that catches that.
|
||||
|
||||
Skipped when talib is unavailable — which is the point of the replacement, so
|
||||
the suite still has to pass without it. Run in an environment that has talib
|
||||
whenever chanlun/indicators/ta.py changes.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
|
||||
from chanlun.indicators import ta # noqa: E402
|
||||
from chanlun.pipeline.resample import resample_to_interval # noqa: E402
|
||||
|
||||
try:
|
||||
import talib.abstract as reference
|
||||
except ImportError: # pragma: no cover
|
||||
reference = None
|
||||
|
||||
requires_talib = unittest.skipIf(reference is None, "talib not installed")
|
||||
|
||||
|
||||
def make_ohlcv(n: int = 900, seed: int = 7) -> pd.DataFrame:
|
||||
"""Random walk with enough range for BBANDS(365) and EMA(208) to warm up."""
|
||||
rng = np.random.default_rng(seed)
|
||||
close = 100.0 + np.cumsum(rng.normal(0.0, 1.0, n))
|
||||
spread = np.abs(rng.normal(0.0, 0.6, n)) + 0.05
|
||||
high = close + spread
|
||||
low = close - spread
|
||||
open_ = np.concatenate([[close[0]], close[:-1]])
|
||||
return pd.DataFrame(
|
||||
{
|
||||
"date": pd.date_range("2024-01-01", periods=n, freq="1min", tz="UTC"),
|
||||
"open": open_,
|
||||
"high": np.maximum.reduce([high, open_, close]),
|
||||
"low": np.minimum.reduce([low, open_, close]),
|
||||
"close": close,
|
||||
"volume": rng.uniform(1.0, 100.0, n),
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
class TAEquivalence(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.df = make_ohlcv()
|
||||
|
||||
def assertSameSeries(self, got, expected, label: str) -> None:
|
||||
g = np.asarray(got, dtype=float)
|
||||
e = np.asarray(expected, dtype=float)
|
||||
self.assertEqual(g.shape, e.shape, f"{label}: shape")
|
||||
np.testing.assert_array_equal(
|
||||
np.isnan(g), np.isnan(e), err_msg=f"{label}: NaN warm-up differs"
|
||||
)
|
||||
mask = ~np.isnan(e)
|
||||
np.testing.assert_allclose(
|
||||
g[mask], e[mask], rtol=1e-9, atol=1e-8, err_msg=f"{label}: values differ"
|
||||
)
|
||||
|
||||
@requires_talib
|
||||
def test_sma(self) -> None:
|
||||
for period in (5, 20, 90, 250):
|
||||
self.assertSameSeries(
|
||||
ta.SMA(self.df, timeperiod=period),
|
||||
reference.SMA(self.df, timeperiod=period),
|
||||
f"SMA({period})",
|
||||
)
|
||||
|
||||
@requires_talib
|
||||
def test_ma(self) -> None:
|
||||
for period in (5, 10, 250):
|
||||
self.assertSameSeries(
|
||||
ta.MA(self.df, timeperiod=period),
|
||||
reference.MA(self.df, timeperiod=period),
|
||||
f"MA({period})",
|
||||
)
|
||||
|
||||
@requires_talib
|
||||
def test_ema(self) -> None:
|
||||
for period in (5, 7, 10, 13, 24, 26, 30, 52, 104, 156, 208):
|
||||
self.assertSameSeries(
|
||||
ta.EMA(self.df, timeperiod=period),
|
||||
reference.EMA(self.df, timeperiod=period),
|
||||
f"EMA({period})",
|
||||
)
|
||||
|
||||
@requires_talib
|
||||
def test_rsi(self) -> None:
|
||||
for period in (7, 14, 21):
|
||||
self.assertSameSeries(
|
||||
ta.RSI(self.df, timeperiod=period),
|
||||
reference.RSI(self.df, timeperiod=period),
|
||||
f"RSI({period})",
|
||||
)
|
||||
|
||||
@requires_talib
|
||||
def test_atr(self) -> None:
|
||||
for period in (7, 14, 30):
|
||||
self.assertSameSeries(
|
||||
ta.ATR(self.df, timeperiod=period),
|
||||
reference.ATR(self.df, timeperiod=period),
|
||||
f"ATR({period})",
|
||||
)
|
||||
|
||||
@requires_talib
|
||||
def test_macd(self) -> None:
|
||||
for fast, slow, signal in ((12, 26, 9), (26, 52, 9), (5, 35, 5)):
|
||||
got = ta.MACD(self.df, fastperiod=fast, slowperiod=slow, signalperiod=signal)
|
||||
exp = reference.MACD(self.df, fastperiod=fast, slowperiod=slow, signalperiod=signal)
|
||||
for col in ("macd", "macdsignal", "macdhist"):
|
||||
self.assertSameSeries(got[col], exp[col], f"MACD({fast},{slow},{signal}).{col}")
|
||||
|
||||
@requires_talib
|
||||
def test_bbands(self) -> None:
|
||||
cases = (
|
||||
(365, 3.0, 3.0),
|
||||
(120, 3.0, 3.0),
|
||||
(41, 2.3, 2.3),
|
||||
(41, 2.0, 2.0),
|
||||
(26, 3.0, 3.0),
|
||||
(20, 2.0, 2.0),
|
||||
(14, 2.0, 2.0),
|
||||
)
|
||||
for period, up, dn in cases:
|
||||
got = ta.BBANDS(self.df, timeperiod=period, nbdevup=up, nbdevdn=dn, matype=0)
|
||||
exp = reference.BBANDS(self.df, timeperiod=period, nbdevup=up, nbdevdn=dn, matype=0)
|
||||
for col in ("upperband", "middleband", "lowerband"):
|
||||
self.assertSameSeries(got[col], exp[col], f"BBANDS({period},{up},{dn}).{col}")
|
||||
|
||||
@requires_talib
|
||||
def test_bbands_is_more_accurate_than_talib_on_tiny_windows(self) -> None:
|
||||
"""A deliberate divergence, documented so nobody "fixes" it back.
|
||||
|
||||
TA-Lib derives the variance from sumsq/n - mean**2, which cancels
|
||||
catastrophically when the window is short and prices are far from zero;
|
||||
at timeperiod=2 it drifts ~1e-6. Rolling std is accurate there, so the
|
||||
two disagree. No timeperiod below 14 is used in this codebase, and the
|
||||
periods that are used agree to ~1e-10 (covered by test_bbands).
|
||||
"""
|
||||
got = ta.BBANDS(self.df, timeperiod=2, nbdevup=1.0, nbdevdn=1.0, matype=0)["upperband"]
|
||||
exp = reference.BBANDS(self.df, timeperiod=2, nbdevup=1.0, nbdevdn=1.0, matype=0)["upperband"]
|
||||
|
||||
window = self.df["close"].rolling(2)
|
||||
truth = window.mean() + window.std(ddof=0)
|
||||
ours = np.nanmax(np.abs((got - truth).to_numpy()))
|
||||
theirs = np.nanmax(np.abs((exp - truth).to_numpy()))
|
||||
self.assertLess(ours, 1e-9)
|
||||
self.assertLess(ours, theirs)
|
||||
|
||||
@requires_talib
|
||||
def test_matches_on_real_price_scale(self) -> None:
|
||||
"""Guard against tolerances that only hold near 100."""
|
||||
df = self.df.copy()
|
||||
for col in ("open", "high", "low", "close"):
|
||||
df[col] *= 900.0
|
||||
self.assertSameSeries(
|
||||
ta.ATR(df, timeperiod=14), reference.ATR(df, timeperiod=14), "ATR@scale"
|
||||
)
|
||||
self.assertSameSeries(
|
||||
ta.RSI(df, timeperiod=14), reference.RSI(df, timeperiod=14), "RSI@scale"
|
||||
)
|
||||
|
||||
|
||||
class ResampleEquivalence(unittest.TestCase):
|
||||
@unittest.skipIf(
|
||||
__import__("importlib").util.find_spec("technical") is None,
|
||||
"technical not installed",
|
||||
)
|
||||
def test_matches_technical(self) -> None:
|
||||
from technical.util import resample_to_interval as ref_resample
|
||||
|
||||
df = make_ohlcv(600)
|
||||
for interval in (5, 15, 60, "5m", "1h"):
|
||||
got = resample_to_interval(df, interval)
|
||||
exp = ref_resample(df, interval)
|
||||
pd.testing.assert_frame_equal(got, exp, check_exact=False, rtol=1e-12)
|
||||
|
||||
def test_shapes_without_reference(self) -> None:
|
||||
df = make_ohlcv(120)
|
||||
out = resample_to_interval(df, 5)
|
||||
self.assertEqual(list(out.columns), ["date", "open", "high", "low", "close", "volume"])
|
||||
self.assertLessEqual(len(out), 120 // 5 + 1)
|
||||
self.assertTrue((out["high"] >= out["low"]).all())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,245 @@
|
||||
"""Bitget v2 合约 REST 的最小客户端,只覆盖实盘执行要用的几个端点。
|
||||
|
||||
## 为什么不用 Hummingbot 下单
|
||||
|
||||
Hummingbot 的 Bitget 连接器只暴露 LIMIT / LIMIT_MAKER / MARKET,没有触发单。
|
||||
于是 `PositionExecutor` 的止损只能在本地控制循环里盯价、触发时才发市价单——
|
||||
**进程一死仓位就是裸的**。
|
||||
|
||||
而交易所本身完全支持:`place-order` 有 `presetStopLossPrice`,下单时就把止损
|
||||
挂到服务端。所以整个结构变成两个调用,止损从入场那一刻起就不依赖我们的进程
|
||||
存活。绕过连接器不是图省事,是为了消掉一整类故障。
|
||||
|
||||
## 止盈为什么不用 presetStopSurplusPrice
|
||||
|
||||
它触发后按**市价**执行。而成本模型里止盈是 maker——那 60% 的出场不吃滑点、
|
||||
按 maker 费率计(见 `lib/shadow_budget.LEG_IS_TAKER`)。用 preset 会让这部分
|
||||
变成 taker,预算模型就不成立了。所以止盈单独挂 `post_only` 的 reduce-only
|
||||
限价单。
|
||||
|
||||
止损反过来:必须是市价。stop-limit 在急跌里可能不成交,损失远大于省下的费。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
|
||||
BASE = "https://api.bitget.com"
|
||||
PRODUCT = "usdt-futures"
|
||||
MARGIN_COIN = "USDT"
|
||||
|
||||
|
||||
class BitgetError(RuntimeError):
|
||||
def __init__(self, code: str, msg: str, path: str):
|
||||
super().__init__(f"{path} → [{code}] {msg}")
|
||||
self.code, self.msg = code, msg
|
||||
|
||||
|
||||
class Bitget:
|
||||
def __init__(self, key: str = "", secret: str = "", passphrase: str = "",
|
||||
dry: bool = False):
|
||||
self.key = key or os.environ.get("BITGET_API_KEY", "")
|
||||
self.secret = secret or os.environ.get("BITGET_API_SECRET", "")
|
||||
# 两个名字都收:另外两项是 BITGET_API_KEY / BITGET_API_SECRET,
|
||||
# 这一项却没有 API_,很容易顺手写成 BITGET_API_PASSPHRASE。写错的
|
||||
# 后果是"密钥像是填了"但签名一直失败,排查起来很绕
|
||||
self.passphrase = (passphrase
|
||||
or os.environ.get("BITGET_PASSPHRASE", "")
|
||||
or os.environ.get("BITGET_API_PASSPHRASE", ""))
|
||||
self.dry = dry
|
||||
self._sess = None
|
||||
|
||||
def _sign(self, ts: str, method: str, path: str, body: str) -> str:
|
||||
msg = f"{ts}{method.upper()}{path}{body}"
|
||||
return base64.b64encode(hmac.new(
|
||||
self.secret.encode(), msg.encode(), hashlib.sha256).digest()
|
||||
).decode()
|
||||
|
||||
async def _req(self, method: str, path: str, params: dict | None = None,
|
||||
body: dict | None = None) -> dict:
|
||||
import aiohttp
|
||||
if self._sess is None:
|
||||
self._sess = aiohttp.ClientSession(
|
||||
timeout=aiohttp.ClientTimeout(total=15))
|
||||
qs = ""
|
||||
if params:
|
||||
qs = "?" + "&".join(f"{k}={v}" for k, v in sorted(params.items()))
|
||||
payload = json.dumps(body) if body else ""
|
||||
ts = str(int(time.time() * 1000))
|
||||
headers = {
|
||||
"ACCESS-KEY": self.key,
|
||||
"ACCESS-SIGN": self._sign(ts, method, path + qs, payload),
|
||||
"ACCESS-PASSPHRASE": self.passphrase,
|
||||
"ACCESS-TIMESTAMP": ts,
|
||||
"Content-Type": "application/json",
|
||||
"locale": "en-US",
|
||||
}
|
||||
async with self._sess.request(method, BASE + path + qs,
|
||||
headers=headers,
|
||||
data=payload or None) as r:
|
||||
d = await r.json()
|
||||
if str(d.get("code")) != "00000":
|
||||
raise BitgetError(str(d.get("code")), str(d.get("msg")), path)
|
||||
return d.get("data")
|
||||
|
||||
async def close(self) -> None:
|
||||
if self._sess is not None:
|
||||
await self._sess.close()
|
||||
self._sess = None
|
||||
|
||||
# ── 只读 ──────────────────────────────────────────────────────
|
||||
async def contracts(self) -> dict:
|
||||
"""合约规则。用于数量步长与价格 tick。"""
|
||||
d = await self._req("GET", "/api/v2/mix/market/contracts",
|
||||
{"productType": PRODUCT})
|
||||
return {c["symbol"]: c for c in d}
|
||||
|
||||
async def positions(self) -> list:
|
||||
d = await self._req("GET", "/api/v2/mix/position/all-position",
|
||||
{"productType": PRODUCT,
|
||||
"marginCoin": MARGIN_COIN})
|
||||
return [p for p in (d or []) if float(p.get("total") or 0) != 0]
|
||||
|
||||
async def history_positions(self, start_ms: int | None = None,
|
||||
limit: int = 100) -> list:
|
||||
"""已平仓位,用来取**已实现盈亏**。
|
||||
|
||||
为什么必须问交易所而不是自己算:止损与止盈都挂在交易所侧成交,本进程
|
||||
看不到成交价;而且要算准还得含手续费与资金费。这个端点的 `netProfit`
|
||||
已经是 `pnl + totalFunding + openFee + closeFee`,正是日亏损上限该用
|
||||
的数。自己按标记价估会把费用漏掉,方向还总是偏乐观。
|
||||
|
||||
返回形状按文档是 `data.list`,但也见过直接给数组的写法,两种都收。
|
||||
时间字段文档写 `ctime/utime`,官方 TS 类型写 `cTime/uTime`,同样都读。
|
||||
"""
|
||||
p: dict = {"productType": PRODUCT, "limit": str(limit)}
|
||||
if start_ms:
|
||||
p["startTime"] = str(int(start_ms))
|
||||
d = await self._req("GET", "/api/v2/mix/position/history-position", p)
|
||||
if isinstance(d, dict):
|
||||
return list(d.get("list") or [])
|
||||
return list(d or [])
|
||||
|
||||
async def fee_rate(self, symbol: str) -> dict:
|
||||
"""账户在该合约上的**实际**费率档。
|
||||
|
||||
这一项决定 ATR 门控阈值(约 5 + 1.1×taker_bp),进而决定可交易币池。
|
||||
接口的合约默认档是 VIP0,不是账户档,必须问这个端点。
|
||||
"""
|
||||
return await self._req("GET", "/api/v2/mix/market/query-position-lever",
|
||||
{"symbol": symbol, "productType": PRODUCT})
|
||||
|
||||
async def account(self) -> dict:
|
||||
return await self._req("GET", "/api/v2/mix/account/account",
|
||||
{"symbol": "BTCUSDT", "productType": PRODUCT,
|
||||
"marginCoin": MARGIN_COIN})
|
||||
|
||||
# ── 写 ────────────────────────────────────────────────────────
|
||||
async def set_leverage(self, symbol: str, lev: int,
|
||||
hold_side: str | None = None) -> dict:
|
||||
body = {"symbol": symbol, "productType": PRODUCT,
|
||||
"marginCoin": MARGIN_COIN, "leverage": str(lev)}
|
||||
if hold_side:
|
||||
body["holdSide"] = hold_side
|
||||
return await self._req("POST", "/api/v2/mix/account/set-leverage",
|
||||
body=body)
|
||||
|
||||
async def set_margin_mode(self, symbol: str,
|
||||
mode: str = "isolated") -> dict:
|
||||
return await self._req("POST", "/api/v2/mix/account/set-margin-mode",
|
||||
body={"symbol": symbol, "productType": PRODUCT,
|
||||
"marginCoin": MARGIN_COIN,
|
||||
"marginMode": mode})
|
||||
|
||||
async def set_position_mode(self, mode: str = "one_way_mode") -> dict:
|
||||
"""单向 / 双向持仓。**按 productType 生效,不是按 symbol。**
|
||||
|
||||
必须显式设,因为它决定下单体的语法,两者不匹配会被整体拒单:
|
||||
|
||||
单向:side=buy/sell,**不带** tradeSide;平仓用 reduceOnly=YES
|
||||
双向:side + tradeSide=open/close;reduceOnly 在这个模式下无效
|
||||
|
||||
实盘上曾因为带着 tradeSide 打到单向账户,连续 8 次下单全被 40774 拒掉
|
||||
(4 个信号 × 2 条腿),而链路其余部分完全正常。
|
||||
|
||||
我们永不同时持有两个方向,所以单向是对的模式;且 reduceOnly 只在单向
|
||||
下可用,而出场腿依赖它防止反手开出反向仓。
|
||||
|
||||
交易所侧有持仓或挂单时切换会失败——所以调用点放在 reconcile 之后。
|
||||
"""
|
||||
return await self._req("POST", "/api/v2/mix/account/set-position-mode",
|
||||
body={"productType": PRODUCT, "posMode": mode})
|
||||
|
||||
async def entry_with_stop(self, symbol: str, side: str, size: str,
|
||||
stop_px: str, client_oid: str) -> dict:
|
||||
"""市价入场,**同时**把止损挂到服务端。
|
||||
|
||||
`presetStopLossPrice` 触发后按市价执行,这正是成本模型要的(止损是
|
||||
taker)。`clientOid` 给交易所级幂等——重发同一个 oid 会被拒,比本地
|
||||
去重可靠,因为「已发出但没收到回复」这种情况本地判不了。
|
||||
|
||||
**不带 `tradeSide`**:账户是单向持仓,带了会被 40774 整体拒单。
|
||||
"""
|
||||
body = {"symbol": symbol, "productType": PRODUCT,
|
||||
"marginMode": "isolated", "marginCoin": MARGIN_COIN,
|
||||
"size": size, "side": side,
|
||||
"orderType": "market", "clientOid": client_oid,
|
||||
"presetStopLossPrice": stop_px}
|
||||
if self.dry:
|
||||
print(f" [dry] 入场+止损 {body}", flush=True)
|
||||
return {"orderId": "dry", "clientOid": client_oid}
|
||||
return await self._req("POST", "/api/v2/mix/order/place-order",
|
||||
body=body)
|
||||
|
||||
async def tp_limit(self, symbol: str, side: str, size: str, px: str,
|
||||
client_oid: str) -> dict:
|
||||
"""挂 maker 止盈。
|
||||
|
||||
`side` 传的是**平仓方向**(多头止盈是 sell)。`post_only` 保证是 maker:
|
||||
成本模型里止盈那 60% 按 maker 费率计且不吃滑点,用 taker 会破坏预算。
|
||||
|
||||
单向持仓下平仓的写法是 `side` 取反 + `reduceOnly=YES`,**不带**
|
||||
`tradeSide`。`reduceOnly` 也正好只在单向模式下有效,它防止反手开出
|
||||
一个反向仓。
|
||||
"""
|
||||
body = {"symbol": symbol, "productType": PRODUCT,
|
||||
"marginMode": "isolated", "marginCoin": MARGIN_COIN,
|
||||
"size": size, "side": side,
|
||||
"orderType": "limit", "price": px, "force": "post_only",
|
||||
"reduceOnly": "YES", "clientOid": client_oid}
|
||||
if self.dry:
|
||||
print(f" [dry] 止盈限价 {body}", flush=True)
|
||||
return {"orderId": "dry", "clientOid": client_oid}
|
||||
return await self._req("POST", "/api/v2/mix/order/place-order",
|
||||
body=body)
|
||||
|
||||
async def close_market(self, symbol: str, hold_side: str,
|
||||
size: str, client_oid: str) -> dict:
|
||||
"""市价平(超时腿与对账用)。
|
||||
|
||||
同 tp_limit:单向持仓下是 `side` 取反 + `reduceOnly`,不带 `tradeSide`。
|
||||
"""
|
||||
side = "sell" if hold_side == "long" else "buy"
|
||||
body = {"symbol": symbol, "productType": PRODUCT,
|
||||
"marginMode": "isolated", "marginCoin": MARGIN_COIN,
|
||||
"size": size, "side": side,
|
||||
"orderType": "market", "reduceOnly": "YES",
|
||||
"clientOid": client_oid}
|
||||
if self.dry:
|
||||
print(f" [dry] 市价平 {body}", flush=True)
|
||||
return {"orderId": "dry"}
|
||||
return await self._req("POST", "/api/v2/mix/order/place-order",
|
||||
body=body)
|
||||
|
||||
async def cancel_all(self, symbol: str) -> dict:
|
||||
body = {"symbol": symbol, "productType": PRODUCT,
|
||||
"marginCoin": MARGIN_COIN}
|
||||
if self.dry:
|
||||
print(f" [dry] 撤全部挂单 {symbol}", flush=True)
|
||||
return {}
|
||||
return await self._req("POST", "/api/v2/mix/order/cancel-all-orders",
|
||||
body=body)
|
||||
@@ -0,0 +1,170 @@
|
||||
# 小额实盘执行器 · 部署
|
||||
|
||||
## 为什么是两台机
|
||||
|
||||
Bitget 的 API key 绑了 IP 白名单,只能从 AWS 那台发单;信号是新加坡那台采集器
|
||||
算出来的。于是分工固定成:
|
||||
|
||||
```
|
||||
新加坡(采集/研究机) AWS(生产机)
|
||||
shadow_hb.py 算信号 ship_signals.py 拉总线
|
||||
└→ ~/chan-live/state/ └→ /var/lib/chan-live/state/
|
||||
signals_live.jsonl ──ssh tail──→ signals_live.jsonl
|
||||
live_exec.py 读总线 → Bitget REST
|
||||
```
|
||||
|
||||
**生产机上不装采集侧的任何东西**(Docker / Hummingbot / pandas / chanlun)。
|
||||
理由不是洁癖,是四条具体代价:
|
||||
|
||||
1. `live_state.json` 原先落在 `research/out/`,而那个目录 `shadow_hb.py` 会在
|
||||
CSV 表头变化时自动 rename 归档、研究脚本会写、人也会手工清数据。那个文件装
|
||||
的是 `MAX_DAY_LOSS` 累计与已处理信号键,**被清掉不报错,只是两道闸静默
|
||||
失效**。现在改到 `/var/lib/chan-live`。
|
||||
2. 采集器十币清空 300~560ms,直接叠在信号到达执行器的延迟上。
|
||||
3. 研究侧的探针 OOM 过一次(14.9 GB、负载 12)。当时若有仓位在场,执行器会被
|
||||
一起杀掉,只剩交易所侧止损兜着。
|
||||
4. 依赖面:执行器只需标准库 + `aiohttp`。原先为读两个常量 import 研究侧的
|
||||
`step43`,把 numpy/pandas/pyarrow 全拖进实盘进程。
|
||||
|
||||
`install.sh` 里有一条断言会真的挡住第 4 条回归。
|
||||
|
||||
## 机器要求
|
||||
|
||||
CPU 无所谓(执行器几乎不算东西,信号 6.8 个/天)。要的是:
|
||||
|
||||
- 出口 IP 固定,且已加进 Bitget 该 key 的白名单
|
||||
- `chrony` 能同步。**这一条是硬要求**:`LIVE_STALE_S` 那道闸靠两机时钟一致才
|
||||
有意义,采集机时钟快 5 分钟就等于把闸放宽 5 分钟,一个早已失效的参考价会被
|
||||
当成新鲜的照做
|
||||
- 能 ssh 到采集机(拉总线用)
|
||||
|
||||
## 一、生产机(AWS)
|
||||
|
||||
```bash
|
||||
# 1. 取代码。只需要 live/ 这一个子树,但整仓克隆更省事
|
||||
git clone -b chan <repo> /tmp/chan && cd /tmp/chan
|
||||
sudo ./live/deploy/install.sh
|
||||
# 或让它自己克隆:sudo REPO=<repo> ./live/deploy/install.sh
|
||||
|
||||
# 2. 填密钥与参数
|
||||
sudo vi /etc/chan-live/live.env
|
||||
```
|
||||
|
||||
`live.env` 里必须改的四项:`BITGET_API_KEY` / `SECRET` / `PASSPHRASE` /
|
||||
`SHIP_FROM`。密钥权限**只勾只读 + 交易,不要勾提币**。
|
||||
|
||||
```bash
|
||||
# 3. 装拉总线用的 ssh key
|
||||
sudo -u chan ssh-keygen -t ed25519 -N '' \
|
||||
-f /var/lib/chan-live/home/.ssh/id_ed25519
|
||||
sudo cat /var/lib/chan-live/home/.ssh/id_ed25519.pub
|
||||
# 把这一行加到采集机的 ~/.ssh/authorized_keys
|
||||
|
||||
# 4. 空跑验全链(不下真单)
|
||||
sudo ./live/deploy/dryrun.sh
|
||||
```
|
||||
|
||||
`dryrun.sh` 验的是那些"上线才暴露、且暴露方式是花钱"的环节:密钥能不能用、
|
||||
IP 白名单对不对、chrony 同步没有、ssh 通不通、对端总线有没有信号、数量与价位
|
||||
取整合不合交易所规则。**不要跳过。**
|
||||
|
||||
```bash
|
||||
# 5. 真跑
|
||||
sudo systemctl enable --now chan-live-ship chan-live-exec
|
||||
./live/deploy/status.sh
|
||||
```
|
||||
|
||||
## 二、采集机(新加坡)
|
||||
|
||||
采集器要重启一次才会开始往总线写——`signal_bus.emit` 是后加的,跑着的进程没有
|
||||
加载。重启会丢已采的几分钟,数据本身不受影响(CSV 是追加的)。
|
||||
|
||||
```bash
|
||||
cd <repo> && git pull
|
||||
docker stop shadow && docker rm shadow
|
||||
SHADOW_SITE=sg-tencent SYMS=BTC,ETH,SOL,BNB,XRP,DOGE,ADA,AVAX,LINK,LTC \
|
||||
bash research/live/deploy/start.sh
|
||||
```
|
||||
|
||||
`start.sh` 会把 `$HOME/chan-live/state` 挂进容器成 `/bus`,总线落在
|
||||
`$HOME/chan-live/state/signals_live.jsonl`。**总线刻意放在仓库外**,因为它是
|
||||
交给另一台机的交接点,而仓库会被 git 动。
|
||||
|
||||
确认在写:
|
||||
|
||||
```bash
|
||||
ls -la ~/chan-live/state/
|
||||
# 等一个信号(6.8 个/天,可能要等几小时)
|
||||
tail -f ~/chan-live/state/signals_live.jsonl
|
||||
```
|
||||
|
||||
## 日常
|
||||
|
||||
```bash
|
||||
./live/deploy/status.sh # 一屏体检
|
||||
journalctl -u chan-live-exec -f # 执行器日志
|
||||
journalctl -u chan-live-ship -f # 搬运日志
|
||||
```
|
||||
|
||||
**怎么判健康:** 信号 6.8 个/天,所以"很久没有新信号"是正常的,不能当健康
|
||||
指标。要看的是 `chan-live-ship` 的心跳(每 5 分钟一条),里面报 ssh 在线时长与
|
||||
重连次数。管道死了但进程还活着是这里最危险的状态——`ServerAliveInterval=15`
|
||||
负责让它变成一次可见的断开。
|
||||
|
||||
**Telegram** 在 `live.env` 里填 `TG_TOKEN` / `TG_CHAT` 就开。启动时会推一条
|
||||
"执行器启动",兼作通道自检——配错了当场就知道,而不是等几小时后第一个真信号
|
||||
来时才发现。之后每小时一条在线(`TG_HB_MIN`,默认 60),带建仓数、当日盈亏
|
||||
和搬运 ssh 新鲜度——执行器活着不代表上游还在投信号。推开仓、平仓(带已实现
|
||||
盈亏)、被硬约束挡住、报错、对账平仓、跨日结算;不推信号过期跳过(常态)和
|
||||
5 分钟日志心跳。约 55 条/天上限。
|
||||
|
||||
## 停机与回滚
|
||||
|
||||
```bash
|
||||
# 停新开仓,但保留在场仓位的管理(48 分钟超时平仓在执行器进程里)
|
||||
sudo systemctl stop chan-live-ship
|
||||
|
||||
# 全停。执行器收到 SIGTERM 会**撤挂单 + 市价平掉在场仓位**再退出
|
||||
# (内部平仓上限 60s,systemd 给了 90s 停机窗口)
|
||||
sudo systemctl stop chan-live-exec
|
||||
|
||||
# 代码回滚
|
||||
cd /opt/chan && sudo git reset --hard <sha> && sudo systemctl restart chan-live-exec
|
||||
```
|
||||
|
||||
⚠️ **状态目录 `/var/lib/chan-live` 不要跟着回滚。** 它存的是当日计数与已处理
|
||||
信号键;清掉等于日上限归零、且可能重开已经做过的仓。
|
||||
|
||||
## 故障处理
|
||||
|
||||
| 症状 | 大概率原因 |
|
||||
|---|---|
|
||||
| 启动即 `40018` / 签名错 | 出口 IP 不在白名单,或密钥抄错。`curl https://api.ipify.org` 对一下 |
|
||||
| 搬运日志 `Permission denied (publickey)` | 第 3 步的 pubkey 没加到采集机 |
|
||||
| 搬运在线但一直没信号 | 采集机没重启过(`signal_bus.emit` 没加载),或对端总线路径不对 |
|
||||
| Telegram 在线报「读不到搬运」 | 搬运没起,或还是没落 `ship_alive.json` 的旧版本,两边一起重启 |
|
||||
| Telegram 在线报「ssh 已断开」 | 采集机 ssh 断了,搬运在重连。看 `chan-live-ship` 日志 |
|
||||
| 日志 `⛔ 时间倒流 Xs` | 两机时钟不同步,**staleness 闸已不可信**。查两边 `chronyc tracking` |
|
||||
| 信号收到但都被跳过 | `age > LIVE_STALE_S`。看是搬运慢还是时钟偏;也可能是重连重放的旧信号(这种跳过是对的) |
|
||||
| `systemctl status` 显示 start-limit-hit | 5 分钟内重启 5 次,systemd 停手了。先看 journal 找真因,再 `systemctl reset-failed` |
|
||||
| 执行器起不来,报缺 numpy/pandas | 有人给生产侧加了研究侧的 import。`install.sh` 的依赖断言就是挡这个 |
|
||||
|
||||
## 已知的退化边界
|
||||
|
||||
- **进程死掉不会变成裸仓。** 止损与止盈都挂在交易所侧(`presetStopLossPrice`
|
||||
与 post-only reduce-only 限价单),只有 48 分钟超时平仓在本进程。所以进程死
|
||||
掉的后果是持仓超过 48 根,不是失去保护。
|
||||
- **崩溃与主动停机的处理不同,是刻意的。** 崩溃后 systemd 几秒内重启,
|
||||
`reconcile` 接着撤挂单 + 平掉遗留仓位,空窗期由交易所侧止损兜着。主动
|
||||
`systemctl stop` 则在退出前就平掉——因为停机后没人重启,仓位会一直挂到止损
|
||||
或止盈,超时腿丢了就不是回测那个出场结构了。
|
||||
- **断线超过 20s 就等于漏掉那期间的信号。** 重连会把总线重放上来,但旧信号会被
|
||||
staleness 闸挡掉。这是对的——参考成交价是次根开盘价,过了就不是回测那个价。
|
||||
- **重启时会平掉交易所上已有的仓位**(`reconcile`)。接管要重建入场价、ATR、
|
||||
剩余半仓状态和已过根数,任一项猜错就跑成另一个收益结构,所以选择平掉。
|
||||
- **日亏损上限依赖 `watch()` 循环**。止损与止盈在交易所侧成交,本进程收不到
|
||||
通知,所以有个 10s 轮询去 `history-position` 取 `netProfit`(含手续费与资金
|
||||
费)记回闸。这个循环停了,`pnl_day` 会恒为 0,`MAX_DAY_LOSS` 静默失效。
|
||||
心跳里会打 `当日 PnL x/-20`,值一直是 0.00 而又确实有平仓,就是它出了问题。
|
||||
- **平仓后 `MAX_OPEN` 名额由 `watch()` 释放**,不是立刻。最坏延迟 10s。若
|
||||
历史记录还没落库,会等下一轮,日志里打"历史未就绪,下轮再结算"。
|
||||
@@ -0,0 +1,49 @@
|
||||
[Unit]
|
||||
Description=chan 小额实盘执行器(读信号总线,直接调 Bitget REST)
|
||||
# 搬运挂了执行器仍要活着——它得继续管在场仓位的 48 分钟超时平仓。
|
||||
# 所以这里只写 Wants(弱依赖),不写 Requires
|
||||
Wants=network-online.target chan-live-ship.service
|
||||
After=network-online.target chan-live-ship.service
|
||||
# 频繁重启说明有真问题,别让它无限打交易所。5 分钟内起 5 次就停下等人。
|
||||
# 注意这两项在 systemd 229+ 属于 [Unit],写在 [Service] 里会被忽略
|
||||
StartLimitIntervalSec=300
|
||||
StartLimitBurst=5
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=chan
|
||||
Group=chan
|
||||
WorkingDirectory=/opt/chan
|
||||
# 密钥与参数在这个文件里,权限必须 600。用 EnvironmentFile 而不是
|
||||
# Environment=,后者会出现在 `systemctl show` 的输出里
|
||||
EnvironmentFile=/etc/chan-live/live.env
|
||||
ExecStart=/opt/chan/.venv/bin/python /opt/chan/live/live_exec.py
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
|
||||
# 收到 stop 时给足时间:执行器接到 SIGTERM 会撤挂单 + 平掉在场仓位再退出
|
||||
# (live_exec.shutdown(),内部平仓上限 60s)。默认的 90s 停机窗口留了余量。
|
||||
# 不需要 KillSignal=SIGINT——SIGTERM 已在代码里显式挂了处理器
|
||||
TimeoutStopSec=90
|
||||
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
SyslogIdentifier=chan-live-exec
|
||||
|
||||
# ── 收紧权限 ──────────────────────────────────────────────────────
|
||||
# 生产进程只需要读 /opt/chan 和读写状态目录,别的一概不给。这几条很廉价,
|
||||
# 但真出了远程代码执行,爆炸半径小很多——而这个进程手里有交易权限的密钥
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=true
|
||||
ReadWritePaths=/var/lib/chan-live
|
||||
ProtectKernelTunables=true
|
||||
ProtectKernelModules=true
|
||||
ProtectControlGroups=true
|
||||
RestrictSUIDSGID=true
|
||||
LockPersonality=true
|
||||
MemoryMax=512M
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,40 @@
|
||||
[Unit]
|
||||
Description=chan 信号搬运(把采集机的总线拉到本机)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
StartLimitIntervalSec=300
|
||||
StartLimitBurst=10
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=chan
|
||||
Group=chan
|
||||
WorkingDirectory=/opt/chan
|
||||
EnvironmentFile=/etc/chan-live/live.env
|
||||
# SHIP_FROM 在 live.env 里给,形如 sg-collector 或 user@1.2.3.4
|
||||
ExecStart=/opt/chan/.venv/bin/python /opt/chan/live/ship_signals.py \
|
||||
--from ${SHIP_FROM} --remote-bus ${SHIP_REMOTE_BUS}
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
SyslogIdentifier=chan-live-ship
|
||||
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectSystem=strict
|
||||
# chan 用户的家目录放在 /var/lib/chan-live/home,只装拉总线用的那一把 ssh
|
||||
# key。这样 ProtectHome=true 挡住 /home 与 /root 的同时,ssh 仍能读到
|
||||
# ~/.ssh(它在 /var/lib 下,不受 ProtectHome 影响),也能写 known_hosts
|
||||
ProtectHome=true
|
||||
Environment=HOME=/var/lib/chan-live/home
|
||||
ReadWritePaths=/var/lib/chan-live
|
||||
ProtectKernelTunables=true
|
||||
ProtectKernelModules=true
|
||||
RestrictSUIDSGID=true
|
||||
LockPersonality=true
|
||||
MemoryMax=256M
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
Executable
+106
@@ -0,0 +1,106 @@
|
||||
#!/usr/bin/env bash
|
||||
# 空跑验全链:真连交易所读规则/持仓,**不下任何真单**。
|
||||
#
|
||||
# 上真单之前必须过这一步。它验的是那些"上线才会暴露、且暴露方式是花钱"的
|
||||
# 环节:密钥能不能用、IP 白名单对不对、时钟同不同步、搬运通不通、
|
||||
# 数量与价位取整合不合交易所规则。
|
||||
set -euo pipefail
|
||||
|
||||
APP=/opt/chan
|
||||
CONF=/etc/chan-live/live.env
|
||||
STATE=/var/lib/chan-live
|
||||
USER_NAME=chan
|
||||
SECS="${SECS:-90}"
|
||||
|
||||
die() { echo "⛔ $*" >&2; exit 1; }
|
||||
ok() { echo " ✓ $*"; }
|
||||
|
||||
[[ $EUID -eq 0 ]] || die "要 root:sudo $0"
|
||||
[[ -f "$CONF" ]] || die "缺 $CONF,先跑 install.sh"
|
||||
|
||||
set -a; . "$CONF"; set +a
|
||||
|
||||
echo "── 1. 配置自检 ──"
|
||||
for v in BITGET_API_KEY BITGET_API_SECRET BITGET_PASSPHRASE SHIP_FROM; do
|
||||
[[ -n "${!v:-}" ]] || die "$CONF 里 $v 还是空的"
|
||||
done
|
||||
ok "密钥三项与 SHIP_FROM 都已填"
|
||||
[[ "$LIVE_HOME" != /opt/chan* ]] || die "LIVE_HOME 不能指到仓库里(会被 git 清掉)"
|
||||
ok "LIVE_HOME=$LIVE_HOME 在仓库外"
|
||||
|
||||
echo "── 2. 时钟 ──"
|
||||
# staleness 闸靠两机时钟一致才有意义。这里只能验本机;跨机偏差由
|
||||
# ship_signals 在收到信号时报「时间倒流」
|
||||
if command -v chronyc >/dev/null; then
|
||||
chronyc tracking | grep -E "Reference ID|System time|Leap status" | sed 's/^/ /'
|
||||
src="$(chronyc tracking | awk '/Reference ID/{print $NF}')"
|
||||
[[ "$src" != "()" && -n "$src" ]] || die "chrony 还没同步上,等一会再跑"
|
||||
ok "chrony 已同步"
|
||||
else
|
||||
die "没装 chrony。staleness 闸不可信,先 apt install chrony"
|
||||
fi
|
||||
|
||||
echo "── 3. 出口 IP 是否在白名单内 ──"
|
||||
myip="$(curl -s --max-time 10 https://api.ipify.org || true)"
|
||||
[[ -n "$myip" ]] && echo " 本机出口 IP:$myip" || echo " ⚠ 取不到出口 IP"
|
||||
echo " 对照 Bitget 后台该 key 的 IP 白名单,不一致下面会报 40018 之类"
|
||||
|
||||
echo "── 4. 能否 ssh 到采集机 ──"
|
||||
# 不能用 ssh <host> 'echo ok' 来试:采集机那侧的 authorized_keys 用了强制命令
|
||||
# (把这把 key 锁成只能跑 tail,拿不到 shell),任何请求都会变成 tail -F,
|
||||
# 而它**永不返回**——ConnectTimeout 只管建连不管命令时长,测试会永久挂住。
|
||||
# 所以照 ship_signals 的真实用法读流:tail -c +0 会先把整个文件吐出来,
|
||||
# 数一下就等于对端总线的条数,之后它挂着等新内容,由 timeout 收掉。
|
||||
rb="${SHIP_REMOTE_BUS:-\$HOME/chan-live/state/signals_live.jsonl}"
|
||||
err="$(mktemp)"; outf="$(mktemp)"
|
||||
# timeout 要套在 sudo **里面**:套外面时 SIGTERM 只到 sudo,未必传给 ssh,
|
||||
# 于是该被收掉的 tail 会继续挂着
|
||||
sudo -u "$USER_NAME" timeout 12 ssh -T -o BatchMode=yes -o ConnectTimeout=10 \
|
||||
"$SHIP_FROM" "tail -c +0 -F $rb" >"$outf" 2>"$err" || true
|
||||
n="$(wc -l < "$outf")"
|
||||
if grep -qi "Host key verification failed" "$err"; then
|
||||
fp="(在采集机上跑 ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub 拿)"
|
||||
rm -f "$err" "$outf"
|
||||
die "主机指纹没确认过。这不是 key 的问题,装 key 也修不了。
|
||||
$USER_NAME 的 known_hosts 是空的,而 BatchMode=yes 不允许交互确认。
|
||||
**不要**用 StrictHostKeyChecking=no 糊过去,那等于放弃中间人防护。
|
||||
正确做法:在采集机上读出权威指纹 $fp,
|
||||
再在这台上写入并核对:
|
||||
sudo -u $USER_NAME ssh-keyscan -t ed25519 <采集机IP> \\
|
||||
| sudo -u $USER_NAME tee -a $STATE/home/.ssh/known_hosts
|
||||
sudo -u $USER_NAME ssh-keygen -lf $STATE/home/.ssh/known_hosts"
|
||||
elif grep -qiE "Permission denied|publickey" "$err"; then
|
||||
rm -f "$err" "$outf"
|
||||
die "认证被拒。指纹是通的,是 key 没装到采集机上:
|
||||
sudo -u $USER_NAME ssh-keygen -t ed25519 -N '' -f $STATE/home/.ssh/id_ed25519
|
||||
再把 $STATE/home/.ssh/id_ed25519.pub 加到采集机的 authorized_keys"
|
||||
elif [[ -s "$err" ]] && ! grep -q "^" "$outf" 2>/dev/null; then
|
||||
msg="$(head -3 "$err")"; rm -f "$err" "$outf"
|
||||
die "ssh $SHIP_FROM 不通:$msg"
|
||||
else
|
||||
ok "ssh $SHIP_FROM 通"
|
||||
echo " 对端总线现有 $n 条信号"
|
||||
[[ "$n" -gt 0 ]] || echo " ⚠ 对端总线是空的。信号 6.8 个/天,刚重启过就是空的很正常;但要确认采集机接了总线"
|
||||
rm -f "$err" "$outf"
|
||||
fi
|
||||
|
||||
echo "── 5. 执行器空跑 ${SECS}s(真连交易所,不下单)──"
|
||||
# --dry-run 下只读不写:下单一律只打印。合约规则那个端点是**公开**的、不验签,
|
||||
# 所以空跑里专门补了一次带签名的 account() —— 否则这一步会"通过"却根本没测到
|
||||
# 密钥与 IP 白名单,等第一个真信号来时才暴露,而那时信号正在过期
|
||||
# 让 chan 自己 source 配置($CONF 是 640 root:chan,它读得到)。
|
||||
# 不用 `env "$(grep ...)"` 那种拼法:值里有空格就会被切开
|
||||
set +e
|
||||
sudo -u "$USER_NAME" bash -c \
|
||||
"set -a; . '$CONF'; set +a; exec timeout $SECS \
|
||||
'$APP/.venv/bin/python' '$APP/live/live_exec.py' --dry-run"
|
||||
rc=$?
|
||||
set -e
|
||||
# timeout 到点是 124,属于预期
|
||||
[[ $rc -eq 124 || $rc -eq 0 ]] || die "空跑退出码 $rc,看上面报错"
|
||||
ok "空跑没有报错退出"
|
||||
|
||||
echo
|
||||
echo "空跑过了。真跑:"
|
||||
echo " sudo systemctl enable --now chan-live-ship chan-live-exec"
|
||||
echo " $APP/live/deploy/status.sh"
|
||||
Executable
+203
@@ -0,0 +1,203 @@
|
||||
#!/usr/bin/env bash
|
||||
# 在生产机(有 Bitget API key 白名单的那台)上装实盘执行器。
|
||||
#
|
||||
# 只装执行侧:标准库 + aiohttp。**不装** Docker / Hummingbot / pandas /
|
||||
# chanlun 引擎——那些是采集与研究侧的依赖,理由见 live/live_exec.py 文件头。
|
||||
#
|
||||
# sudo ./install.sh # 从当前 checkout 安装
|
||||
# sudo REPO=git@host:jack/chan.git ./install.sh # 或指定远程克隆
|
||||
#
|
||||
# 幂等:重复跑只会更新代码与依赖,不动 /etc/chan-live/live.env 和状态目录。
|
||||
set -euo pipefail
|
||||
|
||||
APP=/opt/chan
|
||||
STATE=/var/lib/chan-live
|
||||
CONF=/etc/chan-live
|
||||
USER_NAME=chan
|
||||
BRANCH="${BRANCH:-chan}"
|
||||
REPO="${REPO:-}"
|
||||
|
||||
die() { echo "⛔ $*" >&2; exit 1; }
|
||||
say() { echo " $*"; }
|
||||
|
||||
[[ $EUID -eq 0 ]] || die "要 root:sudo $0"
|
||||
|
||||
# ── --sync-env:把模板新增的项追加到已有配置 ────────────────────────
|
||||
# 独立成一个模式而不是塞进安装流程:追加会改一个装着密钥的文件,这种事应当
|
||||
# 由你显式发起。只追加缺的项(连它上面的注释一起),**不动**已有任何一行。
|
||||
if [[ "${1:-}" == "--sync-env" ]]; then
|
||||
EX="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/live.env.example"
|
||||
[[ -f "$EX" ]] || die "找不到模板 $EX"
|
||||
[[ -f "$CONF/live.env" ]] || die "$CONF/live.env 还不存在,先跑一次 sudo $0"
|
||||
cp -a "$CONF/live.env" "$CONF/live.env.bak.$(date +%Y%m%d%H%M%S)"
|
||||
python3 - "$EX" "$CONF/live.env" <<'PY'
|
||||
import re
|
||||
import sys
|
||||
|
||||
ex_path, live_path = sys.argv[1], sys.argv[2]
|
||||
ex = open(ex_path, encoding="utf-8").read().splitlines()
|
||||
live = open(live_path, encoding="utf-8").read()
|
||||
KEY = re.compile(r"^([A-Z_][A-Z0-9_]*)=")
|
||||
have = set(KEY.match(x).group(1) for x in live.splitlines() if KEY.match(x))
|
||||
|
||||
add, block = [], []
|
||||
for line in ex:
|
||||
if KEY.match(line):
|
||||
k = KEY.match(line).group(1)
|
||||
if k not in have:
|
||||
add += block + [line]
|
||||
block = []
|
||||
elif line.startswith("#") or (not line.strip() and block):
|
||||
block.append(line)
|
||||
else:
|
||||
block = []
|
||||
|
||||
if not add:
|
||||
print(" 配置已是最新,无需追加")
|
||||
raise SystemExit(0)
|
||||
with open(live_path, "a", encoding="utf-8") as f:
|
||||
f.write("\n\n# ── 以下由 install.sh --sync-env 追加 ──\n")
|
||||
f.write("\n".join(add) + "\n")
|
||||
n = sum(1 for x in add if KEY.match(x))
|
||||
print(f" 追加了 {n} 项:" +
|
||||
" ".join(KEY.match(x).group(1) for x in add if KEY.match(x)))
|
||||
PY
|
||||
say "原文件已备份为 $CONF/live.env.bak.*"
|
||||
say "追加的项多半是空值,逐项填完再重启:sudo systemctl restart chan-live-exec"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── 1. 系统依赖 ────────────────────────────────────────────────────
|
||||
say "装系统包"
|
||||
if command -v apt-get >/dev/null; then
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
apt-get update -qq
|
||||
# chrony 不是可选项:staleness 闸靠两机时钟一致才有意义,
|
||||
# 采集机时钟快 5 分钟就等于把闸放宽 5 分钟(见 ship_signals.py)
|
||||
apt-get install -y -qq python3-venv python3-pip git chrony openssh-client
|
||||
elif command -v dnf >/dev/null; then
|
||||
dnf install -y -q python3 python3-pip git chrony openssh-clients
|
||||
else
|
||||
die "只认 apt/dnf,其他发行版请手工装 python3-venv git chrony"
|
||||
fi
|
||||
systemctl enable --now chrony 2>/dev/null || systemctl enable --now chronyd
|
||||
|
||||
# ── 2. 专用用户与目录 ──────────────────────────────────────────────
|
||||
if ! id -u "$USER_NAME" >/dev/null 2>&1; then
|
||||
say "建系统用户 $USER_NAME(无登录 shell)"
|
||||
useradd --system --home-dir "$STATE/home" --create-home \
|
||||
--shell /usr/sbin/nologin "$USER_NAME"
|
||||
fi
|
||||
install -d -o "$USER_NAME" -g "$USER_NAME" -m 750 "$STATE" "$STATE/state" "$STATE/home"
|
||||
install -d -o "$USER_NAME" -g "$USER_NAME" -m 700 "$STATE/home/.ssh"
|
||||
install -d -o root -g "$USER_NAME" -m 750 "$CONF"
|
||||
|
||||
# ── 3. 代码 ────────────────────────────────────────────────────────
|
||||
if [[ -n "$REPO" ]]; then
|
||||
if [[ -d "$APP/.git" ]]; then
|
||||
say "更新已有 checkout"
|
||||
git -C "$APP" fetch --quiet origin "$BRANCH"
|
||||
git -C "$APP" checkout --quiet "$BRANCH"
|
||||
git -C "$APP" reset --hard --quiet "origin/$BRANCH"
|
||||
else
|
||||
say "克隆 $REPO"
|
||||
rm -rf "$APP"; git clone --quiet --branch "$BRANCH" "$REPO" "$APP"
|
||||
fi
|
||||
else
|
||||
SRC="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
[[ -f "$SRC/live/live_exec.py" ]] || die "$SRC 不像仓库根(缺 live/live_exec.py)"
|
||||
if [[ "$SRC" != "$APP" ]]; then
|
||||
say "从 $SRC 同步代码到 $APP"
|
||||
install -d "$APP"
|
||||
# 只同步生产要的那一个子树。研究侧的 research/ 不上生产机:
|
||||
# 它带 pandas/pyarrow/hummingbot,而且探针 OOM 过一次(14.9GB)
|
||||
cp -a "$SRC/live" "$APP/"
|
||||
fi
|
||||
fi
|
||||
chown -R root:root "$APP/live"
|
||||
find "$APP/live" -type f -exec chmod 644 {} + ; chmod 755 "$APP/live" "$APP/live/deploy"
|
||||
chmod 755 "$APP"/live/deploy/*.sh
|
||||
|
||||
# ── 4. venv ───────────────────────────────────────────────────────
|
||||
say "建 venv 并装依赖(应当只有 aiohttp)"
|
||||
[[ -d "$APP/.venv" ]] || python3 -m venv "$APP/.venv"
|
||||
"$APP/.venv/bin/pip" install --quiet --upgrade pip
|
||||
"$APP/.venv/bin/pip" install --quiet -r "$APP/live/requirements.txt"
|
||||
|
||||
# 断言生产进程没被拖进重量级依赖。这条会真挡住——曾经为读两个常量
|
||||
# import 研究侧的 step43,把 numpy/pandas/pyarrow 全拉进实盘进程
|
||||
say "验依赖面"
|
||||
"$APP/.venv/bin/python" - <<'PY' || die "生产进程拖进了重量级依赖,看上面输出"
|
||||
import sys
|
||||
sys.path.insert(0, "/opt/chan/live")
|
||||
import live_exec
|
||||
live_exec.assert_decomposable()
|
||||
heavy = [m for m in ("numpy", "pandas", "pyarrow", "hummingbot", "scipy")
|
||||
if m in sys.modules]
|
||||
if heavy:
|
||||
print(f" ⛔ 启动路径加载了 {heavy}")
|
||||
raise SystemExit(1)
|
||||
print(f" ✓ 只有标准库 + aiohttp · 出场结构 "
|
||||
f"{live_exec.SL_ATR}/{live_exec.SCALE_ATR}/{live_exec.RUNNER_ATR} ATR "
|
||||
f"/{live_exec.MAXB} 根")
|
||||
PY
|
||||
|
||||
# ── 5. 配置模板 ────────────────────────────────────────────────────
|
||||
if [[ ! -f "$CONF/live.env" ]]; then
|
||||
say "写配置模板 $CONF/live.env(密钥要你手工填)"
|
||||
install -o root -g "$USER_NAME" -m 640 \
|
||||
"$APP/live/deploy/live.env.example" "$CONF/live.env"
|
||||
NEED_FILL=1
|
||||
else
|
||||
say "$CONF/live.env 已存在,不覆盖(里面是密钥)"
|
||||
# 但要报出模板新增的项。不报的话,以后往 example 里加配置,已有部署会
|
||||
# **永远拿不到且毫无提示**——静默漂移,等到出事才发现某个开关根本没生效
|
||||
keys() { grep -oE '^[A-Z_][A-Z0-9_]*=' "$1" | tr -d '=' | sort -u; }
|
||||
MISSING="$(comm -23 <(keys "$APP/live/deploy/live.env.example") \
|
||||
<(keys "$CONF/live.env") | tr '\n' ' ')"
|
||||
if [[ -n "${MISSING// }" ]]; then
|
||||
say "⚠ 模板比你的配置多了这些项:$MISSING"
|
||||
say " 逐项看说明:$APP/live/deploy/live.env.example"
|
||||
say " 要把缺的那几行连注释一起追加过去,跑:"
|
||||
say " sudo $APP/live/deploy/install.sh --sync-env"
|
||||
NEED_FILL=1
|
||||
fi
|
||||
OBSOLETE="$(comm -13 <(keys "$APP/live/deploy/live.env.example") \
|
||||
<(keys "$CONF/live.env") | tr '\n' ' ')"
|
||||
[[ -n "${OBSOLETE// }" ]] && \
|
||||
say " 另有模板里已没有的项(可能已废弃):$OBSOLETE"
|
||||
fi
|
||||
|
||||
# ── 6. systemd ────────────────────────────────────────────────────
|
||||
say "装 systemd 单元"
|
||||
install -m 644 "$APP"/live/deploy/chan-live-*.service /etc/systemd/system/
|
||||
systemctl daemon-reload
|
||||
|
||||
echo
|
||||
echo "装好了。接下来按顺序做(**不要**跳过空跑那步):"
|
||||
echo
|
||||
if [[ -n "${NEED_FILL:-}" ]]; then
|
||||
echo " 1. 填密钥与参数:sudo vi $CONF/live.env"
|
||||
echo " BITGET_API_KEY / SECRET / PASSPHRASE 用只读+交易权限,"
|
||||
echo " **不要开提币权限**。SHIP_FROM 填采集机的 ssh 目标。"
|
||||
echo
|
||||
fi
|
||||
echo " 2. 装拉总线用的 ssh key:"
|
||||
echo " sudo -u $USER_NAME ssh-keygen -t ed25519 -N '' -f $STATE/home/.ssh/id_ed25519"
|
||||
echo " # 把 $STATE/home/.ssh/id_ed25519.pub 加到采集机的 authorized_keys"
|
||||
echo
|
||||
echo " 还要写 known_hosts,否则报 Host key verification failed —— 服务跑"
|
||||
echo " 起来会撞同一个墙,因为 BatchMode=yes 不允许交互确认:"
|
||||
echo " sudo -u $USER_NAME ssh-keyscan -t ed25519 <采集机IP> \\"
|
||||
echo " | sudo -u $USER_NAME tee -a $STATE/home/.ssh/known_hosts"
|
||||
echo " sudo -u $USER_NAME ssh-keygen -lf $STATE/home/.ssh/known_hosts"
|
||||
echo " # 把指纹跟采集机上 ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub"
|
||||
echo " # 的输出比一遍。**不要**图省事用 StrictHostKeyChecking=no,"
|
||||
echo " # 那等于放弃中间人防护,而这条链路上跑的是下单信号"
|
||||
echo
|
||||
echo " 3. 空跑验全链(不下真单,跑够看到一次心跳再停):"
|
||||
echo " sudo $APP/live/deploy/dryrun.sh"
|
||||
echo
|
||||
echo " 4. 真跑:"
|
||||
echo " sudo systemctl enable --now chan-live-ship chan-live-exec"
|
||||
echo " $APP/live/deploy/status.sh"
|
||||
@@ -0,0 +1,71 @@
|
||||
# 生产配置。装到 /etc/chan-live/live.env,权限 640 root:chan。
|
||||
# **不要提交填好的版本**——这里有交易权限的密钥。
|
||||
#
|
||||
# 改完要重启:sudo systemctl restart chan-live-exec
|
||||
|
||||
# ── Bitget 密钥 ───────────────────────────────────────────────────
|
||||
# 权限只勾「只读」+「交易」,**不要勾提币**。
|
||||
# IP 白名单填这台机的公网出口 IP(curl -s https://api.ipify.org 看)。
|
||||
# 这也是执行器必须跑在这台机上的唯一原因——密钥绑了这个 IP。
|
||||
BITGET_API_KEY=
|
||||
BITGET_API_SECRET=
|
||||
BITGET_PASSPHRASE=
|
||||
|
||||
# ── 信号来源(采集机)─────────────────────────────────────────────
|
||||
# ssh 目标。可以是 ~/.ssh/config 里的别名,或 user@ip
|
||||
SHIP_FROM=sg-collector
|
||||
# 采集机上总线文件的路径(在对端 shell 里展开,可用 ~)
|
||||
SHIP_REMOTE_BUS=~/chan-live/state/signals_live.jsonl
|
||||
|
||||
# ── 状态与总线 ────────────────────────────────────────────────────
|
||||
# 生产状态的根。**不要指到仓库里**:git checkout/clean 会动仓库,而这里存的
|
||||
# 是日亏损累计与在场仓位,被清掉等于 MAX_DAY_LOSS / MAX_OPEN 两道闸失忆。
|
||||
# systemd 单元里 ReadWritePaths 也是这个路径,改了要一起改
|
||||
LIVE_HOME=/var/lib/chan-live
|
||||
SIGNAL_BUS=/var/lib/chan-live/state/signals_live.jsonl
|
||||
|
||||
# ── 仓位 ─────────────────────────────────────────────────────────
|
||||
# 每笔名义额(USDT)。杠杆**不改**手续费与滑点(都按名义额收),所以抬名义额
|
||||
# 有真实成本;抬它的唯一理由是压掉步长取整:实测最差币的偏差
|
||||
# 100U → 6.7%(SOL)、500U → 1.8%、1000U → 0.6%
|
||||
LIVE_NOTIONAL=500
|
||||
# 杠杆只影响占用保证金,不影响名义敞口/手续费/滑点/盈亏绝对值。
|
||||
# 名义 500 在 10x 下占 50 USDT 保证金;止损在 2 ATR ≈ 0.2%,而 10x 强平约需
|
||||
# 逆向 10% = 100 个 ATR,差 50 倍。交易所侧记得设**逐仓**
|
||||
LIVE_LEVERAGE=10
|
||||
|
||||
# ── 硬约束:封的是「代价不随仓位缩小」的那几类故障 ─────────────────
|
||||
# 并发仓位数。信号 6.8 笔/天、持仓 48 分钟 → 期望并发 0.23 笔,3 已很宽。
|
||||
# 超了说明有 bug,不是行情好
|
||||
LIVE_MAX_OPEN=3
|
||||
# 日开仓上限。专门封「循环里的 bug 反复开仓」——单笔小但笔数无界
|
||||
LIVE_MAX_DAY=15
|
||||
# 日亏损上限(USDT)。一笔止损约 1 USDT,15 笔全亏 15 USDT
|
||||
LIVE_MAX_DAY_LOSS=20
|
||||
# 信号超过这么久就不做。参考成交价是次根开盘价,过期后跑的不是回测那个价。
|
||||
# ⚠️ 这道闸依赖两机时钟一致,chrony 必须在跑(install.sh 会装)
|
||||
LIVE_STALE_S=20
|
||||
|
||||
# 交易的币池。要与采集机一致,否则会收到不做的币的信号(会被忽略但徒增噪声)
|
||||
SYMS=BTC,ETH,SOL,BNB,XRP,DOGE,ADA,AVAX,LINK,LTC
|
||||
|
||||
# 盯交易所侧出场的轮询间隔(秒)。**别关掉这个循环**:止损与止盈在交易所侧
|
||||
# 成交,本进程收不到通知,缺了它 pnl_day 恒为 0,MAX_DAY_LOSS 就是死的
|
||||
LIVE_WATCH_S=10
|
||||
|
||||
# ── Telegram 通知 ─────────────────────────────────────────────────
|
||||
# 拿 token:Telegram 里找 @BotFather → /newbot
|
||||
# 拿 chat id:给 bot 随便发一句,再开
|
||||
# https://api.telegram.org/bot<TOKEN>/getUpdates 看 result[0].message.chat.id
|
||||
#
|
||||
# 留空则完全不推(不报错)。推的内容:开仓、平仓(带已实现盈亏)、被硬约束
|
||||
# 挡住、报错、对账平仓、跨日结算、启动与停机、整点在线。
|
||||
# **不推**信号过期跳过(常态,搬运重连会重放旧信号)和 5 分钟日志心跳。
|
||||
# 量级约 55 条/天上限(含 24 条在线)。
|
||||
TG_TOKEN=
|
||||
TG_CHAT=
|
||||
# 前缀,用来和采集机推的手工信号区分开——两边可以共用同一个 bot 和对话
|
||||
TG_TAG=实盘
|
||||
# 整点在线的间隔(分钟)。0 关掉。60 = 每天 24 条,和成交推送量级相当。
|
||||
# 这条必须带上游新鲜度:执行器活着不代表链路活着
|
||||
TG_HB_MIN=60
|
||||
Executable
+116
@@ -0,0 +1,116 @@
|
||||
#!/usr/bin/env bash
|
||||
# 生产机一屏体检。不改任何状态,随时可跑。
|
||||
set -uo pipefail
|
||||
|
||||
APP=/opt/chan
|
||||
STATE=/var/lib/chan-live/state
|
||||
CONF=/etc/chan-live/live.env
|
||||
|
||||
hr() { printf '─── %s\n' "$1"; }
|
||||
|
||||
hr "服务"
|
||||
for u in chan-live-ship chan-live-exec; do
|
||||
act="$(systemctl is-active "$u" 2>/dev/null)"
|
||||
since="$(systemctl show -p ActiveEnterTimestamp --value "$u" 2>/dev/null)"
|
||||
nrs="$(systemctl show -p NRestarts --value "$u" 2>/dev/null)"
|
||||
printf ' %-16s %-8s 自 %s · 重启 %s 次\n' \
|
||||
"$u" "$act" "${since:-?}" "${nrs:-0}"
|
||||
done
|
||||
|
||||
hr "时钟(staleness 闸依赖它)"
|
||||
if command -v chronyc >/dev/null; then
|
||||
chronyc tracking 2>/dev/null | grep -E "Reference ID|System time" | sed 's/^/ /'
|
||||
else
|
||||
echo " ⚠ 没装 chrony"
|
||||
fi
|
||||
|
||||
hr "信号总线"
|
||||
bus="${SIGNAL_BUS:-$STATE/signals_live.jsonl}"
|
||||
[[ -f "$CONF" ]] && bus="$(grep -E '^SIGNAL_BUS=' "$CONF" | tail -1 | cut -d= -f2-)"
|
||||
bus="${bus:-$STATE/signals_live.jsonl}"
|
||||
if [[ -s "$bus" ]]; then
|
||||
n="$(wc -l < "$bus")"
|
||||
last_ts="$(tail -1 "$bus" | grep -o '"kline_ts":[0-9]*' | cut -d: -f2)"
|
||||
if [[ -n "$last_ts" ]]; then
|
||||
age=$(( $(date +%s) - last_ts / 1000 ))
|
||||
printf ' %s 条 · 最近一条 %d 分钟前\n' "$n" "$((age / 60))"
|
||||
else
|
||||
echo " $n 条(最后一行没有 kline_ts)"
|
||||
fi
|
||||
# 信号 6.8 个/天,所以「几小时没有」是正常的。真要看的是搬运连着没有
|
||||
echo " 注:6.8 个/天,长时间没有新信号是正常的;要判健康看下面的搬运心跳"
|
||||
else
|
||||
echo " 空或不存在:$bus"
|
||||
fi
|
||||
|
||||
hr "闸的状态"
|
||||
# 在场仓位**不落盘**:重启时由 reconcile 查交易所并平掉(见 live_exec.py
|
||||
# 的 reconcile 注释)。所以这里只报当日计数,仓位要看交易所或下面的成交流
|
||||
if [[ -f "$STATE/live_state.json" ]]; then
|
||||
python3 - "$STATE/live_state.json" <<'PY'
|
||||
import json, sys, time
|
||||
d = json.load(open(sys.argv[1]))
|
||||
today = time.strftime("%Y-%m-%d")
|
||||
day = d.get("day", "?")
|
||||
stale = "" if day == today else f" ⚠ 是 {day} 的,跨日后首次开仓时才归零"
|
||||
print(f" 当日 {day}{stale}")
|
||||
pnl, n = d.get("pnl_day", 0.0), d.get("n_day", 0)
|
||||
# pnl 恒为 0 而又确实开过仓,说明 watch() 没在记账 → MAX_DAY_LOSS 是死的
|
||||
warn = " ⚠ 开过仓但盈亏仍为 0,查 watch() 是否在跑" if n and pnl == 0 else ""
|
||||
print(f" 已开 {n} 笔 · 盈亏 {pnl:+.2f} USDT{warn}")
|
||||
print(f" 已处理信号键 {len(d.get('done', []))} 个(幂等去重用,留最近 5000)")
|
||||
PY
|
||||
else
|
||||
echo " 还没有状态文件(没开过仓)"
|
||||
fi
|
||||
|
||||
hr "最近成交"
|
||||
if [[ -s "$STATE/live_trades.jsonl" ]]; then
|
||||
tail -5 "$STATE/live_trades.jsonl" | sed 's/^/ /'
|
||||
# 「下过单但一次都没建上」是明确的故障,而它的外在表现和"没信号"一样,
|
||||
# 不主动判读就会白跑几小时。首日就是这么丢掉 4 个信号的
|
||||
nf="$(grep -c '"ev": "entry_fail"' "$STATE/live_trades.jsonl" || true)"
|
||||
nb="$(grep -c '"ev": "opened", "key": [^]]*\[{' "$STATE/live_trades.jsonl" || true)"
|
||||
ne="$(grep -c '"ev": "entry"' "$STATE/live_trades.jsonl" || true)"
|
||||
if [[ "${nf:-0}" -gt 0 ]]; then
|
||||
echo
|
||||
echo " ⛔ 有 $nf 次入场被拒(共尝试 $ne 个信号)"
|
||||
echo " 最后一条错误:"
|
||||
grep '"ev": "entry_fail"' "$STATE/live_trades.jsonl" | tail -1 \
|
||||
| sed 's/^/ /'
|
||||
echo " 40774 = 持仓模式不匹配 · 40762/40786 = 保证金不足"
|
||||
fi
|
||||
else
|
||||
echo " 还没有成交记录"
|
||||
fi
|
||||
|
||||
hr "搬运存活文件"
|
||||
if [[ -f "$STATE/ship_alive.json" ]]; then
|
||||
python3 - "$STATE/ship_alive.json" <<'PY'
|
||||
import json, sys, time
|
||||
d = json.load(open(sys.argv[1]))
|
||||
age = time.time() - d.get("ts", 0)
|
||||
up = d.get("up_s", 0)
|
||||
conn = d.get("connected")
|
||||
if age > 720:
|
||||
state = f"⛔ 已停更 {age/60:.0f} 分钟,进程可能死了"
|
||||
elif conn is False:
|
||||
state = "⛔ ssh 已断开,正在重连"
|
||||
elif conn is True:
|
||||
state = f"ssh 在线 {up/60:.0f} 分钟"
|
||||
else:
|
||||
state = "文件是旧格式(没有 connected),重启搬运后才会有"
|
||||
print(f" {state} · 重连 {d.get('n_reconnect', 0)} 次 · 新增 {d.get('n_new', 0)} 条")
|
||||
PY
|
||||
else
|
||||
echo " 还没有 ship_alive.json(搬运没起来,或还是没落盘的旧版本)"
|
||||
fi
|
||||
|
||||
hr "搬运心跳(最近 3 条)"
|
||||
journalctl -u chan-live-ship -n 200 --no-pager 2>/dev/null \
|
||||
| grep -F "[心跳]" | tail -3 | sed 's/^/ /' \
|
||||
|| echo " 还没有心跳(每 5 分钟一条)"
|
||||
|
||||
hr "最近报错"
|
||||
journalctl -u chan-live-exec -u chan-live-ship -n 400 --no-pager -p warning 2>/dev/null \
|
||||
| tail -8 | sed 's/^/ /' || echo " 无"
|
||||
@@ -0,0 +1,118 @@
|
||||
"""钱在哪个账户里。
|
||||
|
||||
用来解一个具体的矛盾:你确认往 U 本位合约充了钱,但
|
||||
`/api/v2/mix/account/account` 报的 accountEquity 只有一小部分。
|
||||
|
||||
`accountEquity` 是**总权益**而不是可用余额,locked 也是 0,所以不是被挂单
|
||||
或持仓占着。剩下的可能都是「这把 key 看到的不是你充钱的那个账户」:
|
||||
|
||||
· key 属于子账户,钱在主账户(或反过来)
|
||||
· 钱在现货账户,没划转到合约
|
||||
· 钱在 USDC 本位 / 币本位合约,不是 USDT 本位
|
||||
· 账户已迁到统一账户(UTA),经典 mix 接口读到的不是同一个池子
|
||||
|
||||
`/api/v2/account/all-account-balance` 会按账户类型列出全部余额,一次看清。
|
||||
|
||||
sudo -u chan bash -c 'set -a; . /etc/chan-live/live.env; set +a; \
|
||||
/opt/chan/.venv/bin/python /opt/chan/live/deploy/whereismoney.py'
|
||||
|
||||
只读,不下单、不划转。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
|
||||
from bitget_rest import MARGIN_COIN, PRODUCT, Bitget # noqa: E402
|
||||
|
||||
|
||||
def _perm_hint(e: Exception) -> str:
|
||||
"""把 40014 说成"正常"而不是"故障"。
|
||||
|
||||
这把 key 刻意只开合约权限,所以现货类端点必然报 40014。**不要**为了让
|
||||
这个探针看全就去加现货权限——那是白扩爆炸半径,而同样的信息在 App 里
|
||||
看一眼就有。
|
||||
"""
|
||||
s = str(e)
|
||||
if "40014" in s:
|
||||
return ("跳过:这把 key 没开现货权限(刻意的,最小权限)。"
|
||||
"这一栏改用 Bitget App 看,别为了探针去加权限")
|
||||
return f"查不了:{type(e).__name__}: {e}"
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
api = Bitget()
|
||||
if not (api.key and api.secret and api.passphrase):
|
||||
raise SystemExit("⛔ 没读到密钥。要 source /etc/chan-live/live.env")
|
||||
try:
|
||||
print("── 跨账户类型总览 ──")
|
||||
try:
|
||||
for b in await api._req("GET", "/api/v2/account/all-account-balance") or []:
|
||||
amt = float(b.get("usdtBalance") or 0)
|
||||
flag = " ← 钱在这里" if amt > 1 else ""
|
||||
print(f" {b.get('accountType'):<16} {amt:>12.2f} USDT{flag}")
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f" {_perm_hint(e)}")
|
||||
|
||||
print(f"\n── {PRODUCT} 下的各保证金币种 ──")
|
||||
try:
|
||||
rows = await api._req("GET", "/api/v2/mix/account/accounts",
|
||||
{"productType": PRODUCT}) or []
|
||||
for a in rows:
|
||||
eq = float(a.get("accountEquity") or 0)
|
||||
if eq or a.get("marginCoin") == MARGIN_COIN:
|
||||
print(f" {a.get('marginCoin'):<8} 权益 {eq:>12.4f} · "
|
||||
f"可用 {float(a.get('available') or 0):>12.4f}")
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f" {_perm_hint(e)}")
|
||||
|
||||
print("\n── 其他合约类型(钱可能充错了本位)──")
|
||||
for pt in ("coin-futures", "usdc-futures"):
|
||||
try:
|
||||
rows = await api._req("GET", "/api/v2/mix/account/accounts",
|
||||
{"productType": pt}) or []
|
||||
hit = [(a.get("marginCoin"), float(a.get("accountEquity") or 0))
|
||||
for a in rows if float(a.get("accountEquity") or 0) > 0]
|
||||
print(f" {pt:<14} {hit if hit else '空'}")
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f" {pt:<14} 查不了:{type(e).__name__}")
|
||||
|
||||
print("\n── 现货 ──")
|
||||
try:
|
||||
rows = await api._req("GET", "/api/v2/spot/account/assets") or []
|
||||
hit = [(a.get("coin"), float(a.get("available") or 0)) for a in rows
|
||||
if float(a.get("available") or 0) > 0]
|
||||
print(f" {hit if hit else '空'}"
|
||||
f"{' ← 要划转到 U 本位合约' if hit else ''}")
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f" {_perm_hint(e)}")
|
||||
|
||||
print("\n── 这把 key 属于哪个账户 ──")
|
||||
for path in ("/api/v2/spot/account/info", "/api/v2/user/account-info"):
|
||||
try:
|
||||
d = await api._req("GET", path) or {}
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f" {path} 查不了:{type(e).__name__}: {e}")
|
||||
continue
|
||||
uid, par = d.get("userId"), d.get("parentId")
|
||||
print(f" userId {uid}")
|
||||
if par:
|
||||
print(f" parentId {par} → **这是子账户**。主账户的钱这把 key"
|
||||
f" 看不到也动不了,这是好事(爆炸半径被账户边界封住)。"
|
||||
f"\n 但充值要充到 userId {uid} 的 U 本位合约里。"
|
||||
f"\n 注意主→子划转默认落在子账户的**现货**钱包,"
|
||||
f"还要在子账户内部再划一次到 U 本位合约")
|
||||
else:
|
||||
print(" 没有 parentId → 这是主账户")
|
||||
print(f" 权限 {d.get('authorities')}(没有现货权限是刻意的)")
|
||||
print(f" IP 白名单 {d.get('ips')}")
|
||||
break
|
||||
finally:
|
||||
await api.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,27 @@
|
||||
"""出场结构的唯一来源。**只用标准库**,这是硬约束。
|
||||
|
||||
为什么单独一个模块、且不许引第三方库:执行器要跑在只装了 `aiohttp` 的生产机
|
||||
上。这几个数原先从 `research/step43_fill_aware_budget.py` 读,那个模块顶层
|
||||
`import pandas`,于是生产机为了两个 float 得装 pandas + pyarrow(实测
|
||||
`assert_decomposable()` 一调就把 numpy/pandas/pyarrow 全拖进来)。
|
||||
|
||||
方向也要注意:**生产拥有这个契约,研究侧反过来读它。** 反过来写成生产 import
|
||||
研究侧,就等于把回测的依赖树绑到实盘进程上。
|
||||
|
||||
⚠️ 研究侧还散着 6 处同样的字面量(step44/47/48/49/52 与 exit_model 的默认
|
||||
参数),本次没有统一。改这里的值**不会**自动改到那些脚本,对表要手工。
|
||||
统一它们要重跑那批脚本确认结果不变,属于独立的一次改动。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
# 以 ATR 为单位的出场结构。来源:research/step43_fill_aware_budget.py 的
|
||||
# 参数扫描结论(1m 主线)。含义见 live_exec.py 顶部注释
|
||||
SL = 2.0 # 止损:入场价的 2 ATR
|
||||
SCALE_AT = 3.0 # 减半点:3 ATR 处平掉一半
|
||||
RUNNER = 8.0 # 剩余半仓的目标:8 ATR
|
||||
RUNNER_STOP = 2.0 # 剩余半仓的止损,**不移动**,仍在入场价的 2 ATR
|
||||
MAXB = 48 # 超时:48 根(1m 上即 48 分钟)
|
||||
|
||||
# 两个半仓共用同一个不动止损,这是「一次入场拆两腿」能等价于回测的前提。
|
||||
# RUNNER_STOP != SL 时该等价性失效,`live_exec.assert_decomposable()` 会硬挡
|
||||
DECOMPOSABLE = RUNNER_STOP == SL
|
||||
@@ -0,0 +1,880 @@
|
||||
"""自动化小额实盘执行器。读信号总线,直接调 Bitget v2 REST 下单。
|
||||
|
||||
## 为什么不用 Hummingbot 的 PositionExecutor
|
||||
|
||||
它的连接器只暴露 LIMIT / LIMIT_MAKER / MARKET,没有触发单,于是
|
||||
`control_stop_loss()` 只能在本地盯价、触发时才发市价单——**进程一死仓位就是
|
||||
裸的**。而交易所本身支持 `place-order` 带 `presetStopLossPrice`,下单时就把
|
||||
止损挂到服务端。绕过连接器不是图省事,是为了消掉一整类故障。
|
||||
|
||||
另外 `TripleBarrierConfig` 只有单级止盈,装不下 3 ATR 减半 + 8 ATR 目标;
|
||||
自己写反而更短。
|
||||
|
||||
## 出场结构为什么能拆成两个半仓
|
||||
|
||||
回测结构是 2 ATR 止损 / 3 ATR 减半 / 8 ATR 目标 / 48 根超时,且**剩余半仓的
|
||||
止损保持在入场价的 2 ATR、不移动**。已核实 `research/lib/exit_model.py:151`——
|
||||
`runner_stops` 的 `ret` 是 `(entry - low[j]) / a`,从入场价算,且
|
||||
`RUNNER_STOP == SL == 2.0`。两半共用同一个不动的止损,所以:
|
||||
|
||||
半仓 A 市价入场 + 服务端止损 2 ATR · maker 止盈 3 ATR
|
||||
半仓 B 市价入场 + 服务端止损 2 ATR · maker 止盈 8 ATR
|
||||
|
||||
止损先到则两半都在 -2 ATR 出场;3 ATR 先到则 A 出场、B 继续且止损仍在 2 ATR。
|
||||
与回测逐情形一致。若哪天把 RUNNER_STOP 改成不等于 SL(比如移到成本),这个
|
||||
分解就**不再成立**,`assert_decomposable()` 会在启动时挡住。
|
||||
|
||||
## 三条出场腿各自挂在哪
|
||||
|
||||
止损 交易所侧(presetStopLossPrice,随入场单一起到)→ 进程死了仍在
|
||||
止盈 交易所侧(post_only reduce-only 限价) → 进程死了仍在
|
||||
超时 **本进程**,48 分钟到点市价平
|
||||
|
||||
所以进程死掉只会让持仓超过 48 根,不会变成裸仓——退化是良性的。
|
||||
|
||||
## 硬约束才是这个文件的重点
|
||||
|
||||
一笔止损只亏约 1 USDT,所以"亏损可控"对单笔成立。但三类故障的代价**不随仓位
|
||||
缩小**,必须显式封住:
|
||||
|
||||
失控下单 循环里的 bug 反复开仓,单笔小但笔数无界 → MAX_OPEN / MAX_DAY
|
||||
亏损累积 策略真的不行,但没人盯着 → MAX_DAY_LOSS
|
||||
裸仓 进程在"已入场、止损未挂"之间死掉 → 服务端止损 + 重启对账
|
||||
|
||||
## 为什么单独一个 live/ 子树、不放在 research/ 下
|
||||
|
||||
生产与研究共处一个目录/进程/机器有四条具体代价,其中第一条已经咬过一次:
|
||||
|
||||
1. `live_state.json` 原先落在 `research/out/`,而那里 `shadow_hb.py` 会在
|
||||
CSV 表头变化时自动 rename 归档、研究脚本会写、人也会手工清数据。那个文件
|
||||
装的是 MAX_DAY_LOSS 累计与在场仓位,**闸的状态被清掉不报错,只是静默
|
||||
失效**。所以生产状态改到独立目录(LIVE_HOME)。
|
||||
2. 采集器十币清空 300~560ms,直接叠在信号到达执行器的延迟上。
|
||||
3. 研究侧的探针 OOM 过一次(14.9GB、负载 12),当时若有仓位在场,执行器会
|
||||
被一起杀掉,只剩交易所侧止损兜着。
|
||||
4. 依赖面:本文件只需标准库 + aiohttp。原先为读两个常量 import 研究侧的
|
||||
step43,把 numpy/pandas/pyarrow 全拖进生产进程。
|
||||
|
||||
因此本目录**不 import research/ 下的任何东西**(`exit_params.py` 是生产自己
|
||||
持有的契约,研究侧反过来读它)。
|
||||
|
||||
python live/live_exec.py --dry-run # 只打印不下单
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import signal
|
||||
import sys
|
||||
import time
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve()
|
||||
# 只插自己所在目录。**不要**把 research/ 加进来——见文件头第 4 条
|
||||
sys.path.insert(0, str(HERE.parent))
|
||||
|
||||
import signal_bus # noqa: E402
|
||||
from bitget_rest import Bitget # noqa: E402
|
||||
import tg # noqa: E402
|
||||
from exit_params import MAXB, RUNNER, RUNNER_STOP, SCALE_AT, SL # noqa: E402
|
||||
|
||||
# 生产状态的根目录。默认放 ~/chan-live,**不落在仓库里**:仓库会被 git
|
||||
# checkout/clean 动,而这里存的是日亏损累计与在场仓位,丢了等于闸失忆
|
||||
LIVE_HOME = Path(os.environ.get("LIVE_HOME", Path.home() / "chan-live"))
|
||||
|
||||
SYMS = os.environ.get(
|
||||
"SYMS", "BTC,ETH,SOL,BNB,XRP,DOGE,ADA,AVAX,LINK,LTC").split(",")
|
||||
# 只认这些交易对。交易所的 all-position 返回**账户全部**仓位,不过滤的话
|
||||
# 账户上任何第三方仓位(手工单、另一个策略、试单忘了平)都会被 reconcile
|
||||
# 在下次重启时市价平掉;watch() 还会因为该 symbol 一直在场而永不结算对应的
|
||||
# key,MAX_OPEN 名额泄漏、pnl_day 不再更新。把「账户只归执行器」这个前提
|
||||
# 从口头约定变成代码里的过滤
|
||||
PAIRS = frozenset(f"{s}USDT" for s in SYMS)
|
||||
NOTIONAL = float(os.environ.get("LIVE_NOTIONAL", "500"))
|
||||
LEVERAGE = int(os.environ.get("LIVE_LEVERAGE", "10"))
|
||||
|
||||
# ── 硬约束 ────────────────────────────────────────────────────────────
|
||||
# 并发仓位数。1 笔约占 50 USDT 保证金,3 笔 150 USDT。信号速率 5.3 笔/天、
|
||||
# 持仓 48 分钟,期望并发只有 0.18 笔,所以 3 已经很宽——超了说明有 bug
|
||||
MAX_OPEN = int(os.environ.get("LIVE_MAX_OPEN", "3"))
|
||||
# 日开仓上限。实测 5.3 笔/天,给 3 倍余量。这一条专门封"失控下单"
|
||||
MAX_DAY = int(os.environ.get("LIVE_MAX_DAY", "15"))
|
||||
# 日亏损上限(USDT)。一笔止损约 1 USDT,15 笔全亏 15 USDT
|
||||
MAX_DAY_LOSS = float(os.environ.get("LIVE_MAX_DAY_LOSS", "20"))
|
||||
# 信号超过这么久就不做了。参考成交价是次根开盘价,过期后跑的不是回测那个价
|
||||
STALE_S = float(os.environ.get("LIVE_STALE_S", "20"))
|
||||
# 盯交易所侧出场的轮询间隔。10s 足够:出场后要做的只是记账与放开 MAX_OPEN
|
||||
# 名额,不涉及下单时效。太密会白耗 API 配额
|
||||
WATCH_S = float(os.environ.get("LIVE_WATCH_S", "10"))
|
||||
# 整点在线推送的间隔(分钟),0 关掉。60 分钟 = 24 条/天,和成交推送量级相当
|
||||
# 不会淹掉真事。调到 5 以下没意义:日志心跳就是 5 分钟一次
|
||||
TG_HB_MIN = float(os.environ.get("TG_HB_MIN", "60"))
|
||||
|
||||
STATE = Path(os.environ.get("LIVE_STATE", LIVE_HOME / "state" / "live_state.json"))
|
||||
TRADES = Path(os.environ.get("LIVE_TRADES", LIVE_HOME / "state" / "live_trades.jsonl"))
|
||||
|
||||
# 出场结构从 exit_params 读,本文件不再抄一份字面量。抄一份的问题不是难看,
|
||||
# 是改了回测参数后这边不会跟上,而且不报错
|
||||
SL_ATR, SCALE_ATR, RUNNER_ATR = SL, SCALE_AT, RUNNER
|
||||
|
||||
|
||||
def assert_decomposable() -> None:
|
||||
"""两个半仓的分解依赖 RUNNER_STOP == SL,不成立就必须停机。
|
||||
|
||||
若有人把剩余半仓的止损改成移到成本(RUNNER_STOP=0)或任何 != SL 的值,
|
||||
这个分解就变成"两半共用同一止损"的错误近似,实盘跑的是另一个收益结构,
|
||||
而且不会报错。所以在启动时硬挡。
|
||||
"""
|
||||
if float(RUNNER_STOP) != float(SL):
|
||||
raise SystemExit(
|
||||
f"⛔ RUNNER_STOP({RUNNER_STOP}) != SL({SL}),两个半仓的分解不再\n"
|
||||
f" 成立。live_exec 的出场结构会与回测不一致且不报错。\n"
|
||||
f" 要改成单执行器 + 手工两级止盈,或把这两个值改回一致。")
|
||||
|
||||
|
||||
class Guard:
|
||||
"""硬约束与当日计数。状态落盘,重启后不清零。
|
||||
|
||||
不落盘的话,进程反复重启就等于反复重置日上限——"失控下单"这一类恰好常常
|
||||
伴随反复重启,那时上限必须还记得。
|
||||
"""
|
||||
|
||||
def __init__(self, path: Path = STATE):
|
||||
self.path = path
|
||||
self.day = time.strftime("%Y-%m-%d")
|
||||
self.n_day = 0
|
||||
self.pnl_day = 0.0
|
||||
self.done: set = set()
|
||||
self.pending_roll: tuple | None = None
|
||||
self._load()
|
||||
|
||||
def _load(self) -> None:
|
||||
try:
|
||||
d = json.loads(self.path.read_text())
|
||||
except Exception:
|
||||
return
|
||||
# 跨日则计数归零,但已处理过的信号键要保留,否则会重开旧仓
|
||||
if d.get("day") == self.day:
|
||||
self.n_day = int(d.get("n_day", 0))
|
||||
self.pnl_day = float(d.get("pnl_day", 0.0))
|
||||
self.done = set(d.get("done", []))
|
||||
|
||||
def save(self) -> None:
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = self.path.with_suffix(".tmp")
|
||||
# 原子替换:直接覆写时若在写一半崩溃,状态文件会变成半个 JSON,
|
||||
# 重启后读不出来 → 日计数归零 → 上限失效
|
||||
tmp.write_text(json.dumps({
|
||||
"day": self.day, "n_day": self.n_day, "pnl_day": self.pnl_day,
|
||||
# 只留最近的,否则文件无界增长
|
||||
"done": sorted(self.done)[-5000:]}))
|
||||
tmp.replace(self.path)
|
||||
|
||||
def roll(self) -> None:
|
||||
today = time.strftime("%Y-%m-%d")
|
||||
if today != self.day:
|
||||
print(f" [guard] 跨日 {self.day} → {today},"
|
||||
f"当日 {self.n_day} 笔 / PnL {self.pnl_day:+.2f} USDT",
|
||||
flush=True)
|
||||
# roll() 是同步的,推送要 await,所以只留个待发件,由心跳取走
|
||||
self.pending_roll = (self.day, self.n_day, self.pnl_day)
|
||||
self.day, self.n_day, self.pnl_day = today, 0, 0.0
|
||||
self.save()
|
||||
|
||||
def blocks(self, key: str, n_open: int) -> str | None:
|
||||
"""返回拒绝原因,None 表示放行。"""
|
||||
self.roll()
|
||||
if key in self.done:
|
||||
return "已处理过(幂等)"
|
||||
if n_open >= MAX_OPEN:
|
||||
return f"并发仓位已达上限 {MAX_OPEN}"
|
||||
if self.n_day >= MAX_DAY:
|
||||
return f"当日开仓已达上限 {MAX_DAY}"
|
||||
if self.pnl_day <= -MAX_DAY_LOSS:
|
||||
return (f"当日亏损 {self.pnl_day:.2f} 已达上限 "
|
||||
f"-{MAX_DAY_LOSS},停止开新仓")
|
||||
return None
|
||||
|
||||
def took(self, key: str) -> None:
|
||||
self.done.add(key)
|
||||
self.n_day += 1
|
||||
self.save()
|
||||
|
||||
def realized(self, pnl: float) -> None:
|
||||
self.pnl_day += pnl
|
||||
self.save()
|
||||
|
||||
|
||||
def legs() -> list[dict]:
|
||||
"""两个半仓的止盈位,用 ATR 倍数表达。
|
||||
|
||||
止损两半相同(SL_ATR),所以不写在这里——它在 open_position 里算一次。
|
||||
"""
|
||||
return [{"tag": "scale", "atr": SCALE_ATR},
|
||||
{"tag": "runner", "atr": RUNNER_ATR}]
|
||||
|
||||
|
||||
def oid_of(key: str, tag: str) -> str:
|
||||
"""把信号键变成交易所能接受的 clientOid。
|
||||
|
||||
信号键形如 `SOL:1787904388411:+1`,里面的 `:` 和 `+` 未必被交易所接受,
|
||||
带过去会直接拒单——而拒单发生在入场腿上,等于这笔信号静默漏掉。只留
|
||||
字母数字和下划线。
|
||||
|
||||
clientOid 是**交易所级幂等**:重发同一个 oid 会被拒。这比本地去重可靠,
|
||||
因为「已发出但没收到回复」这种情况本地判不了,重试就会开两次仓。
|
||||
|
||||
调用方拼后缀时也要守这个字符集(用 `_tp` 而不是 `-tp`)。曾经拼过 `-`,
|
||||
和这里的理由自相矛盾;真被拒的话止盈单挂不上,而那条路径只告警不停机,
|
||||
收益结构会静默退化成「只有止损 + 超时」,空跑还验不到(dry 直接返回
|
||||
假成功)。
|
||||
"""
|
||||
# 方向必须显式编码:直接把非字母数字换成下划线,会让 `+1` 和 `-1` 都变成
|
||||
# `_1`,同一根上的多空信号得到相同 oid,第二笔被交易所当重复拒掉
|
||||
k = key.replace(":+1", ":L").replace(":-1", ":S")
|
||||
safe = "".join(c if c.isalnum() else "_" for c in f"{k}_{tag}")
|
||||
return safe[:60]
|
||||
|
||||
|
||||
def log_trade(rec: dict, path: Path = TRADES) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open("a") as f:
|
||||
f.write(json.dumps(rec) + "\n")
|
||||
f.flush()
|
||||
os.fsync(f.fileno())
|
||||
|
||||
|
||||
class Exec:
|
||||
def __init__(self, dry: bool, bus: Path):
|
||||
self.dry = dry
|
||||
self.bus = bus
|
||||
self.guard = Guard()
|
||||
self.api: Bitget | None = None
|
||||
self.rules: dict = {}
|
||||
self.execs: dict = {} # key → 该笔的腿与超时时刻
|
||||
self.offset = 0 # 已读到总线的哪一行
|
||||
self.n_seen = self.n_took = self.n_skip = 0
|
||||
# 建仓成功/失败分开计。只看 n_took 分不出"做了但下单被拒"——首日
|
||||
# 那 5 小时里 n_took=4 而实际一笔都没建上
|
||||
self.n_built = self.n_order_fail = 0
|
||||
self.t0 = time.time()
|
||||
# 置 0 让第一次 5 分钟心跳就推,不必等满一个周期:重启后最该尽早确认
|
||||
# 的是「上游也通」,而这个只有在线那条带得出来
|
||||
self.tg_hb_at = 0.0
|
||||
|
||||
# ── 启动 ──────────────────────────────────────────────────────
|
||||
async def start(self) -> None:
|
||||
assert_decomposable()
|
||||
# 只从"现在"往后做。历史信号的参考成交价早已过期,补做等于随机入场
|
||||
self.offset = sum(1 for _ in signal_bus.read_all(self.bus))
|
||||
print(f" 总线已有 {self.offset} 条历史信号,全部跳过(参考价已过期)",
|
||||
flush=True)
|
||||
|
||||
self.api = Bitget(dry=self.dry)
|
||||
self.rules = await self.api.contracts()
|
||||
print(f" 合约规则 {len(self.rules)} 个", flush=True)
|
||||
# 启动推送兼作"通道通不通"的自检:配错了这里就收不到,而不是等到
|
||||
# 几小时后第一个真信号来时才发现
|
||||
await tg.started(NOTIONAL, LEVERAGE, len(SYMS), self.dry)
|
||||
print(f" Telegram {'已启用' if tg.ENABLED else '未配置(不推送)'}",
|
||||
flush=True)
|
||||
if self.dry:
|
||||
print(" ⚠ 空跑模式:不下真单", flush=True)
|
||||
# 但仍要发一次**带签名**的请求。上面的 contracts() 是公开端点,
|
||||
# 不验签,光靠它空跑会"通过"却根本没测到密钥与 IP 白名单——
|
||||
# 那种假保证比不测更糟:等到第一个真信号来时才暴露,而信号那时
|
||||
# 正在过期,没有从容排查的余地
|
||||
if self.api.key and self.api.secret and self.api.passphrase:
|
||||
try:
|
||||
acc = await self.api.account() or {}
|
||||
print(" ✓ 密钥与 IP 白名单通", flush=True)
|
||||
# 打全几个余额字段,不只看 available:我们用逐仓,真正
|
||||
# 决定能不能开的是 isolatedMaxAvailable。只看一个字段,
|
||||
# 取错了就会把"有钱"误报成"没钱",或者反过来
|
||||
fields = ("accountEquity", "usdtEquity", "available",
|
||||
"isolatedMaxAvailable", "crossedMaxAvailable",
|
||||
"maxTransferOut", "locked", "unrealizedPL")
|
||||
shown = {k: acc.get(k) for k in fields if k in acc}
|
||||
print(f" 余额 {shown}", flush=True)
|
||||
# 逐仓下取 isolatedMaxAvailable,缺了才退回 available
|
||||
av = float(acc.get("isolatedMaxAvailable")
|
||||
or acc.get("available") or 0)
|
||||
need = NOTIONAL / LEVERAGE * MAX_OPEN
|
||||
print(f" 可开保证金 {av:.2f} USDT · {MAX_OPEN} 笔并发"
|
||||
f"需约 {need:.0f}(名义 {NOTIONAL:.0f} / "
|
||||
f"{LEVERAGE}x)", flush=True)
|
||||
if av < need:
|
||||
per = NOTIONAL / LEVERAGE
|
||||
fit = int(av // per) if per > 0 else 0
|
||||
print(f" ⚠ 保证金只够 {fit} 笔,而 MAX_OPEN="
|
||||
f"{MAX_OPEN}。", flush=True)
|
||||
# 这里不只是"少做几笔"。两条腿是分别下单的,第一条
|
||||
# 成了、第二条因保证金不足失败,就留下一个半仓——
|
||||
# 收益结构从「50% 在 3 ATR + 50% 在 8 ATR」变成只剩
|
||||
# 一条腿,而且是静默的。让闸按真实余额拦,比让交易所
|
||||
# 拒单干净
|
||||
print(f" 要么充钱到 {need:.0f}+ USDT,要么把 "
|
||||
f"LIVE_MAX_OPEN 降到 {max(fit, 1)}。不改的话"
|
||||
f"超出的信号会下单失败,且可能只成一条腿、"
|
||||
f"静默变成半仓(收益结构就不是设计的那个了)",
|
||||
flush=True)
|
||||
if fit == 0:
|
||||
print(f" 现在连一笔都开不了(每笔需 "
|
||||
f"{per:.0f} USDT)", flush=True)
|
||||
except Exception as e: # noqa: BLE001
|
||||
# 退出前要关会话。SystemExit 会绕过 run() 里的收尾,
|
||||
# 漏了就在日志尾部留一串 "Unclosed client session",
|
||||
# 把真正的报错顶出视野
|
||||
await self.api.close()
|
||||
raise SystemExit(
|
||||
f"⛔ 带签名的请求失败:{type(e).__name__}: {e}\n"
|
||||
f" 40018 = 出口 IP 不在该 key 的白名单里;\n"
|
||||
f" 40037 = key 不存在(填错或已删);\n"
|
||||
f" 40001/40009 = secret/passphrase 不对;\n"
|
||||
f" 40099 之类 = 权限没开够(要读+交易)。")
|
||||
else:
|
||||
print(" ⚠ 没填密钥,本次**未**验证密钥与 IP 白名单",
|
||||
flush=True)
|
||||
return
|
||||
if not (self.api.key and self.api.secret and self.api.passphrase):
|
||||
await self.api.close()
|
||||
raise SystemExit("⛔ 缺 BITGET_API_KEY / BITGET_API_SECRET / "
|
||||
"BITGET_PASSPHRASE(末项无 API_)。先跑 --dry-run。")
|
||||
await self.setup_symbols()
|
||||
await self.reconcile()
|
||||
# 放在 reconcile 之后:有持仓或挂单时交易所不允许切换持仓模式,而
|
||||
# reconcile 刚把仓位平干净
|
||||
await self.setup_position_mode()
|
||||
|
||||
async def setup_position_mode(self) -> None:
|
||||
"""把持仓模式钉成单向,并读回核对。
|
||||
|
||||
为什么必须显式设而不是假设:持仓模式决定下单体的语法,两者不匹配是
|
||||
**整体拒单**,不是部分降级。实盘上就因为带着 `tradeSide`(双向语法)
|
||||
打到单向账户,连续 8 次下单全被 40774 拒掉——4 个信号 × 2 条腿,而
|
||||
投递链路那时完全正常(延后 0.0~0.2s、ssh 零重连)。
|
||||
|
||||
单向是我们要的:永不同时持有两个方向,且出场腿依赖 `reduceOnly`,
|
||||
而它只在单向模式下有效。
|
||||
"""
|
||||
try:
|
||||
d = await self.api.set_position_mode("one_way_mode") or {}
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f" ⛔ 设持仓模式失败 {type(e).__name__}: {e}", flush=True)
|
||||
print(" 若账户实际是双向持仓,下单会被 40774 全部拒掉。"
|
||||
"先在 App 里平掉所有仓位与挂单,再切成单向", flush=True)
|
||||
await tg.error("设持仓模式失败,下单可能被 40774 全拒", str(e))
|
||||
return
|
||||
got = str(d.get("posMode") or "")
|
||||
if got and got != "one_way_mode":
|
||||
print(f" ⛔ 持仓模式仍是 {got},下单体是单向语法,会被 40774 拒",
|
||||
flush=True)
|
||||
await tg.error(f"持仓模式是 {got},不是单向", "下单会被 40774 拒掉")
|
||||
else:
|
||||
print(f" 已设持仓模式 单向{'(已读回核对)' if got else ''}",
|
||||
flush=True)
|
||||
|
||||
async def setup_symbols(self) -> None:
|
||||
"""逐仓 + 杠杆。每次启动都设一遍,不假设交易所侧的状态。
|
||||
|
||||
杠杆若被人在 App 里改过,仓位大小就不是我们算的那个。设成幂等操作比
|
||||
读回来核对简单,且失败会直接暴露。
|
||||
"""
|
||||
bad: list[str] = []
|
||||
for s in SYMS:
|
||||
sym = f"{s}USDT"
|
||||
for fn, arg in ((self.api.set_margin_mode, "isolated"),
|
||||
(self.api.set_leverage, LEVERAGE)):
|
||||
try:
|
||||
await fn(sym, arg)
|
||||
except Exception as e:
|
||||
print(f" ⚠ {sym} 设置失败 {type(e).__name__}: {e}",
|
||||
flush=True)
|
||||
bad.append(f"{sym}/{fn.__name__}")
|
||||
if not bad:
|
||||
print(f" 已设 {len(SYMS)} 个币为逐仓 {LEVERAGE}x", flush=True)
|
||||
return
|
||||
# 不能无条件报"已设好"。杠杆设失败是有经济后果的:仓位大小由名义额
|
||||
# 算、与杠杆无关,但保证金要求会变。若交易所侧实际是 1x,每笔要 100
|
||||
# USDT 保证金,第 2、3 笔必然失败,且可能只成一条腿变成半仓
|
||||
print(f" ⛔ {len(bad)} 项设置失败,交易所侧的逐仓/杠杆**不是** "
|
||||
f"{LEVERAGE}x:{bad[:6]}{' …' if len(bad) > 6 else ''}",
|
||||
flush=True)
|
||||
print(" 后果不是不能交易,而是保证金要求与我们算的不一致——"
|
||||
"可能只成一条腿、静默变成半仓。先在 App 里核一遍再跑",
|
||||
flush=True)
|
||||
await tg.error(f"{len(bad)} 项逐仓/杠杆设置失败",
|
||||
f"交易所侧不是 {LEVERAGE}x:{bad[:10]}")
|
||||
|
||||
async def reconcile(self) -> None:
|
||||
"""启动时把交易所的实际持仓对上。
|
||||
|
||||
崩溃重启后交易所可能还有仓位。它们的服务端止损仍在(presetStopLossPrice
|
||||
挂在交易所侧,不随进程消失),但**超时腿丢了**,会一直持有到止损或止盈。
|
||||
|
||||
选择平掉而非接管:接管要重建入场价、ATR、剩余半仓状态和已过根数,任一项
|
||||
猜错就让出场结构变成另一个东西且不报错;平掉的代价只是一笔小额亏损,
|
||||
且行为确定。
|
||||
"""
|
||||
try:
|
||||
pos = await self.my_positions()
|
||||
except Exception as e:
|
||||
print(f" ⚠ 对账读持仓失败 {type(e).__name__}: {e}", flush=True)
|
||||
return
|
||||
if not pos:
|
||||
print(" 对账:交易所无持仓,干净启动", flush=True)
|
||||
return
|
||||
print(f" ⚠ 对账:发现 {len(pos)} 个遗留持仓,撤挂单后市价平掉",
|
||||
flush=True)
|
||||
await tg.error(f"对账:发现 {len(pos)} 个遗留持仓,撤挂单后市价平掉",
|
||||
"、".join(f"{p['symbol']} {p['holdSide']} {p['total']}"
|
||||
for p in pos))
|
||||
for p in pos:
|
||||
sym, hs, sz = p["symbol"], p["holdSide"], p["total"]
|
||||
print(f" {sym} {hs} {sz} @ {p.get('openPriceAvg')}",
|
||||
flush=True)
|
||||
try:
|
||||
await self.api.cancel_all(sym)
|
||||
await self.api.close_market(
|
||||
sym, hs, sz, f"recon:{int(time.time() * 1000)}")
|
||||
log_trade({"ev": "reconcile_flatten", "symbol": sym,
|
||||
"hold_side": hs, "size": sz})
|
||||
except Exception as e:
|
||||
print(f" ⛔ 平仓失败 {type(e).__name__}: {e},"
|
||||
f"需人工介入", flush=True)
|
||||
|
||||
# ── 主循环 ────────────────────────────────────────────────────
|
||||
async def poll(self) -> None:
|
||||
while True:
|
||||
try:
|
||||
await self.step()
|
||||
except Exception as e:
|
||||
import traceback
|
||||
print(f" ⛔ 主循环异常 {type(e).__name__}: {e}", flush=True)
|
||||
traceback.print_exc()
|
||||
await asyncio.sleep(0.2)
|
||||
|
||||
async def step(self) -> None:
|
||||
recs = list(signal_bus.read_all(self.bus))
|
||||
if len(recs) <= self.offset:
|
||||
return
|
||||
new, self.offset = recs[self.offset:], len(recs)
|
||||
for r in new:
|
||||
self.n_seen += 1
|
||||
await self.on_signal(r)
|
||||
|
||||
# 一条记录要能下单,这些字段一个都不能缺
|
||||
NEEDED = ("key", "sym", "emit_ms", "direction", "entry_px", "atr_pct")
|
||||
|
||||
async def my_positions(self) -> list:
|
||||
"""只看 SYMS 里那些币的仓位。
|
||||
|
||||
**不要**直接用 api.positions():它返回账户全部仓位,理由见 PAIRS。
|
||||
残留的局限要知道:同一个币上的第三方仓位(比如你手工开了 ADA)仍然
|
||||
分不出来,那需要按 clientOid 逐单认领,成本不划算。所以账户仍应专用。
|
||||
"""
|
||||
return [p for p in await self.api.positions() if p["symbol"] in PAIRS]
|
||||
|
||||
async def on_signal(self, r: dict) -> None:
|
||||
# 先校验再取值。缺了这一步,总线上一条字段不全的记录会抛 KeyError,
|
||||
# 把 poll 任务打死,进而**整个执行器停止交易**——一行坏数据换全面停摆,
|
||||
# 代价完全不对等。搬运侧已挡半行,但挡不住字段级的不全
|
||||
miss = [k for k in self.NEEDED if r.get(k) is None]
|
||||
if miss:
|
||||
self.n_skip += 1
|
||||
print(f" ⊘ 丢弃畸形记录,缺字段 {miss}:{str(r)[:200]}", flush=True)
|
||||
log_trade({"ev": "malformed", "missing": miss, "rec": str(r)[:500]})
|
||||
await tg.error("总线上有畸形记录,已丢弃",
|
||||
f"缺字段 {miss}\n{str(r)[:300]}")
|
||||
return
|
||||
age = time.time() - r["emit_ms"] / 1000.0
|
||||
n_open = sum(1 for v in self.execs.values() if v)
|
||||
why = self.guard.blocks(r["key"], n_open)
|
||||
# 同币占用检查。整套记账隐含「一个币最多一个仓位」这个前提,但它原先
|
||||
# 只存在于口头上。同币开两笔时交易所会**净成一个仓位**,于是:
|
||||
# · watch() 按 pair 判出场,两个 key 同时进 gone;_match_hist 给它们
|
||||
# 返回同一条历史记录,realized() 被调两次 → pnl_day 翻倍。而这正是
|
||||
# MAX_DAY_LOSS 读的数:亏损翻倍会提前停机,盈利翻倍会让闸变迟钝
|
||||
# · sweep() 在第一笔截止时平掉合并后的整个仓位,把第二笔才持有十分钟
|
||||
# 的部分一并平掉
|
||||
# 合并后的行为(一个止损、两个不同价位的止盈、超时一锅端)不是任何一版
|
||||
# 回测建模的东西,所以跳过第二个信号是最接近安全的近似。期望并发 0.18
|
||||
# 笔,这一跳损失极小
|
||||
if why is None:
|
||||
dup = [k for k, v in self.execs.items() if v["sym"] == r["sym"]]
|
||||
if dup:
|
||||
why = (f"{r['sym']} 已有在场仓位 {dup[0]}——同币开两笔会在"
|
||||
f"交易所侧净成一个仓位,把盈亏记账和超时腿都搞错")
|
||||
if why is None and age > STALE_S:
|
||||
why = f"信号已过期 {age:.1f}s > {STALE_S:.0f}s"
|
||||
if why:
|
||||
self.n_skip += 1
|
||||
print(f" ⊘ {r['key']} 跳过:{why}", flush=True)
|
||||
log_trade({"ev": "skip", "key": r["key"], "why": why,
|
||||
"age_s": round(age, 2)})
|
||||
# 只有被硬约束挡住才推。过期跳过是常态(搬运重连会重放旧信号),
|
||||
# 推了会把真事淹掉
|
||||
if "过期" not in why and "已做过" not in why:
|
||||
await tg.blocked(r["key"], why)
|
||||
return
|
||||
|
||||
self.guard.took(r["key"])
|
||||
self.n_took += 1
|
||||
lg = legs()
|
||||
side = "LONG" if r["direction"] > 0 else "SHORT"
|
||||
print(f" ▶ {r['key']} {side} 名义 {NOTIONAL:.0f} {LEVERAGE}x "
|
||||
f"· 延后 {age:.1f}s · ATR {r['atr_pct'] * 1e4:.1f}bp",
|
||||
flush=True)
|
||||
for x in lg:
|
||||
print(f" {x['tag']:<7}止盈 {x['atr']:.0f} ATR = "
|
||||
f"{x['atr'] * r['atr_pct'] * 1e4:.1f}bp · 止损 "
|
||||
f"{SL_ATR * r['atr_pct'] * 1e4:.1f}bp · 超时 {MAXB}min",
|
||||
flush=True)
|
||||
log_trade({"ev": "entry", "key": r["key"], "side": side,
|
||||
"entry_px": r["entry_px"], "atr_pct": r["atr_pct"],
|
||||
"notional": NOTIONAL, "leverage": LEVERAGE,
|
||||
"age_s": round(age, 2), "legs": lg, "dry": self.dry})
|
||||
# 空跑也要走完 open_position:数量取整、价位对齐 tick、请求体构造都在
|
||||
# 那里,跳过等于什么都没验。不下真单由 Bitget(dry=True) 负责
|
||||
await self.open_position(r, lg)
|
||||
|
||||
def qty_of(self, sym: str, entry: float) -> tuple[str, str]:
|
||||
"""入场量与半仓量,都对齐步长。
|
||||
|
||||
入场量取到**步长的偶数倍**,半仓才是精确一半。不这么做 SOL 的半仓会是
|
||||
全仓的 43%(步长 0.1 币 ≈ 10.7 USDT),而回测假设 50/50。
|
||||
"""
|
||||
r = self.rules.get(f"{sym}USDT")
|
||||
if not r:
|
||||
raise RuntimeError(f"{sym} 没有合约规则")
|
||||
step = Decimal(str(r["sizeMultiplier"]))
|
||||
px = Decimal(str(entry))
|
||||
grid = step * 2
|
||||
n = max(Decimal("1"),
|
||||
(Decimal(str(NOTIONAL)) / px / grid).quantize(Decimal("1")))
|
||||
qty = n * grid
|
||||
return str(qty), str(qty / 2)
|
||||
|
||||
def snap(self, sym: str, px: float) -> str:
|
||||
r = self.rules[f"{sym}USDT"]
|
||||
tick = Decimal(str(r["priceEndStep"])) * (
|
||||
Decimal(10) ** -int(r["pricePlace"]))
|
||||
q = (Decimal(str(px)) / tick).quantize(Decimal("1")) * tick
|
||||
return str(q)
|
||||
|
||||
async def open_position(self, r: dict, lg: list[dict]) -> None:
|
||||
"""两笔「市价入场 + 服务端止损」,再各挂一个 maker 止盈。
|
||||
|
||||
止损随入场单一起到交易所(presetStopLossPrice),所以不存在"已入场、
|
||||
止损未挂"的裸仓窗口——那是本地盯价方案最危险的一段。
|
||||
|
||||
止盈单独挂 post_only 限价:成本模型里止盈按 maker 计且不吃滑点,用
|
||||
preset(触发后市价)会让这部分变成 taker,预算就不成立了。
|
||||
"""
|
||||
sym, d = r["sym"], r["direction"]
|
||||
pair = f"{sym}USDT"
|
||||
entry = r["entry_px"]
|
||||
a = entry * r["atr_pct"]
|
||||
_, half = self.qty_of(sym, entry)
|
||||
side = "buy" if d > 0 else "sell"
|
||||
close_side = "sell" if d > 0 else "buy"
|
||||
hold = "long" if d > 0 else "short"
|
||||
stop_px = self.snap(sym, entry - d * SL_ATR * a)
|
||||
|
||||
opened, failed = [], []
|
||||
for x in lg:
|
||||
oid = oid_of(r["key"], x["tag"])
|
||||
try:
|
||||
await self.api.entry_with_stop(pair, side, half, stop_px, oid)
|
||||
except Exception as e:
|
||||
print(f" ⛔ {x['tag']} 入场失败 {e}", flush=True)
|
||||
log_trade({"ev": "entry_fail", "key": r["key"],
|
||||
"tag": x["tag"], "err": str(e)})
|
||||
failed.append(f"{x['tag']} 入场:{e}")
|
||||
continue
|
||||
tp_px = self.snap(sym, entry + d * x["atr"] * a)
|
||||
try:
|
||||
await self.api.tp_limit(pair, close_side, half, tp_px,
|
||||
oid + "_tp")
|
||||
except Exception as e:
|
||||
# 入场成了但止盈没挂上:仓位仍有服务端止损,不是裸仓。
|
||||
# 超时腿会兜住它,所以只告警不强平
|
||||
print(f" ⚠ {x['tag']} 止盈挂单失败 {e}"
|
||||
f"(仓位有服务端止损,超时腿会兜)", flush=True)
|
||||
log_trade({"ev": "tp_fail", "key": r["key"],
|
||||
"tag": x["tag"], "err": str(e)})
|
||||
failed.append(f"{x['tag']} 止盈:{e}")
|
||||
opened.append({"tag": x["tag"], "oid": oid, "size": half,
|
||||
"tp_px": tp_px, "stop_px": stop_px})
|
||||
print(f" {x['tag']:<7}{half} 币 · 止损 {stop_px} · "
|
||||
f"止盈 {tp_px}", flush=True)
|
||||
|
||||
log_trade({"ev": "opened", "key": r["key"], "legs": opened,
|
||||
"stop_px": stop_px})
|
||||
# 下单失败必须推。这类失败是**静默的经济损失**:日志里在报,但表现只是
|
||||
# "一直没开仓",看起来和"没信号"一样。实盘首日就因为这个白跑 5 小时——
|
||||
# 8 次下单全被 40774 拒掉而无人知道。所以推送不是可选的
|
||||
if failed:
|
||||
self.n_order_fail += 1
|
||||
if not opened:
|
||||
await tg.error(
|
||||
f"{r['key']} 建仓全部失败,这个信号丢了",
|
||||
"\n".join(failed) +
|
||||
f"\n\n累计失败 {self.n_order_fail} 次。"
|
||||
f"链路正常但下不了单——常见是下单体字段被拒"
|
||||
f"(40774 持仓模式不匹配)、保证金不足、或该币被限制交易")
|
||||
else:
|
||||
await tg.error(f"{r['key']} 部分腿失败,收益结构已偏离设计",
|
||||
"\n".join(failed))
|
||||
if opened:
|
||||
self.n_built += 1
|
||||
self.execs[r["key"]] = {
|
||||
"sym": sym, "pair": pair, "hold": hold,
|
||||
"deadline": time.time() + MAXB * 60, "legs": opened,
|
||||
# watch() 靠这个时间戳去 history-position 里认领对应的平仓记录
|
||||
"opened_ms": int(time.time() * 1000),
|
||||
"side": "LONG" if d > 0 else "SHORT"}
|
||||
await tg.opened(sym, self.execs[r["key"]]["side"], entry,
|
||||
r["atr_pct"], NOTIONAL, LEVERAGE, stop_px,
|
||||
opened, time.time() - r["emit_ms"] / 1000.0)
|
||||
|
||||
async def watch(self) -> None:
|
||||
"""盯交易所侧的出场,把已实现盈亏记回闸。
|
||||
|
||||
为什么必须有这个循环:止损与止盈都挂在交易所侧,成交时本进程收不到
|
||||
任何通知。缺了它有两个后果,都是静默的:
|
||||
|
||||
1. `Guard.realized()` 没人调用 → `pnl_day` 恒为 0 →
|
||||
**MAX_DAY_LOSS 这道闸完全不生效**。三条硬约束里最重要的一条。
|
||||
2. `self.execs` 的条目要挂到 48 分钟截止才清 → `MAX_OPEN` 把已经
|
||||
出场的仓位继续算在场 → 新信号被白挡掉。方向保守但不是本意。
|
||||
|
||||
盈亏取交易所的 `netProfit`(= pnl + 资金费 + 开平手续费),不自己按
|
||||
标记价估——估会漏掉费用,而且方向总是偏乐观。
|
||||
"""
|
||||
while True:
|
||||
await asyncio.sleep(WATCH_S)
|
||||
if self.dry or not self.execs:
|
||||
continue
|
||||
try:
|
||||
live = {p["symbol"] for p in await self.my_positions()}
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f" ⚠ 盯仓读持仓失败 {type(e).__name__}: {e}", flush=True)
|
||||
continue
|
||||
gone = [k for k, st in self.execs.items()
|
||||
if st["pair"] not in live]
|
||||
if not gone:
|
||||
continue
|
||||
# 两个半仓在同一 symbol 上会被交易所净成一个仓位,所以一个 key
|
||||
# 对应一条历史记录。按最早的入场时间取一次历史,够覆盖全部
|
||||
since = min(self.execs[k]["opened_ms"] for k in gone) - 60_000
|
||||
try:
|
||||
hist = await self.api.history_positions(start_ms=since)
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f" ⚠ 读历史持仓失败 {type(e).__name__}: {e},"
|
||||
f"本轮不结算(下轮重试,不会漏)", flush=True)
|
||||
continue
|
||||
# 一条历史记录只能被一个 key 认领。同币并发已在 on_signal 拦住,
|
||||
# 但记账是花钱的路径:让这个不变量在本地成立,而不是依赖两百行外
|
||||
# 的另一处检查。重复认领会让 realized() 被调两次、pnl_day 翻倍
|
||||
claimed: set = set()
|
||||
for key in gone:
|
||||
st = self.execs[key]
|
||||
rec = self._match_hist(hist, st, claimed)
|
||||
if rec is None:
|
||||
# 常见于刚平掉、历史还没落库。留着下轮再试;真丢了也有
|
||||
# 48 分钟截止那条路兜住 execs 的清理
|
||||
print(f" … {key} 已出场但历史未就绪,下轮再结算",
|
||||
flush=True)
|
||||
continue
|
||||
pnl = float(rec.get("netProfit") or 0.0)
|
||||
self.guard.realized(pnl)
|
||||
self.guard.save()
|
||||
self.execs.pop(key, None)
|
||||
print(f" ◀ {key} 交易所侧出场 · 已实现 {pnl:+.2f} USDT "
|
||||
f"· 当日累计 {self.guard.pnl_day:+.2f}", flush=True)
|
||||
log_trade({"ev": "closed", "key": key, "net_profit": pnl,
|
||||
"open_px": rec.get("openAvgPrice"),
|
||||
"close_px": rec.get("closeAvgPrice")})
|
||||
await tg.closed(st["sym"], st["side"], pnl,
|
||||
float(rec.get("openAvgPrice") or 0),
|
||||
float(rec.get("closeAvgPrice") or 0),
|
||||
self.guard.pnl_day, self.guard.n_day)
|
||||
|
||||
@staticmethod
|
||||
def _match_hist(hist: list, st: dict,
|
||||
claimed: set | None = None) -> dict | None:
|
||||
"""在历史持仓里认领属于这一笔的记录。
|
||||
|
||||
按 symbol + holdSide 匹配,并要求收盘时间不早于入场时间(减 60s 容差,
|
||||
两边时钟与落库都有抖动)。同一 symbol 有多条时取最近的一条。
|
||||
|
||||
`claimed` 装已被别的 key 认走的 positionId,防止两个 key 认到同一条。
|
||||
"""
|
||||
best, best_t, best_id = None, -1.0, None
|
||||
for r in hist:
|
||||
if r.get("symbol") != st["pair"] or r.get("holdSide") != st["hold"]:
|
||||
continue
|
||||
pid = r.get("positionId")
|
||||
if claimed is not None and pid is not None and pid in claimed:
|
||||
continue
|
||||
t = float(r.get("utime") or r.get("uTime") or 0)
|
||||
if t < st["opened_ms"] - 60_000:
|
||||
continue
|
||||
if t > best_t:
|
||||
best, best_t, best_id = r, t, pid
|
||||
if best is not None and claimed is not None and best_id is not None:
|
||||
claimed.add(best_id)
|
||||
return best
|
||||
|
||||
async def sweep(self) -> None:
|
||||
"""超时腿:48 分钟到点市价平。
|
||||
|
||||
这是唯一必须靠本进程存活的出场腿。止损与止盈都在交易所侧,所以进程
|
||||
死掉只会让持仓超过 48 根,不会变成裸仓——退化是良性的。
|
||||
"""
|
||||
while True:
|
||||
await asyncio.sleep(5)
|
||||
now = time.time()
|
||||
for key, st in list(self.execs.items()):
|
||||
if now < st["deadline"]:
|
||||
continue
|
||||
try:
|
||||
pos = [p for p in await self.my_positions()
|
||||
if p["symbol"] == st["pair"]]
|
||||
if not pos:
|
||||
print(f" ◀ {key} 超时前已全部出场", flush=True)
|
||||
log_trade({"ev": "timeout_noop", "key": key})
|
||||
else:
|
||||
for p in pos:
|
||||
await self.api.cancel_all(st["pair"])
|
||||
await self.api.close_market(
|
||||
st["pair"], p["holdSide"], p["total"],
|
||||
oid_of(key, "timeout"))
|
||||
print(f" ◀ {key} 超时市价平 {pos[0]['total']} 币",
|
||||
flush=True)
|
||||
log_trade({"ev": "timeout_close", "key": key,
|
||||
"size": pos[0]["total"]})
|
||||
except Exception as e:
|
||||
print(f" ⛔ {key} 超时平仓失败 {type(e).__name__}: {e}",
|
||||
flush=True)
|
||||
continue
|
||||
self.execs.pop(key, None)
|
||||
|
||||
def ship_state(self) -> dict | None:
|
||||
"""读搬运器落的存活文件。读不到返回 None——那本身就是要报的事。"""
|
||||
try:
|
||||
p = self.bus.parent / "ship_alive.json"
|
||||
with open(p, encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
except Exception: # noqa: BLE001
|
||||
return None
|
||||
|
||||
async def heartbeat(self) -> None:
|
||||
while True:
|
||||
await asyncio.sleep(300)
|
||||
if TG_HB_MIN and time.time() - self.tg_hb_at >= TG_HB_MIN * 60:
|
||||
self.tg_hb_at = time.time()
|
||||
await tg.alive(time.time() - self.t0, self.n_seen,
|
||||
self.n_built, self.n_took,
|
||||
sum(1 for v in self.execs.values() if v),
|
||||
MAX_OPEN, self.guard.n_day, MAX_DAY,
|
||||
self.guard.pnl_day, MAX_DAY_LOSS,
|
||||
self.n_order_fail, self.ship_state(), self.dry)
|
||||
# 跨日结算是 roll() 里同步留下的,在这里发出去
|
||||
self.guard.roll()
|
||||
if self.guard.pending_roll:
|
||||
await tg.day_rolled(*self.guard.pending_roll)
|
||||
self.guard.pending_roll = None
|
||||
n_open = sum(1 for v in self.execs.values() if v)
|
||||
# 「已做 N 但建仓 0」是明确的故障,不能只把数字并排列出来让人自己
|
||||
# 看。首日那 5 小时的心跳里 "已做 4 · 在场 0/3" 一直在打,但没有
|
||||
# 任何一处说这是异常
|
||||
bad = ""
|
||||
if self.n_took and not self.n_built:
|
||||
bad = f" ⛔ 做了 {self.n_took} 笔却一次都没建上,下单被拒"
|
||||
elif self.n_order_fail:
|
||||
bad = f" ⚠ 有 {self.n_order_fail} 次下单失败"
|
||||
print(f" [心跳] 见信号 {self.n_seen} · 已做 {self.n_took} · "
|
||||
f"建仓 {self.n_built} · 跳过 {self.n_skip} · "
|
||||
f"在场 {n_open}/{MAX_OPEN} · "
|
||||
f"当日 {self.guard.n_day}/{MAX_DAY} 笔 · "
|
||||
f"当日 PnL {self.guard.pnl_day:+.2f}/-{MAX_DAY_LOSS}{bad}",
|
||||
flush=True)
|
||||
|
||||
async def shutdown(self) -> None:
|
||||
"""收到停机信号:撤挂单 + 平掉在场仓位,然后退出。
|
||||
|
||||
为什么停机要平仓,而崩溃不需要:崩溃后 systemd/docker 会在几秒内重启,
|
||||
`reconcile` 接着就把遗留仓位清掉,空窗期有交易所侧止损兜着。而**主动
|
||||
停机后没人重启**,仓位会一直挂到止损或止盈——48 分钟超时腿丢了,跑的
|
||||
就不是回测那个出场结构了。
|
||||
|
||||
直接复用 reconcile:它做的正是"撤挂单 + 市价平掉一切"。
|
||||
"""
|
||||
print("\n 收到停机信号,撤挂单并平掉在场仓位", flush=True)
|
||||
await tg.stopping(len(self.execs))
|
||||
try:
|
||||
if self.dry:
|
||||
print(" 空跑模式,无仓位可平", flush=True)
|
||||
else:
|
||||
await asyncio.wait_for(self.reconcile(), timeout=60.0)
|
||||
except asyncio.TimeoutError:
|
||||
print(" ⛔ 平仓超过 60s 未完成。仓位仍有交易所侧止损,"
|
||||
"但超时腿已丢,去交易所确认", flush=True)
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f" ⛔ 停机平仓失败 {type(e).__name__}: {e},需人工介入",
|
||||
flush=True)
|
||||
finally:
|
||||
# 不关会话会在日志里留 "Unclosed client session",且反复重启
|
||||
# (Restart=always)会漏 socket
|
||||
try:
|
||||
await self.api.close()
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
|
||||
async def run(self) -> None:
|
||||
await self.start()
|
||||
stop = asyncio.Event()
|
||||
loop = asyncio.get_running_loop()
|
||||
# 必须显式挂 SIGTERM:docker stop 与 systemd stop 默认发的都是它,
|
||||
# 而 Python 对 SIGTERM 不抛 KeyboardInterrupt,不挂就是直接消失、
|
||||
# 没有任何清理。SIGINT 一并挂上,省得依赖 KillSignal= 那种绕法
|
||||
for sig in (signal.SIGTERM, signal.SIGINT):
|
||||
loop.add_signal_handler(sig, stop.set)
|
||||
work = [asyncio.create_task(c)
|
||||
for c in (self.poll(), self.watch(), self.sweep(),
|
||||
self.heartbeat())]
|
||||
done, _ = await asyncio.wait(
|
||||
[*work, asyncio.create_task(stop.wait())],
|
||||
return_when=asyncio.FIRST_COMPLETED)
|
||||
for t in work:
|
||||
t.cancel()
|
||||
await asyncio.gather(*work, return_exceptions=True)
|
||||
# 任一主循环自己退出(异常)也走这里:仓位不能留给没人管的进程
|
||||
for t in done:
|
||||
if t in work and (exc := t.exception()) is not None:
|
||||
print(f" ⛔ 主循环异常退出 {type(exc).__name__}: {exc}",
|
||||
flush=True)
|
||||
await tg.error("主循环异常退出,正在平仓并退出",
|
||||
f"{type(exc).__name__}: {exc}")
|
||||
await self.shutdown()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--dry-run", action="store_true",
|
||||
help="不连交易所、不下单,只验总线与约束逻辑")
|
||||
ap.add_argument("--bus", default=str(signal_bus.BUS))
|
||||
a = ap.parse_args()
|
||||
|
||||
print(f"实盘执行器 · 名义 {NOTIONAL:.0f} USDT · {LEVERAGE}x · "
|
||||
f"并发≤{MAX_OPEN} · 日开仓≤{MAX_DAY} · 日亏损≤{MAX_DAY_LOSS}")
|
||||
asyncio.run(Exec(a.dry_run, Path(a.bus)).run())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,8 @@
|
||||
# 生产执行器的全部依赖。**保持这个文件只有一行。**
|
||||
#
|
||||
# 每加一个包都要能回答"实盘进程崩在这个包里我怎么办"。numpy/pandas/pyarrow
|
||||
# 曾经因为读两个常量被拖进来(见 live/exit_params.py 的说明),已经切掉。
|
||||
#
|
||||
# 版本下限的理由:3.9 起 aiohttp 才在 Python 3.12+ 上稳定编译;不锁上限是
|
||||
# 因为这里只用 ClientSession.request 这一个最稳定的 API 面。
|
||||
aiohttp>=3.9
|
||||
@@ -0,0 +1,232 @@
|
||||
"""把产信号那台机的总线搬到本机(生产机)。跑在**消费侧**,即 AWS 上。
|
||||
|
||||
## 为什么要搬
|
||||
|
||||
Bitget 的 API key 绑了 IP 白名单,只能从 AWS 那台发单;而信号是新加坡那台
|
||||
采集器算出来的。执行器不自己算信号的理由见 `signal_bus.py` 顶部——最要紧的
|
||||
是「实盘交易的必须是影子测量的那一个信号」,各算一份会悄悄分叉。
|
||||
|
||||
## 为什么是拉而不是推
|
||||
|
||||
拉的一侧是生产机,它对自己的输入负责。推的话,研究机上一个脚本挂了就会静默
|
||||
断供,而生产机看不出区别(信号本来就 6.8 个/天,长时间没有是正常的)。
|
||||
|
||||
## 断线怎么自愈
|
||||
|
||||
每次重连都 `tail -c +0`,即从文件头重放全部内容,本地按 `key` 去重后只追加
|
||||
新的。所以断线期间产生的信号会在重连时补齐,不需要记录偏移量。
|
||||
|
||||
⚠️ 补齐**不等于**补做:重放上来的旧信号会被 `live_exec` 的 `LIVE_STALE_S`
|
||||
(默认 20s)挡掉。这是对的——参考成交价是次根开盘价,过了就不是回测那个价。
|
||||
所以断线超过 20s 就等于漏掉那些信号,这是可接受的退化,不是 bug。
|
||||
|
||||
## 为什么单独一个进程
|
||||
|
||||
搬运挂掉时,执行器要继续管在场仓位(48 分钟超时平仓在本进程里)。合成一个
|
||||
进程会让传输故障连坐到仓位管理。
|
||||
|
||||
python live/ship_signals.py --from sg-collector # 用 ~/.ssh/config 的别名
|
||||
python live/ship_signals.py --from user@1.2.3.4 --remote-bus /home/user/chan-live/state/signals_live.jsonl
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve()
|
||||
sys.path.insert(0, str(HERE.parent))
|
||||
|
||||
import signal_bus # noqa: E402
|
||||
|
||||
# ssh 参数的理由:
|
||||
# BatchMode 不要交互提示密码,否则进程会挂在那里等输入
|
||||
# ServerAliveInterval/CountMax 45s 内探测不到就断开重连。没有这两条,
|
||||
# NAT 静默丢弃连接后 tail 会永远挂着不返回,表现为
|
||||
# 「进程活着但再也收不到信号」——最难发现的那种故障
|
||||
# ExitOnForwardFailure/StrictHostKeyChecking 留默认,主机指纹要人工确认过
|
||||
SSH_OPTS = ["-T", "-o", "BatchMode=yes",
|
||||
"-o", "ServerAliveInterval=15", "-o", "ServerAliveCountMax=3",
|
||||
"-o", "ConnectTimeout=10"]
|
||||
|
||||
REMOTE_BUS = os.environ.get(
|
||||
"SHIP_REMOTE_BUS", "~/chan-live/state/signals_live.jsonl")
|
||||
BACKOFF_MAX = 60.0
|
||||
# 允许的负龄。1s 覆盖正常的 NTP 抖动与网络传输,超出就该当时钟问题查
|
||||
SKEW_TOL_S = 1.0
|
||||
|
||||
|
||||
class Shipper:
|
||||
def __init__(self, host: str, remote_bus: str, local_bus: Path) -> None:
|
||||
self.host = host
|
||||
self.remote_bus = remote_bus
|
||||
self.bus = local_bus
|
||||
self.seen: set[str] = set()
|
||||
self.n_new = 0
|
||||
self.n_dup = 0
|
||||
self.connected_at = 0.0
|
||||
self.last_signal_ts = 0.0
|
||||
self.n_reconnect = 0
|
||||
self.n_skew = 0
|
||||
self.ssh_up = False
|
||||
self.alive = local_bus.parent / "ship_alive.json"
|
||||
|
||||
def load_seen(self) -> None:
|
||||
"""本地已有的键先读进来,避免重启后把整个文件再追加一遍。"""
|
||||
self.bus.parent.mkdir(parents=True, exist_ok=True)
|
||||
for rec in signal_bus.read_all(self.bus):
|
||||
k = rec.get("key")
|
||||
if k:
|
||||
self.seen.add(k)
|
||||
print(f" 本地已有 {len(self.seen)} 条信号,按 key 去重", flush=True)
|
||||
|
||||
def absorb(self, line: str) -> None:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
return
|
||||
try:
|
||||
rec = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
# 半行:tail 在写入中途读到。重连重放时会拿到完整的那一行
|
||||
print(f" ⚠ 跳过无法解析的一行({len(line)} 字节)", flush=True)
|
||||
return
|
||||
k = rec.get("key")
|
||||
if not k:
|
||||
print(f" ⚠ 跳过无 key 的记录:{line[:80]}", flush=True)
|
||||
return
|
||||
if k in self.seen:
|
||||
self.n_dup += 1
|
||||
return
|
||||
self.seen.add(k)
|
||||
self.n_new += 1
|
||||
self.last_signal_ts = time.time()
|
||||
# 原样追加,不重新序列化——保持与源文件逐字节一致,便于事后对账
|
||||
with self.bus.open("a", encoding="utf-8") as f:
|
||||
f.write(line + "\n")
|
||||
f.flush()
|
||||
os.fsync(f.fileno())
|
||||
age = time.time() - rec.get("kline_ts", 0) / 1000.0
|
||||
if age < -SKEW_TOL_S:
|
||||
# 负龄说明产信号那台机的时钟快于本机。这不是无害的:staleness 闸
|
||||
# 靠 age 判断,时钟快 5 分钟就等于把闸放宽 5 分钟,一个早已失效的
|
||||
# 参考价会被当成新鲜的照做。两台都必须挂 NTP(部署文档里是硬要求)
|
||||
self.n_skew += 1
|
||||
print(f" ⛔ {k} 时间倒流 {-age:.1f}s —— 两机时钟不同步,"
|
||||
f"staleness 闸已不可信。查 chronyd/systemd-timesyncd",
|
||||
flush=True)
|
||||
mark = "" if age <= 20 else " ⚠ 已超 20s,执行器会挡掉"
|
||||
print(f" ▶ 收到 {k} · 距参考价成立 {age:.1f}s{mark}", flush=True)
|
||||
|
||||
async def pump(self) -> None:
|
||||
"""连一次,读到断为止。返回即表示需要重连。"""
|
||||
cmd = ["ssh", *SSH_OPTS, self.host,
|
||||
f"tail -c +0 -F {self.remote_bus}"]
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
*cmd, stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE)
|
||||
self.connected_at = time.time()
|
||||
self.ssh_up = True
|
||||
self.touch_alive(0) # 立刻落盘,别等 5 分钟心跳——执行器第一轮
|
||||
# 整点推送会读这个文件,晚写就会误报上游断了
|
||||
print(f" ssh 已连上 {self.host}", flush=True)
|
||||
assert proc.stdout is not None
|
||||
try:
|
||||
async for raw in proc.stdout:
|
||||
self.absorb(raw.decode("utf-8", "replace"))
|
||||
finally:
|
||||
err = b""
|
||||
if proc.stderr is not None:
|
||||
try:
|
||||
err = await asyncio.wait_for(proc.stderr.read(), 2.0)
|
||||
except asyncio.TimeoutError:
|
||||
pass
|
||||
if proc.returncode is None:
|
||||
proc.kill()
|
||||
await proc.wait()
|
||||
up = time.time() - self.connected_at
|
||||
self.ssh_up = False
|
||||
self.touch_alive(up) # 立刻标断开。只靠停更来发现的话,心跳还在
|
||||
# 刷 ts,执行器会以为管道还活着
|
||||
msg = err.decode("utf-8", "replace").strip()
|
||||
print(f" ssh 断开(在线 {up:.0f}s,退出码 {proc.returncode})"
|
||||
f"{':' + msg if msg else ''}", flush=True)
|
||||
|
||||
async def run(self) -> None:
|
||||
self.load_seen()
|
||||
asyncio.create_task(self.heartbeat())
|
||||
backoff = 1.0
|
||||
while True:
|
||||
try:
|
||||
await self.pump()
|
||||
backoff = 1.0 # 正常断开:立刻重连
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f" ⚠ 搬运出错:{type(e).__name__}: {e}", flush=True)
|
||||
self.n_reconnect += 1
|
||||
await asyncio.sleep(backoff)
|
||||
backoff = min(backoff * 2, BACKOFF_MAX)
|
||||
|
||||
async def heartbeat(self) -> None:
|
||||
"""信号 6.8 个/天,所以「很久没收到」是正常的,不能当健康指标。
|
||||
|
||||
真正要报的是**连接**在不在:ssh 在线时长与重连次数。管道死了但进程
|
||||
活着是这里最危险的状态,ServerAliveInterval 负责让它变成一次断开。
|
||||
"""
|
||||
while True:
|
||||
await asyncio.sleep(300)
|
||||
up = time.time() - self.connected_at if self.connected_at else 0
|
||||
last = (f"{(time.time() - self.last_signal_ts) / 60:.0f} 分钟前"
|
||||
if self.last_signal_ts else "本次启动后还没有")
|
||||
skew = f" · ⛔ 时钟倒流 {self.n_skew} 次" if self.n_skew else ""
|
||||
print(f" [心跳] ssh 在线 {up / 60:.0f} 分钟 · 重连 "
|
||||
f"{self.n_reconnect} 次 · 新增 {self.n_new} 条"
|
||||
f"(重放去重 {self.n_dup})· 最近一条 {last}{skew}",
|
||||
flush=True)
|
||||
self.touch_alive(up)
|
||||
|
||||
def touch_alive(self, up: float) -> None:
|
||||
"""把连接状态落到文件,供执行器的整点推送读。
|
||||
|
||||
为什么要落盘:Telegram 推送在执行器那侧,而它看不到本进程的日志。
|
||||
「执行器活着」单独没有意义——搬运管道死掉时执行器一样心跳正常、一样
|
||||
什么都不做,那正是最危险的状态。所以推送里必须带上游的新鲜度,
|
||||
这个文件是唯一的传递途径。
|
||||
|
||||
写失败只打日志:搬运的正事是投信号,不能因为写不了状态文件而中断。
|
||||
"""
|
||||
try:
|
||||
self.alive.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = self.alive.with_suffix(".tmp")
|
||||
tmp.write_text(json.dumps({
|
||||
"ts": time.time(), "up_s": round(up),
|
||||
"connected": self.ssh_up,
|
||||
"n_reconnect": self.n_reconnect, "n_new": self.n_new,
|
||||
"n_skew": self.n_skew,
|
||||
"last_signal_ts": self.last_signal_ts}), encoding="utf-8")
|
||||
tmp.replace(self.alive) # 原子替换,读侧不会看到半个文件
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f" ⚠ 写存活文件失败 {type(e).__name__}: {e}", flush=True)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--from", dest="host", required=True,
|
||||
help="产信号那台机的 ssh 目标,如 sg-collector 或 user@ip")
|
||||
ap.add_argument("--remote-bus", default=REMOTE_BUS,
|
||||
help="对端总线路径(对端 shell 展开,可用 ~)")
|
||||
ap.add_argument("--bus", default=str(signal_bus.BUS),
|
||||
help="本机总线路径,执行器读同一个")
|
||||
a = ap.parse_args()
|
||||
s = Shipper(a.host, a.remote_bus, Path(a.bus))
|
||||
print(f"信号搬运:{a.host}:{a.remote_bus} → {a.bus}", flush=True)
|
||||
try:
|
||||
asyncio.run(s.run())
|
||||
except KeyboardInterrupt:
|
||||
print(" 停止", flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,78 @@
|
||||
"""影子把过滤网的信号写到这里,实盘执行器读这里。
|
||||
|
||||
## 为什么不让实盘自己算信号
|
||||
|
||||
三个理由,第三个最要紧:
|
||||
|
||||
1. 2 核上再来一份十币计算,清空会从 247ms 推到 500ms+
|
||||
2. 实盘进程崩溃不该影响正在采的数据集
|
||||
3. **实盘交易的必须是影子测量的那一个信号。** 各算一份会让两边悄悄分叉,
|
||||
之后就没法把实盘的实际成交和影子测的滑点曲线对照——而那个对照是整件事
|
||||
的目的
|
||||
|
||||
## 为什么用 append-only 文件而不是队列
|
||||
|
||||
崩溃安全 + 留审计轨迹。实盘进程重启后能从文件里看到自己漏掉了哪些信号,
|
||||
而不是像内存队列那样直接消失。文件也让"影子在跑、实盘没在跑"这种状态成为
|
||||
可观测的(信号在攒着),而不是静默丢弃。
|
||||
|
||||
每行一个 JSON,字段见 `emit`。`key` 是幂等键,实盘按它去重——同一根被重复
|
||||
处理(补根、进程池重建后重放)不该开两次仓。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
# 默认不落在仓库里:git checkout/clean 会动仓库,而这里是跨进程(甚至跨机)
|
||||
# 的交接点,被清掉就等于信号静默丢失。产信号的一侧和消费的一侧各自指到
|
||||
# 自己的路径即可,同机时指到同一个文件
|
||||
BUS = Path(os.environ.get(
|
||||
"SIGNAL_BUS", Path.home() / "chan-live" / "state" / "signals_live.jsonl"))
|
||||
|
||||
|
||||
def key_of(sym: str, kline_ts: int, direction: int) -> str:
|
||||
return f"{sym}:{int(kline_ts)}:{int(direction):+d}"
|
||||
|
||||
|
||||
def emit(sym: str, kline_ts: int, direction: int, entry_px: float,
|
||||
atr_pct: float, lag_ms: float, path: Path | None = None) -> None:
|
||||
"""追写一条信号。任何失败只打日志——总线写不进去不能连坐采集。
|
||||
|
||||
`entry_px` 是次根开盘价,也就是回测口径的成交价。实盘据此算止损/止盈的
|
||||
绝对价位,**不要**用实盘自己看到的现价,否则价位会随执行延迟漂移,跑的
|
||||
就不是回测那个结构。
|
||||
"""
|
||||
p = path or BUS
|
||||
rec = {"key": key_of(sym, kline_ts, direction),
|
||||
"sym": sym, "kline_ts": int(kline_ts),
|
||||
"direction": int(direction),
|
||||
"entry_px": float(entry_px), "atr_pct": float(atr_pct),
|
||||
"lag_ms": float(lag_ms),
|
||||
"emit_ms": int(time.time() * 1000)}
|
||||
try:
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
with p.open("a") as f:
|
||||
f.write(json.dumps(rec) + "\n")
|
||||
f.flush()
|
||||
# 实盘要在毫秒级看到,且进程被 SIGKILL 时不能丢——这两点都要求
|
||||
# 落到磁盘,不能只停在 libc 缓冲里
|
||||
os.fsync(f.fileno())
|
||||
except Exception as e:
|
||||
print(f" [bus] 写信号失败 {type(e).__name__}: {e}", flush=True)
|
||||
|
||||
|
||||
def read_all(path: Path | None = None):
|
||||
"""读全部信号。坏行跳过——半行只可能出现在文件末尾的崩溃点。"""
|
||||
p = path or BUS
|
||||
if not p.exists():
|
||||
return
|
||||
for line in p.read_text().splitlines():
|
||||
if not line.strip():
|
||||
continue
|
||||
try:
|
||||
yield json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
+172
@@ -0,0 +1,172 @@
|
||||
"""生产侧 Telegram 通知。只用标准库 + aiohttp。
|
||||
|
||||
## 推什么、不推什么
|
||||
|
||||
推送的价值在于**你会因此做点什么**。按这个标准筛:
|
||||
|
||||
推 开仓 能立刻眼看一遍方向/价位对不对
|
||||
推 平仓 带已实现盈亏(含手续费与资金费),这是唯一的真账
|
||||
推 闸拦截 仅 MAX_OPEN / MAX_DAY / MAX_DAY_LOSS —— 说明有 bug 或策略在流血
|
||||
推 报错、对账平仓、跨日结算
|
||||
推 整点在线 每 60 分钟一条,见下
|
||||
不推 信号过期跳过 这是常态(重连重放会带上旧信号),推了就淹掉真事
|
||||
不推 5 分钟心跳 日志里有,推了每天 288 条
|
||||
|
||||
量级:信号 6.8 个/天、日开仓上限 15、在线 24 条,所以最多约 55 条/天。
|
||||
|
||||
## 整点在线那条为什么不违反上面的标准
|
||||
|
||||
只说「我还活着」的推送看两天就会被忽略,那时它就成了噪声。所以这条必须带
|
||||
**能暴露问题的数字**,尤其是上游新鲜度——执行器活着不代表链路活着,搬运
|
||||
管道死掉时执行器一样心跳正常、一样什么都不做,那是最危险的状态。异常时这
|
||||
条会显式标出来,而不是把数字并排列出来让人自己看。
|
||||
|
||||
## 失败一律只打日志
|
||||
|
||||
推送挂了不能连坐交易。所有异常在这里吞掉——上层不该因为 Telegram 抽风而
|
||||
影响下单或平仓。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import time
|
||||
|
||||
TOKEN = os.environ.get("TG_TOKEN", "")
|
||||
CHAT = os.environ.get("TG_CHAT", "")
|
||||
ENABLED = bool(TOKEN and CHAT)
|
||||
# 生产机上给这条推送打个前缀,免得和采集机推的手工信号混在一个对话里分不清
|
||||
TAG = os.environ.get("TG_TAG", "实盘")
|
||||
|
||||
|
||||
async def send(text: str) -> None:
|
||||
"""推一条。任何失败都只打日志——推送挂了不能连坐交易。"""
|
||||
if not ENABLED:
|
||||
return
|
||||
try:
|
||||
import aiohttp
|
||||
url = f"https://api.telegram.org/bot{TOKEN}/sendMessage"
|
||||
async with aiohttp.ClientSession() as s:
|
||||
async with s.post(url,
|
||||
json={"chat_id": CHAT, "text": text},
|
||||
timeout=aiohttp.ClientTimeout(total=10)) as r:
|
||||
if r.status != 200:
|
||||
body = (await r.text())[:200]
|
||||
print(f" [TG] 推送失败 HTTP {r.status} {body}", flush=True)
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f" [TG] 推送异常 {type(e).__name__}: {e}", flush=True)
|
||||
|
||||
|
||||
def _fmt(px: float) -> str:
|
||||
"""按量级选小数位。跨币种从 79000 到 0.087,固定位数会难读或丢精度。"""
|
||||
a = abs(px)
|
||||
if a >= 1000:
|
||||
return f"{px:.1f}"
|
||||
if a >= 10:
|
||||
return f"{px:.3f}"
|
||||
if a >= 1:
|
||||
return f"{px:.4f}"
|
||||
return f"{px:.6f}"
|
||||
|
||||
|
||||
async def opened(sym: str, side: str, entry: float, atr_pct: float,
|
||||
notional: float, leverage: int, stop_px: str,
|
||||
legs: list[dict], age_s: float) -> None:
|
||||
arrow = "多" if side == "LONG" else "空"
|
||||
lines = [f"[{TAG}] 开仓 {sym} {arrow}",
|
||||
f"参考价 {_fmt(entry)} · ATR {atr_pct * 1e4:.1f}bp",
|
||||
f"名义 {notional:.0f} USDT · {leverage}x · "
|
||||
f"保证金 {notional / leverage:.0f} USDT",
|
||||
f"止损 {stop_px}(两腿共用,不移动)"]
|
||||
for lg in legs:
|
||||
lines.append(f" {lg['tag']:<6} {lg['size']} 币 · 止盈 {lg['tp_px']}")
|
||||
lines.append(f"距参考价成立 {age_s:.1f}s")
|
||||
await send("\n".join(lines))
|
||||
|
||||
|
||||
async def closed(sym: str, side: str, pnl: float, open_px: float,
|
||||
close_px: float, day_pnl: float, day_n: int,
|
||||
reason: str = "") -> None:
|
||||
arrow = "多" if side == "LONG" else "空"
|
||||
mark = "盈" if pnl > 0 else ("亏" if pnl < 0 else "平")
|
||||
tail = f" · {reason}" if reason else ""
|
||||
await send(f"[{TAG}] 平仓 {sym} {arrow} {mark} {pnl:+.2f} USDT{tail}\n"
|
||||
f"开 {_fmt(open_px)} → 平 {_fmt(close_px)}\n"
|
||||
f"当日 {day_n} 笔 · 累计 {day_pnl:+.2f} USDT")
|
||||
|
||||
|
||||
async def blocked(key: str, why: str) -> None:
|
||||
"""只在被硬约束挡住时推。过期跳过属于常态,不走这里。"""
|
||||
await send(f"[{TAG}] ⛔ 信号被挡 {key}\n{why}")
|
||||
|
||||
|
||||
async def error(what: str, detail: str) -> None:
|
||||
await send(f"[{TAG}] ⛔ {what}\n{detail[:500]}")
|
||||
|
||||
|
||||
async def started(notional: float, leverage: int, syms: int,
|
||||
dry: bool) -> None:
|
||||
mode = "空跑(不下真单)" if dry else "真跑"
|
||||
await send(f"[{TAG}] 执行器启动 · {mode}\n"
|
||||
f"名义 {notional:.0f} USDT · {leverage}x · {syms} 币")
|
||||
|
||||
|
||||
async def day_rolled(day: str, n: int, pnl: float) -> None:
|
||||
await send(f"[{TAG}] {day} 结算 · {n} 笔 · {pnl:+.2f} USDT")
|
||||
|
||||
|
||||
def _dur(s: float) -> str:
|
||||
s = max(0, int(s))
|
||||
if s < 60:
|
||||
return f"{s}秒"
|
||||
if s < 3600:
|
||||
return f"{s // 60}分钟"
|
||||
if s < 86400:
|
||||
h, m = s // 3600, (s % 3600) // 60
|
||||
return f"{h}小时{m}分" if m else f"{h}小时"
|
||||
return f"{s / 86400:.1f}天"
|
||||
|
||||
|
||||
async def alive(up_s: float, seen: int, built: int, took: int, n_open: int,
|
||||
max_open: int, day_n: int, max_day: int, day_pnl: float,
|
||||
max_loss: float, order_fail: int, ship: dict | None,
|
||||
dry: bool) -> None:
|
||||
"""整点在线。异常在第一行,正常时才是「在线」。
|
||||
|
||||
`ship` 是搬运器落的 ship_alive.json 解出来的字典,None 表示读不到——
|
||||
那本身就是要报的事:搬运没在跑、或者跑的是没有这个文件的旧版本。
|
||||
"""
|
||||
warn = []
|
||||
if order_fail and not built:
|
||||
warn.append(f"⛔ 做了 {took} 笔却一次都没建上,下单全被拒")
|
||||
elif order_fail:
|
||||
warn.append(f"⚠ 累计 {order_fail} 次下单失败")
|
||||
if ship is None:
|
||||
warn.append("⛔ 读不到搬运状态,信号可能根本没在进来")
|
||||
else:
|
||||
gap = time.time() - ship.get("ts", 0)
|
||||
# 搬运连上/断开/每 5 分钟都会落一次。超过 12 分钟是进程自己死了
|
||||
if gap > 720:
|
||||
warn.append(f"⛔ 搬运状态已停更 {_dur(gap)},上游可能已断")
|
||||
elif ship.get("connected") is False:
|
||||
warn.append("⛔ 搬运 ssh 已断开,正在重连")
|
||||
if ship.get("n_skew"):
|
||||
warn.append(f"⛔ 搬运侧时钟倒流 {ship['n_skew']} 次")
|
||||
|
||||
head = warn[0] if warn else ("在线(空跑)" if dry else "在线")
|
||||
lines = [f"[{TAG}] {head} · 已跑 {_dur(up_s)}",
|
||||
f"信号 {seen} · 建仓 {built} · 在场 {n_open}/{max_open}",
|
||||
f"当日 {day_n}/{max_day} 笔 · 盈亏 {day_pnl:+.2f}/-{max_loss:.1f}"]
|
||||
if ship is not None:
|
||||
last = ship.get("last_signal_ts") or 0
|
||||
lines.append(
|
||||
f"搬运 ssh 在线 {_dur(ship.get('up_s', 0))} · "
|
||||
f"重连 {ship.get('n_reconnect', 0)} 次 · 最近信号 "
|
||||
+ (f"{_dur(time.time() - last)}前" if last else "启动后还没有"))
|
||||
lines += warn[1:]
|
||||
await send("\n".join(lines))
|
||||
|
||||
|
||||
async def stopping(n_open: int) -> None:
|
||||
await send(f"[{TAG}] 收到停机信号,平掉在场 {n_open} 笔后退出。"
|
||||
f"\n注意:停机后不再有超时平仓与新开仓。"
|
||||
f"发生时间 {time.strftime('%H:%M:%S')}")
|
||||
@@ -0,0 +1,32 @@
|
||||
# Tests and research/ — everything beyond the core runtime.
|
||||
#
|
||||
# .venv/bin/pip install -r requirements.txt -r requirements-dev.txt
|
||||
# .venv/bin/python -m pytest chanlun/tests web/tests
|
||||
|
||||
-r requirements.txt
|
||||
|
||||
# --- tests -----------------------------------------------------------------
|
||||
pytest>=8.0
|
||||
|
||||
# --- research/ -------------------------------------------------------------
|
||||
# pyarrow reads the data/binance/*.feather klines (research/lib/data.py) and
|
||||
# writes the parquet cache. Needed for research/, not by the web app.
|
||||
pyarrow>=15.0
|
||||
|
||||
# Only research/step10_direct_return_label.py and step11_breakout_follow.py.
|
||||
# Neither is installed by default; skip this pair unless running those steps.
|
||||
scikit-learn>=1.4
|
||||
lightgbm>=4.3
|
||||
|
||||
# --- indicator parity check (optional) -------------------------------------
|
||||
# chanlun/indicators/ta.py replaced TA-Lib and technical, so nothing here needs
|
||||
# them at runtime. chanlun/tests/test_ta_compat.py pins our output against
|
||||
# TA-Lib bar for bar and SKIPS SILENTLY when they are absent — meaning a change
|
||||
# to ta.py can look tested when it was not. Install these and re-run that file
|
||||
# whenever ta.py changes.
|
||||
#
|
||||
# TA-Lib needs the C library first:
|
||||
# sudo apt-get install -y libta-lib0 ta-lib-dev
|
||||
#
|
||||
# TA-Lib>=0.6
|
||||
# technical>=1.7
|
||||
@@ -0,0 +1,29 @@
|
||||
# Core runtime — what `web/` and `chanlun/` need to run.
|
||||
#
|
||||
# python -m venv .venv && .venv/bin/pip install -r requirements.txt
|
||||
#
|
||||
# Tests and research/ pull in more; see requirements-dev.txt.
|
||||
#
|
||||
# Requires Python >= 3.11 (imposed by pandas 3.x / numpy 2.x, not by our code).
|
||||
#
|
||||
# Lower bounds are the oldest versions believed safe. Verified set as of
|
||||
# 2026-08-27 on Python 3.14.4:
|
||||
# pandas 3.0.5 · numpy 2.5.2 · Flask 3.1.3 · requests 2.34.2
|
||||
# python-dateutil 2.9.0 · pytz 2026.3 · ccxt 4.5.75 · akshare 1.18.94
|
||||
|
||||
# chanlun/ — kline pipeline and indicators.
|
||||
# pandas >= 2.2 for the "5min" offset alias used by chanlun/pipeline/resample.py.
|
||||
pandas>=2.2
|
||||
numpy>=1.26
|
||||
|
||||
# web/ — Flask API and templates.
|
||||
Flask>=3.0
|
||||
requests>=2.31
|
||||
python-dateutil>=2.9
|
||||
pytz>=2024.1
|
||||
|
||||
# Market data. ccxt drives the live websocket/REST state in
|
||||
# web/services/runtime/state.py; akshare only backs the A-share endpoints in
|
||||
# web/services/cn_stock.py.
|
||||
ccxt>=4.4
|
||||
akshare>=1.16
|
||||
+3062
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,7 @@
|
||||
# 已 superseded
|
||||
|
||||
v0 改走免费交易所,由 **data_provider** 拉,chan 只从中转取。
|
||||
|
||||
见 [PROMOTE_deriv_relay.md](./PROMOTE_deriv_relay.md)。
|
||||
|
||||
CoinGlass 付费档等看完交易所效果再开,不要和本阶段混做。
|
||||
@@ -0,0 +1,94 @@
|
||||
# 需求:资金面数据(data_provider → chan)
|
||||
|
||||
**提出方:** chan
|
||||
**执行方:** data_provider
|
||||
**阶段:** Paper。先看效果。不进 Live。
|
||||
|
||||
---
|
||||
|
||||
## 要什么
|
||||
|
||||
chan 要在缠论图上叠资金面,和现有 K 线对得上。
|
||||
|
||||
data_provider 对外提供资金面;chan **只从 data_provider 取**,不访问交易所,不访问 CoinGlass。
|
||||
|
||||
K 线路径不动(现有 `/api/candles` 与 K 线推送)。本次只加资金面。
|
||||
|
||||
---
|
||||
|
||||
## 数据
|
||||
|
||||
先三个币:**BTC、ETH、SOL**(USDT 永续,符号与现有 K 线相同,如 `BTC/USDT:USDT`)。
|
||||
|
||||
要两样:
|
||||
|
||||
1. **持仓量(OI)**
|
||||
- 要历史,能覆盖缠论常用周期:`15m`、`30m`、`4h`、`1d`(有 `1h`/`2h` 更好)。
|
||||
- 要当前最新值。
|
||||
- 历史长度至少约 30 天。
|
||||
|
||||
2. **资金费率(funding)**
|
||||
- 要当前值。
|
||||
- 要历史结算序列。
|
||||
- 对齐到各周期 K 线:结算点落到所在那根;非结算 bar 沿用上一次结算值,不要插值编造。
|
||||
|
||||
来源:交易所公开数据即可,本阶段不买 CoinGlass。OI 历史哪家所没有,用另一家所公开数据补,需标明来源。
|
||||
|
||||
**本阶段不要:** 清算、热力图、多空比、订单簿、CoinGlass。
|
||||
|
||||
---
|
||||
|
||||
## 给 chan 的接口
|
||||
|
||||
与 `/api/candles` 同一套约定:
|
||||
|
||||
- `symbol` 与蜡烛相同
|
||||
- 时间戳毫秒 UTC
|
||||
- 按周期 `tf` 取序列
|
||||
- 支持 `start` / `end` / `limit`(默认 `limit=500`)
|
||||
|
||||
示例:
|
||||
|
||||
```http
|
||||
GET /api/deriv?symbol=BTC/USDT:USDT&tf=15m&metrics=oi,funding
|
||||
```
|
||||
|
||||
每根:
|
||||
|
||||
| 字段 | 要求 |
|
||||
|---|---|
|
||||
| `timestamp` | 与同 `tf` 的 `/api/candles` **开盘时间**对齐;对不齐的不要 |
|
||||
| `oi` | 该 bar 持仓量;缺则 `null` |
|
||||
| `oi_src` | 该值来自哪家所 |
|
||||
| `funding` | 该 bar 资金费率;缺则 `null` |
|
||||
| `funding_src` | 该值来自哪家所 |
|
||||
|
||||
健康状态要能看出:资金面是否可用、各所是否通、上次成功时间。
|
||||
|
||||
---
|
||||
|
||||
## 约束
|
||||
|
||||
- 全程 HTTPS REST。本阶段不要求资金面 WebSocket。
|
||||
- chan、浏览器不得直连交易所。
|
||||
- 现有 K 线接口行为不变。
|
||||
- 一家所挂了:缺那家字段,另一家仍要能出;两边都没有且无可用数据时明确失败。
|
||||
- 上游限流或超时:不要拖垮 K 线。
|
||||
|
||||
---
|
||||
|
||||
## 不算本次
|
||||
|
||||
- CoinGlass / 付费数据
|
||||
- 清算、热力、多空
|
||||
- Live、下单
|
||||
- chan 叠图(等本接口可用再做)
|
||||
|
||||
---
|
||||
|
||||
## 怎样算齐
|
||||
|
||||
1. `GET /api/deriv?symbol=BTC/USDT:USDT&tf=15m` 能拿到 `oi`、`funding`,时间能对上同参数的 `/api/candles`。
|
||||
2. chan / 浏览器零次访问交易所。
|
||||
3. 只挂一家所时,接口仍可用,只缺对应字段。
|
||||
4. K 线不受影响。
|
||||
@@ -0,0 +1,94 @@
|
||||
"""下载样本外验证用的新币种数据,存成与 freqtrade 一致的 feather。
|
||||
|
||||
现有结论全部建立在 BTC/ETH/SOL 三个币上,参数(级别对、tol、SL/TP)也是
|
||||
在这三个币上挑的,存在选择偏差。本脚本补齐一批完全没参与过调参的品种。
|
||||
|
||||
只拉 30m/2h:那是实测最强的一对,用它做样本外足够,且请求量只有全级别的四成。
|
||||
|
||||
限速要点(踩过的坑):
|
||||
klines(limit=1500) 权重 30,上限 2400/分钟 = 80 次/分钟 = 0.75s/次,
|
||||
0.8s 间隔正好卡在边缘,一旦触发 429,短退避跨不过计数窗口就会连续失败。
|
||||
故间隔放到 1.3s、权重阈值压到 1400、429 时等满一个窗口。
|
||||
另外分页失败不再丢弃整只币,已抓到的部分照样落盘。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import pandas as pd
|
||||
import requests
|
||||
|
||||
OUT = Path(__file__).resolve().parents[1] / "data" / "binance" / "futures"
|
||||
BASE = "https://fapi.binance.com/fapi/v1/klines"
|
||||
PROXY = {"http": "http://127.0.0.1:7897", "https": "http://127.0.0.1:7897"}
|
||||
SYMBOLS = ["BNB", "XRP", "DOGE", "ADA", "AVAX", "LINK", "LTC", "TRX"]
|
||||
TFS = ["2h", "30m"]
|
||||
START_MS = int(pd.Timestamp("2019-01-01", tz="UTC").timestamp() * 1000)
|
||||
COLS = ["date", "open", "high", "low", "close", "volume"]
|
||||
GAP = 1.3
|
||||
WEIGHT_CAP = 1400
|
||||
|
||||
|
||||
def fetch(sess: requests.Session, symbol: str, tf: str) -> pd.DataFrame | None:
|
||||
rows: list[list] = []
|
||||
cur = START_MS
|
||||
while True:
|
||||
batch = None
|
||||
for attempt in range(6):
|
||||
try:
|
||||
r = sess.get(BASE, params={"symbol": f"{symbol}USDT", "interval": tf,
|
||||
"startTime": cur, "limit": 1500}, timeout=40)
|
||||
if r.status_code in (418, 429):
|
||||
time.sleep(65)
|
||||
continue
|
||||
if r.status_code == 400:
|
||||
return None
|
||||
r.raise_for_status()
|
||||
batch = r.json()
|
||||
if int(r.headers.get("X-MBX-USED-WEIGHT-1M", 0) or 0) > WEIGHT_CAP:
|
||||
time.sleep(40)
|
||||
break
|
||||
except Exception:
|
||||
time.sleep(5 * (attempt + 1))
|
||||
if not batch:
|
||||
break # 抓不动或抓完了,保留已有部分
|
||||
rows.extend(batch)
|
||||
nxt = int(batch[-1][0]) + 1
|
||||
if nxt <= cur or len(batch) < 1500:
|
||||
break
|
||||
cur = nxt
|
||||
time.sleep(GAP)
|
||||
if len(rows) < 1000:
|
||||
return None
|
||||
df = pd.DataFrame(rows).iloc[:, :6]
|
||||
df.columns = ["ts", "open", "high", "low", "close", "volume"]
|
||||
df["date"] = pd.to_datetime(pd.to_numeric(df["ts"]), unit="ms", utc=True)
|
||||
for c in ("open", "high", "low", "close", "volume"):
|
||||
df[c] = pd.to_numeric(df[c])
|
||||
return df[COLS].drop_duplicates("date").sort_values("date").reset_index(drop=True)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
sess = requests.Session()
|
||||
sess.proxies.update(PROXY)
|
||||
jobs = [(s, tf) for tf in TFS for s in SYMBOLS]
|
||||
print(f"[下载] {len(jobs)} 个任务,单线程 {GAP}s 间隔", flush=True)
|
||||
for i, (sym, tf) in enumerate(jobs, 1):
|
||||
path = OUT / f"{sym}_USDT_USDT-{tf}-futures.feather"
|
||||
if path.exists():
|
||||
print(f" [{i}/{len(jobs)}] {sym} {tf} 已存在", flush=True)
|
||||
continue
|
||||
df = fetch(sess, sym, tf)
|
||||
if df is None:
|
||||
print(f" [{i}/{len(jobs)}] {sym} {tf} 失败", flush=True)
|
||||
continue
|
||||
df.to_feather(path)
|
||||
print(f" [{i}/{len(jobs)}] {sym} {tf} {len(df)} 根 "
|
||||
f"{df['date'].min():%Y-%m-%d}~{df['date'].max():%Y-%m-%d}", flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,239 @@
|
||||
"""出场模拟,但止盈按限价单的**真实成交量**结算,而非假定全额成交。
|
||||
|
||||
## 为什么要另写一份
|
||||
|
||||
`exit_model.walk_exits` 给止盈记的毛收益是 `target * a / entry`,即假定挂在
|
||||
目标价的限价单全额成交在目标价。影子交易的成交流数据显示这个假定在真实
|
||||
仓位上不成立:止盈位被首次触及那一根,限价落在该根价格区间中的位置中位
|
||||
k≈0.28,而该位置之上可供成交的主动买量,对 32 万仓位只够覆盖 30%/16%/1.5%
|
||||
(BTC/ETH/SOL)。
|
||||
|
||||
不成交不等于仓位消失——它继续持有,结果从「继续走下去」的分布里抽,其中
|
||||
包含反转打到止损。所以这不是给预算打折能修的事,是出场规则变了。
|
||||
|
||||
## 模型
|
||||
|
||||
两张挂单常驻:半仓在 `scale_at`、半仓在 `runner`。每根按该根在限价之上的
|
||||
可成交量逐步吃进,未成交部分继续持有;整仓止损始终有效,触发时未成交的
|
||||
部分市价平掉;到 `maxb` 根仍未了结的按收盘市价平。
|
||||
|
||||
某根在限价 P 之上的可成交量:
|
||||
|
||||
P ≤ low 整根成交量都在限价之上 avail = V
|
||||
P > high 该根没到限价 avail = 0
|
||||
否则 k = (high−P)/(high−low) avail = f(k) × V
|
||||
|
||||
`V` 是该根的**主动买**成交额(多头出场靠主动买盘打上来)。实测买卖大致
|
||||
均衡,取总成交额的一半;以 BTC 校验,历史 `volume×close` 中位与影子成交流
|
||||
实测差 0.3%。`f` 由成交流定,实测几乎是线性(f(k)≈k,即区间内均匀分布),
|
||||
所以结论对形状假设不敏感。
|
||||
|
||||
## 仍然乐观的两处
|
||||
|
||||
1. **未计排队**。我们的单排在该价位既有挂单之后,真实成交更少。
|
||||
2. **未计自身的流动性效应**。大单挂在 3ATR 会吸收本该冲到 8ATR 的买盘,
|
||||
即两张挂单在真实市场里互相竞争,此处按独立处理。
|
||||
|
||||
两处都指向同一方向:真实成交率比本模型更低。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from lib.exit_model import FEE_MAKER, FEE_TAKER, SLIP
|
||||
|
||||
# 成交流实测的区间内成交分布形状(BTC/ETH/SOL 均值,见 shadow_depth.tape_shape)
|
||||
SHAPE_K = np.linspace(0.0, 1.0, 21)
|
||||
# 多头止盈挂卖出,靠**主动买**打上来 —— 自区间顶部向下累积
|
||||
SHAPE_F = np.array([0.044, 0.088, 0.110, 0.179, 0.204, 0.240, 0.282, 0.316,
|
||||
0.390, 0.430, 0.465, 0.537, 0.583, 0.617, 0.662, 0.714,
|
||||
0.761, 0.804, 0.857, 0.904, 1.000])
|
||||
# 空头止盈挂买回,靠**主动卖**打下来 —— 自区间底部向上累积。
|
||||
# 两侧并不对称:主动买集中在区间顶部(k=0.2 处已 40%),主动卖在底部只有
|
||||
# 18%,BTC/ETH 平均偏差 0.21。原先两侧共用 SHAPE_F,等于把空头的可成交量
|
||||
# 按多头的分布高估。
|
||||
# ⚠ 这条曲线只有 45 根样本(每币 14~16),够说明「不对称」这个方向,不够
|
||||
# 定具体数值——一周数据到手要重新导出。用它而非 SHAPE_F 的理由是方向正确
|
||||
# 优于数值精确:结论对形状不敏感(见 step43_fill_aware_budget 的敏感性检查),
|
||||
# 但用错方向是系统性偏乐观。
|
||||
SHAPE_F_SHORT = np.array([0.031, 0.076, 0.121, 0.193, 0.230, 0.265, 0.297,
|
||||
0.356, 0.382, 0.420, 0.453, 0.485, 0.523, 0.552,
|
||||
0.586, 0.629, 0.681, 0.754, 0.813, 0.871, 1.000])
|
||||
TAKER_SHARE = 0.5
|
||||
|
||||
|
||||
def avail_at(price: float, hi: float, lo: float, vol_notional: float,
|
||||
is_long: bool, kgrid=SHAPE_K, f=None,
|
||||
f_short=None) -> float:
|
||||
"""该根里能打到限价 `price` 的对手方成交额。
|
||||
|
||||
多头在 `price` 挂卖出,靠价格 ≥ price 的主动买成交;空头挂买回,靠
|
||||
价格 ≤ price 的主动卖成交。方向用显式参数而非「把价格取负」——取负会
|
||||
让所有价格变成负数,任何对价格正负的假设都会静默失效。
|
||||
|
||||
两个方向查各自的成交分布曲线(实测二者不对称,见 SHAPE_F_SHORT)。
|
||||
"""
|
||||
f = SHAPE_F if f is None else f
|
||||
f_short = SHAPE_F_SHORT if f_short is None else f_short
|
||||
if vol_notional <= 0 or not (np.isfinite(hi) and np.isfinite(lo)):
|
||||
return 0.0
|
||||
if hi <= lo:
|
||||
# 该根无波动:只要限价被覆盖就算整根可成交
|
||||
return vol_notional if (hi >= price if is_long else lo <= price) \
|
||||
else 0.0
|
||||
if is_long:
|
||||
if price > hi:
|
||||
return 0.0 # 该根没涨到限价
|
||||
if price <= lo:
|
||||
return vol_notional # 整根都在限价之上
|
||||
k = (hi - price) / (hi - lo)
|
||||
else:
|
||||
if price < lo:
|
||||
return 0.0 # 该根没跌到限价
|
||||
if price >= hi:
|
||||
return vol_notional # 整根都在限价之下
|
||||
k = (price - lo) / (hi - lo)
|
||||
return float(np.interp(k, kgrid, f if is_long else f_short)) * vol_notional
|
||||
|
||||
|
||||
def walk_filled(cdf: pd.DataFrame, sig: pd.DataFrame, notional: float,
|
||||
sl: float = 2.0, scale_at: float = 3.0, runner: float = 8.0,
|
||||
runner_stop: float = 2.0, maxb: int = 48,
|
||||
taker_share: float = TAKER_SHARE,
|
||||
f=None, f_short=None) -> pd.DataFrame:
|
||||
"""前推每笔信号,返回按成交量结算的出场权重与毛收益。
|
||||
|
||||
每行的 `w_*` 是各出场去向占**全仓名义额**的比例,四者相加为 1。
|
||||
|
||||
`f` / `f_short` 是两个方向的区间内成交分布曲线,留出接口是为了能做敏感性
|
||||
检查——这两条曲线目前只有 45 根样本,必须能验证结论对它们不敏感。
|
||||
"""
|
||||
high = cdf["high"].to_numpy(float)
|
||||
low = cdf["low"].to_numpy(float)
|
||||
close = cdf["close"].to_numpy(float)
|
||||
open_ = cdf["open"].to_numpy(float)
|
||||
vol = cdf["volume"].to_numpy(float)
|
||||
atr = cdf["atr"].to_numpy(float)
|
||||
n = len(cdf)
|
||||
out = []
|
||||
|
||||
for s, d in zip(sig["entry_idx"].astype(int), sig["direction"].astype(int)):
|
||||
e = s + 1
|
||||
if e >= n - 1:
|
||||
continue
|
||||
a = atr[s]
|
||||
if not np.isfinite(a) or a <= 0:
|
||||
continue
|
||||
entry = open_[e]
|
||||
cap = min(e + maxb, n - 1)
|
||||
|
||||
p_stop = entry - d * sl * a
|
||||
p_scale = entry + d * scale_at * a
|
||||
p_run = entry + d * runner * a
|
||||
|
||||
# 两张挂单各半仓,单位是「占全仓的比例」
|
||||
rem_scale, rem_run = 0.5, 0.5
|
||||
w_scale = w_run = w_stop = w_time = 0.0
|
||||
scaled_any = False
|
||||
stop_bar = None
|
||||
|
||||
for j in range(e, cap + 1):
|
||||
hi, lo = high[j], low[j]
|
||||
# 同根内止损优先,与 walk_exits 一致,宁可低估
|
||||
hit_stop = (lo <= p_stop) if d == 1 else (hi >= p_stop)
|
||||
if hit_stop:
|
||||
stop_bar = j
|
||||
w_stop = rem_scale + rem_run
|
||||
rem_scale = rem_run = 0.0
|
||||
break
|
||||
|
||||
v = vol[j] * close[j] * taker_share
|
||||
is_long = d == 1
|
||||
if rem_scale > 0:
|
||||
got = avail_at(p_scale, hi, lo, v, is_long, SHAPE_K, f, f_short)
|
||||
fill = min(rem_scale, got / notional) if notional > 0 else \
|
||||
rem_scale
|
||||
if fill > 0:
|
||||
rem_scale -= fill
|
||||
w_scale += fill
|
||||
scaled_any = True
|
||||
if rem_run > 0:
|
||||
got = avail_at(p_run, hi, lo, v, is_long, SHAPE_K, f, f_short)
|
||||
fill = min(rem_run, got / notional) if notional > 0 else \
|
||||
rem_run
|
||||
if fill > 0:
|
||||
rem_run -= fill
|
||||
w_run += fill
|
||||
if rem_scale <= 1e-12 and rem_run <= 1e-12:
|
||||
break
|
||||
|
||||
# 减仓成交后,剩余半仓的止损位可以另设;此处 runner_stop 等于初始 SL
|
||||
# 即止损不动,与 step42 的 k=2.0 一致,故上面那个统一止损已覆盖
|
||||
left = rem_scale + rem_run
|
||||
if left > 1e-12 and stop_bar is None:
|
||||
w_time = left
|
||||
r_scale = d * (p_scale - entry) / entry
|
||||
r_run = d * (p_run - entry) / entry
|
||||
r_stop = d * (p_stop - entry) / entry
|
||||
r_time = d * (close[cap] - entry) / entry
|
||||
|
||||
gross = (w_scale * r_scale + w_run * r_run
|
||||
+ w_stop * r_stop + w_time * r_time)
|
||||
# 入场整仓 taker;两张挂单成交的部分是 maker;止损与超时是 taker
|
||||
fee = (FEE_TAKER * 1.0 + FEE_MAKER * (w_scale + w_run)
|
||||
+ FEE_TAKER * (w_stop + w_time))
|
||||
tk = 1.0 + w_stop + w_time
|
||||
out.append({"sig_idx": s, "direction": d, "atr_pct": a / entry,
|
||||
"w_scale": w_scale, "w_run": w_run,
|
||||
"w_stop": w_stop, "w_time": w_time,
|
||||
"maker_share": w_scale + w_run,
|
||||
"gross": gross, "fee": fee, "taker_notional": tk,
|
||||
"scaled": int(scaled_any)})
|
||||
return pd.DataFrame(out)
|
||||
|
||||
|
||||
def budget_bp(r: pd.DataFrame) -> float:
|
||||
"""盈亏平衡的单边滑点上限(bp)。与 exit_model.slip_budget 同口径。"""
|
||||
if r.empty:
|
||||
return float("nan")
|
||||
net = r["gross"].mean() - r["fee"].mean()
|
||||
return net / r["taker_notional"].mean() * 1e4
|
||||
|
||||
|
||||
def assert_converges(cdf: pd.DataFrame, sig: pd.DataFrame, sl: float = 2.0,
|
||||
scale_at: float = 3.0, runner: float = 8.0,
|
||||
runner_stop: float = 2.0, maxb: int = 48,
|
||||
tol: float = 1e-9) -> None:
|
||||
"""仓位趋近 0 时必须逐笔收敛到 exit_model.walk_exits,否则抛错。
|
||||
|
||||
这个断言是必需的。首版实现里空头的可成交量恒为 0(负价格空间踩到了一个
|
||||
`hi <= 0` 的守卫),后果是空头全被拖到超时收盘——而下跌段里那比 3ATR
|
||||
目标赚得多,于是预算反而**偏高** 0.76bp,看上去像个合理的模型差异。
|
||||
没有这条断言,这种错只会表现为「数字有点不一样」。
|
||||
"""
|
||||
from lib.exit_model import cfg_name, walk_exits
|
||||
|
||||
old = walk_exits(cdf, sig, [sl], [scale_at], [maxb], scale_at,
|
||||
[runner], [runner_stop])
|
||||
c = cfg_name(sl, runner, maxb, runner_stop)
|
||||
new = walk_filled(cdf, sig, 1e-12, sl, scale_at, runner, runner_stop, maxb)
|
||||
j = old[["sig_idx"]].copy()
|
||||
j["g_old"] = old[f"{c}_g"].to_numpy()
|
||||
j = j.merge(new[["sig_idx", "gross"]], on="sig_idx")
|
||||
d = (j["gross"] - j["g_old"]).abs()
|
||||
bad = int((d > tol).sum())
|
||||
if bad:
|
||||
worst = j.loc[d.idxmax()]
|
||||
raise AssertionError(
|
||||
f"仓位趋近 0 时应与 walk_exits 一致,但 {bad}/{len(j)} 笔不符;"
|
||||
f"最大差 {d.max():.3e}(sig_idx {int(worst['sig_idx'])}:"
|
||||
f"老 {worst['g_old']:.6f} 新 {worst['gross']:.6f})")
|
||||
|
||||
|
||||
def net_bp(r: pd.DataFrame, slip: float = SLIP) -> float:
|
||||
"""扣掉手续费与滑点后的净均收益(bp)。"""
|
||||
if r.empty:
|
||||
return float("nan")
|
||||
net = r["gross"] - r["fee"] - slip * r["taker_notional"]
|
||||
return float(net.mean() * 1e4)
|
||||
@@ -0,0 +1,236 @@
|
||||
"""出场模拟与费率模型,step41(5m/15m/30m)与 step42(1m)共用。
|
||||
|
||||
单独抽出来是因为两边必须用同一份实现——`fast_bsp3` 当初分散在两处的教训。
|
||||
|
||||
## 费率模型(用户 2026-08-27 指出)
|
||||
|
||||
原先所有数字都按「双边 taker」算(§5.3 的 6bp),这在两个方向上都错了:
|
||||
费率档位记高了(见下方 FEE_TAKER 注释),且没有区分出场性质——
|
||||
|
||||
入场 信号在收盘出现,次根开盘市价单进场 → **taker + 滑点**
|
||||
止盈 挂在目标价的限价单被动成交 → **maker,无滑点**
|
||||
止损 stop-market,触发后市价成交 → **taker + 滑点**
|
||||
超时 到点市价平 → **taker + 滑点**
|
||||
|
||||
止损做不成 maker:stop-limit 在急跌里可能不成交,那种情况下的损失远大于
|
||||
省下的 2bp。所以按出场原因分别计费,不是一刀切。
|
||||
|
||||
分批离场不额外增加费用:手续费按名义额收,入场 1.0、出场 0.5+0.5,总额不变。
|
||||
但**第一批必然是止盈成交(maker)**,这正是分批在成本上占便宜的地方。
|
||||
|
||||
## 出场配置
|
||||
|
||||
整仓 `s{SL}_tp{TP}_m{MAXB}`
|
||||
分批 `s{SL}_so{目标}_k{剩余半仓止损}_m{MAXB}`
|
||||
到 3 ATR 平一半,剩余半仓止损挪到「开仓价下方 k 个 ATR」。
|
||||
k=0 即保本损,k 等于原 SL 即止损不动,k 更大则是主动放宽。
|
||||
|
||||
## 实现
|
||||
|
||||
每笔只前推两遍:第一遍记录各价位的首次触及根,第二遍从减仓根起记录剩余
|
||||
半仓各止损位的首次触及根。之后所有配置都是解析推导,不再重复走 K 线。
|
||||
同一根内止损与目标并存时一律判止损先到,宁可低估。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
# 单边费率。用户 2026-08-27 给的实际档位:返佣前 taker 0.040% / maker 0.016%,
|
||||
# API 返 50% → taker 0.020% / maker 0.008%。
|
||||
# 注意:此前这里写的是 0.030/0.010,隐含「原始 taker 6bp」的错误前提,
|
||||
# 所以 step41/42 首轮跑出来的所有数字都偏保守(taker 高估 50%)。
|
||||
FEE_TAKER = 0.00020
|
||||
FEE_MAKER = 0.00008
|
||||
SLIP = 0.00010
|
||||
|
||||
TP, SL_, TIME = 0, 1, 2 # 出场原因编码
|
||||
|
||||
|
||||
def taker_notional(reason: np.ndarray, scaled: np.ndarray) -> np.ndarray:
|
||||
"""每笔走 taker 的名义额(入场 1.0,加上非止盈出场的部分)。滑点只发生在这上面。"""
|
||||
exit_taker = np.where(reason == TP, 0.0, 1.0)
|
||||
exit_taker = np.where(scaled == 1, 0.5 * exit_taker, exit_taker)
|
||||
return 1.0 + exit_taker
|
||||
|
||||
|
||||
def fee_of(reason: np.ndarray, scaled: np.ndarray,
|
||||
fee_taker: float = FEE_TAKER, fee_maker: float = FEE_MAKER) -> np.ndarray:
|
||||
"""只算手续费,不含滑点。分批时第一半必然是止盈成交(maker)。"""
|
||||
exit_fee = np.where(reason == TP, fee_maker, fee_taker)
|
||||
exit_fee = np.where(scaled == 1, 0.5 * fee_maker + 0.5 * exit_fee, exit_fee)
|
||||
return fee_taker + exit_fee
|
||||
|
||||
|
||||
def cost_of(reason: np.ndarray, scaled: np.ndarray, slip: float = SLIP) -> np.ndarray:
|
||||
"""新口径:入场 taker、止盈 maker、止损/超时 taker,滑点只加在 taker 腿上。"""
|
||||
return fee_of(reason, scaled) + slip * taker_notional(reason, scaled)
|
||||
|
||||
|
||||
def all_taker_cost(reason: np.ndarray, scaled: np.ndarray) -> np.ndarray:
|
||||
"""旧口径:进出都当 taker,双边费 + 双边滑点。用来对照新旧差多少。"""
|
||||
return np.full(len(reason), 2.0 * (FEE_TAKER + SLIP))
|
||||
|
||||
|
||||
def flat_cost(reason: np.ndarray, scaled: np.ndarray) -> np.ndarray:
|
||||
"""研究口径的固定 5bp,用于和 step28/35/37 的历史数字对齐。"""
|
||||
return np.full(len(reason), 0.0005)
|
||||
|
||||
|
||||
def slip_budget(gross: np.ndarray, reason: np.ndarray, scaled: np.ndarray) -> float:
|
||||
"""盈亏平衡的单边滑点上限(bp):毛收益扣掉手续费后,摊到走 taker 的名义额上。"""
|
||||
net_of_fee = gross.mean() - fee_of(reason, scaled).mean()
|
||||
return net_of_fee / taker_notional(reason, scaled).mean() * 10000
|
||||
|
||||
|
||||
def cfg_name(sl: float, target, maxb: int, rstop=None) -> str:
|
||||
"""整仓 s{SL}_tp{T}_m{B};分批 s{SL}_so{T}_k{K}_m{B}。target=None 表示不设目标。"""
|
||||
t = f"{target:g}" if target is not None else "R"
|
||||
if rstop is None:
|
||||
return f"s{sl:g}_tp{t}_m{maxb}"
|
||||
return f"s{sl:g}_so{t}_k{rstop:g}_m{maxb}"
|
||||
|
||||
|
||||
def walk_exits(cdf: pd.DataFrame, sig: pd.DataFrame, sls, tps, maxbs,
|
||||
scale_at: float = 3.0, runners=(5.0, 6.0, 8.0, None),
|
||||
runner_stops=(0.0, 0.5, 1.0, 1.5, 2.0)) -> pd.DataFrame:
|
||||
"""前推每笔信号,解析出全部出场配置的结果。
|
||||
|
||||
每个配置四列:`{cfg}_g` 毛收益率、`{cfg}_r` 出场原因、`{cfg}_c` 是否分批、
|
||||
`{cfg}_b` 持仓根数。另有 `s{SL}_mfe` 各初始止损下的最大有利偏移。
|
||||
"""
|
||||
high = cdf["high"].to_numpy(float)
|
||||
low = cdf["low"].to_numpy(float)
|
||||
open_ = cdf["open"].to_numpy(float)
|
||||
close = cdf["close"].to_numpy(float)
|
||||
atr = cdf["atr"].to_numpy(float)
|
||||
n = len(cdf)
|
||||
horizon = max(maxbs)
|
||||
ups = sorted(set(list(tps) + [scale_at] + [r for r in runners if r]))
|
||||
downs = sorted(set(list(sls) + list(runner_stops)))
|
||||
out = []
|
||||
|
||||
for s, d in zip(sig["entry_idx"].astype(int), sig["direction"].astype(int)):
|
||||
e = s + 1
|
||||
if e >= n - 1:
|
||||
continue
|
||||
a = atr[s]
|
||||
if not np.isfinite(a) or a <= 0:
|
||||
continue
|
||||
entry = open_[e]
|
||||
end = min(e + horizon, n - 1)
|
||||
|
||||
# 第一遍:各价位的首次触及根(与止损无关,纯价格事件)
|
||||
up_bar = {t: None for t in ups}
|
||||
dn_bar = {L: None for L in downs}
|
||||
mfe = {sl: 0.0 for sl in sls}
|
||||
alive = {sl: True for sl in sls}
|
||||
for j in range(e, end + 1):
|
||||
adv = (high[j] - entry) / a if d == 1 else (entry - low[j]) / a
|
||||
ret = (entry - low[j]) / a if d == 1 else (high[j] - entry) / a
|
||||
for L in downs:
|
||||
if dn_bar[L] is None and ret >= L:
|
||||
dn_bar[L] = j
|
||||
for sl in sls:
|
||||
if alive[sl]:
|
||||
if dn_bar[sl] is not None and dn_bar[sl] == j:
|
||||
alive[sl] = False # 同根内止损优先,不更新 MFE
|
||||
elif adv > mfe[sl]:
|
||||
mfe[sl] = adv
|
||||
for t in ups:
|
||||
if up_bar[t] is None and adv >= t:
|
||||
up_bar[t] = j
|
||||
for sl in sls:
|
||||
mfe[sl] = mfe[sl]
|
||||
|
||||
# 第二遍:从减仓根起,剩余半仓各止损位的首次触及根
|
||||
j0 = up_bar[scale_at]
|
||||
dn_after = {L: None for L in runner_stops}
|
||||
if j0 is not None:
|
||||
for j in range(j0, end + 1):
|
||||
ret = (entry - low[j]) / a if d == 1 else (high[j] - entry) / a
|
||||
for L in runner_stops:
|
||||
if dn_after[L] is None and ret >= L:
|
||||
dn_after[L] = j
|
||||
if all(v is not None for v in dn_after.values()):
|
||||
break
|
||||
|
||||
row = {"sig_idx": s, "direction": d, "atr_pct": a / entry}
|
||||
row.update({f"s{sl:g}_mfe": mfe[sl] for sl in sls})
|
||||
|
||||
def resolve(target, stop_bar, stop_ret, cap, start):
|
||||
"""在 cap 根之前,目标与止损谁先到。返回 (毛收益率, 原因, 出场根)。"""
|
||||
tb = up_bar[target] if target is not None else None
|
||||
if tb is not None and tb <= cap and (stop_bar is None or tb < stop_bar):
|
||||
return target * a / entry, TP, tb
|
||||
if stop_bar is not None and stop_bar <= cap:
|
||||
return stop_ret, SL_, stop_bar
|
||||
k = min(cap, n - 1)
|
||||
return d * (close[k] - entry) / entry, TIME, k
|
||||
|
||||
for sl in sls:
|
||||
dead = dn_bar[sl]
|
||||
for b in maxbs:
|
||||
cap = e + b
|
||||
for t in tps:
|
||||
g, r, xb = resolve(t, dead, -sl * a / entry, cap, e)
|
||||
c = cfg_name(sl, t, b)
|
||||
row[f"{c}_g"], row[f"{c}_r"], row[f"{c}_c"], row[f"{c}_b"] = g, r, 0, xb - e + 1
|
||||
# 分批:先看能否走到减仓点
|
||||
scaled_ok = (j0 is not None and j0 <= cap
|
||||
and (dead is None or j0 < dead))
|
||||
for rn in runners:
|
||||
for k in runner_stops:
|
||||
c = cfg_name(sl, rn, b, k)
|
||||
if not scaled_ok:
|
||||
g, r, xb = resolve(scale_at, dead, -sl * a / entry, cap, e)
|
||||
row[f"{c}_g"], row[f"{c}_r"] = g, r
|
||||
row[f"{c}_c"], row[f"{c}_b"] = 0, xb - e + 1
|
||||
else:
|
||||
rg, rr, rxb = resolve(rn, dn_after[k], -k * a / entry, cap, j0)
|
||||
row[f"{c}_g"] = 0.5 * (scale_at * a / entry) + 0.5 * rg
|
||||
row[f"{c}_r"], row[f"{c}_c"] = rr, 1
|
||||
row[f"{c}_b"] = rxb - e + 1
|
||||
out.append(row)
|
||||
return pd.DataFrame(out)
|
||||
|
||||
|
||||
def stat(g: pd.DataFrame, cfg: str, label: str, cost_fn=cost_of, minn: int = 25) -> dict:
|
||||
"""给一个配置出统计。`滑点余量bp` 是滑点的盈亏平衡上限。"""
|
||||
gross = g[f"{cfg}_g"].to_numpy()
|
||||
if len(gross) < minn:
|
||||
return {}
|
||||
reason, scaled = g[f"{cfg}_r"].to_numpy(), g[f"{cfg}_c"].to_numpy()
|
||||
r = gross - cost_fn(reason, scaled)
|
||||
w, o = r[r > 0], r[r <= 0]
|
||||
sd = r.std(ddof=1)
|
||||
t10 = r[r <= np.quantile(r, 0.90)]
|
||||
return {"口径": label, "笔数": len(r),
|
||||
"胜率": f"{(r > 0).mean() * 100:.1f}%",
|
||||
"毛bp": f"{gross.mean() * 10000:.2f}",
|
||||
"净均收益": f"{r.mean() * 100:+.3f}%",
|
||||
"中位": f"{np.median(r) * 100:+.3f}%",
|
||||
"PF": f"{w.sum() / abs(o.sum()):.2f}" if len(o) else "inf",
|
||||
"t值": f"{r.mean() / (sd / np.sqrt(len(r))):+.2f}",
|
||||
"剔10%PF": f"{t10[t10 > 0].sum() / abs(t10[t10 <= 0].sum()):.2f}" if (t10 <= 0).any() else "inf",
|
||||
"滑点余量bp": f"{slip_budget(gross, reason, scaled):.2f}",
|
||||
"止盈占比": f"{(reason == TP).mean() * 100:.0f}%",
|
||||
"均持仓": f"{g[f'{cfg}_b'].mean():.1f}"}
|
||||
|
||||
|
||||
def rstat(g: pd.DataFrame, cfg: str, label: str, sl: float, cost_fn=cost_of) -> dict:
|
||||
"""R 倍数口径:固定风险仓位下,不同初始止损之间唯一可比的量。
|
||||
|
||||
SL 越宽,每笔风险越大、仓位越小,直接比百分比收益会把仓位差异算成策略优势。
|
||||
"""
|
||||
gross = g[f"{cfg}_g"].to_numpy()
|
||||
reason, scaled = g[f"{cfg}_r"].to_numpy(), g[f"{cfg}_c"].to_numpy()
|
||||
net = gross - cost_fn(reason, scaled)
|
||||
risk = sl * g["atr_pct"].to_numpy()
|
||||
r = net / risk
|
||||
return {"口径": label, "笔数": len(r), "风险%": f"{risk.mean() * 100:.2f}%",
|
||||
"净均收益": f"{net.mean() * 100:+.3f}%",
|
||||
"均R": f"{r.mean():+.3f}", "中位R": f"{np.median(r):+.3f}",
|
||||
"R夏普": f"{r.mean() / r.std(ddof=1):.3f}",
|
||||
"最差1%R": f"{np.percentile(r, 1):.2f}",
|
||||
"胜率": f"{(r > 0).mean() * 100:.1f}%"}
|
||||
@@ -0,0 +1,225 @@
|
||||
"""快速一类买卖点(fast B1/S1)——把引擎的 B1/S1 改写成当根可判的形式。
|
||||
|
||||
**动机**:step55 实测引擎 `find_all_bsp` 的 B1/S1 在 5m/15m 上是重亏的
|
||||
(PF 0.20~0.24、胜率 15~21%、t −17~−22),而且原因是几何性的:引擎在
|
||||
`leave_bi.sure_time` 才发信号,中位滞后 8~9 根,此时价格已朝反弹方向跑了
|
||||
1.68 ATR。逆势信号上,这个滞后是加倍惩罚——新入场价往下 2 ATR 的止损落在
|
||||
比原始低点还低 0.3 ATR 处,几乎贴着极值。
|
||||
|
||||
B3 当年也是同样的病(PF 0.66),解法不是调参而是**重新定义成实时判据**
|
||||
(B4,见 `chanlun/analysis/fast_bsp.py`)。本模块对一类做同样的事。
|
||||
|
||||
引擎 B1 是三个条件的合取(`bsp.py: find_all_bsp` + `check_bi_div`):
|
||||
|
||||
中枢已成 -> 一笔向下离开中枢 -> 该笔 MACD 面积 < 进入笔面积(底背驰)
|
||||
|
||||
三条都有当根可判的代理,滞后来源只有一处——`leave_bi.is_sure`:
|
||||
|
||||
中枢已成 zs.is_sure -> zones.available_ts(与 B4 同口径,已解决)
|
||||
向下离开 leave_bi 端点在 zd 下 -> **收盘** < zd,当根可判
|
||||
背驰 整笔面积对比 -> 进入笔面积在中枢确认时已是历史,
|
||||
离开段面积从突破根起逐根累加,当根可判
|
||||
触发 等笔确认(滞后之源) -> 收盘反向越过前一根极值(沿用 B4 的转强判据)
|
||||
|
||||
**与三类的方向关系相反**:三类顺着突破方向做,一类逆着做。中枢向下突破后,
|
||||
B4 会做空,而 fast B1 是在同一段下跌里找力竭然后**做多**。
|
||||
|
||||
⚠️ 先验不利,落地前请先看数:§3.391 实测 `mom60` 最高的四分位是最差的
|
||||
(余量只剩 1.44bp),而一类按定义就住在一段已走完的行情末端。本模块是用来
|
||||
证伪这个先验的,不是用来假定它不成立的。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from chanlun.analysis.fast_bsp import _resolve_avail_bi, timestamps_ms
|
||||
|
||||
|
||||
def zones_with_enter_area(zs_list, src: pd.DataFrame,
|
||||
avail_bi: int | None = None) -> pd.DataFrame:
|
||||
"""区间表 + 进入笔的 MACD 面积与方向。
|
||||
|
||||
这两列**不引入滞后**:进入笔是中枢第一笔的前一笔,中枢确认时它早已收尾。
|
||||
背驰判据里唯一需要等待的是离开段,而那一段可以逐根累加。
|
||||
|
||||
面积口径必须和 `ChanBI.cal_macd_hist` 一致:只累加顺方向那一侧的 hist
|
||||
并取绝对值(上升笔累加正值,下降笔累加负值的绝对值),故恒非负。
|
||||
"""
|
||||
ts_of = dict(zip(src["date"].dt.strftime("%Y-%m-%d %H:%M:%S"),
|
||||
timestamps_ms(src)))
|
||||
i = _resolve_avail_bi(avail_bi)
|
||||
|
||||
rows = []
|
||||
for zs in zs_list:
|
||||
bis = getattr(zs, "bi_list", [])
|
||||
if not bis:
|
||||
continue
|
||||
key_bi = bis[i] if (i == -1 or len(bis) > i) else bis[-1]
|
||||
avail = (ts_of.get(str(getattr(key_bi, "sure_time", "") or ""))
|
||||
or ts_of.get(str(getattr(key_bi, "end_time", "") or "")))
|
||||
if avail is None:
|
||||
continue
|
||||
enter_bi = getattr(bis[0], "pre", None)
|
||||
if enter_bi is None:
|
||||
continue
|
||||
# dir 用 +1/−1 表示,避免把引擎枚举漏进研究侧
|
||||
e_dir = 1 if str(enter_bi.dir).endswith("UP") else -1
|
||||
rows.append({
|
||||
"zg": float(zs.zg), "zd": float(zs.zd),
|
||||
"start_ts": ts_of.get(str(bis[0].start_time), avail),
|
||||
"available_ts": int(avail),
|
||||
"enter_area": abs(float(enter_bi.macd_hist)),
|
||||
"enter_dir": e_dir,
|
||||
})
|
||||
out = pd.DataFrame(rows)
|
||||
if out.empty:
|
||||
return out
|
||||
return out.sort_values("available_ts").reset_index(drop=True)
|
||||
|
||||
|
||||
def find_fast_bsp1(
|
||||
df: pd.DataFrame,
|
||||
zones: pd.DataFrame,
|
||||
scan: int = 200,
|
||||
leave_win: int = 60,
|
||||
min_leave_bars: int = 3,
|
||||
max_div: float = 1.0,
|
||||
max_per_zone: int = 1,
|
||||
diag: dict | None = None,
|
||||
) -> pd.DataFrame:
|
||||
"""扫描每个中枢,找「离开中枢后力度背驰、随即转向」的入场点。
|
||||
|
||||
zones 需含 zg / zd / available_ts / enter_area / enter_dir,
|
||||
即 `zones_with_enter_area` 的输出。
|
||||
|
||||
`min_leave_bars` 要求离开段至少走这么多根才允许触发。没有它的话,转强判据
|
||||
会在突破后第一根小反弹就打进去,那时离开段的面积还没累起来,背驰条件几乎
|
||||
自动成立——会退化成「随便一个反弹就抄底」。
|
||||
|
||||
`max_div` 是背驰的松紧:离开段面积 / 进入段面积 < max_div 才算背驰。
|
||||
1.0 等于引擎的 `macdhist_div < 0`;调小则只认力度衰减更明显的。
|
||||
|
||||
返回列:
|
||||
entry_idx 实时可下单的K线
|
||||
direction +1 一买 / −1 一卖(**与突破方向相反**)
|
||||
bo_idx 离开中枢的突破根
|
||||
ext_idx 离开段的极值根
|
||||
lag entry_idx − ext_idx,即相对理想入场点的滞后
|
||||
div 离开段面积 / 进入段面积,越小背驰越强
|
||||
depth 离开段极值越过中枢边界的幅度(相对边界)
|
||||
"""
|
||||
need = {"zg", "zd", "available_ts", "enter_area", "enter_dir"}
|
||||
if zones.empty or not need <= set(zones.columns):
|
||||
return pd.DataFrame()
|
||||
|
||||
ts = df["timestamp"].to_numpy()
|
||||
close = df["close"].to_numpy(float)
|
||||
high = df["high"].to_numpy(float)
|
||||
low = df["low"].to_numpy(float)
|
||||
hist = df["macdhist"].to_numpy(float)
|
||||
atr = df["atr"].to_numpy(float)
|
||||
n = len(df)
|
||||
rows = []
|
||||
|
||||
def note(key: str) -> None:
|
||||
if diag is not None:
|
||||
diag[key] = diag.get(key, 0) + 1
|
||||
|
||||
for zone_i, (_, z) in enumerate(zones.iterrows()):
|
||||
note("中枢总数")
|
||||
zg, zd = float(z["zg"]), float(z["zd"])
|
||||
e_area, e_dir = float(z["enter_area"]), int(z["enter_dir"])
|
||||
if zg <= zd or not np.isfinite(e_area) or e_area <= 0:
|
||||
note("×无效中枢或进入段无面积")
|
||||
continue
|
||||
start = int(np.searchsorted(ts, z["available_ts"], side="left"))
|
||||
if start >= n - 2:
|
||||
note("×中枢太靠后")
|
||||
continue
|
||||
scan_end = min(start + scan * max_per_zone, n)
|
||||
cursor = start
|
||||
|
||||
for occ in range(1, max_per_zone + 1):
|
||||
if cursor >= n - 2:
|
||||
break
|
||||
was_inside = False
|
||||
bo_idx, d = None, 0
|
||||
for j in range(cursor, scan_end):
|
||||
c = close[j]
|
||||
if zd <= c <= zg:
|
||||
was_inside = True
|
||||
continue
|
||||
if not was_inside:
|
||||
continue
|
||||
bo_idx, d = j, (1 if c > zg else -1)
|
||||
break
|
||||
if bo_idx is None:
|
||||
if occ == 1:
|
||||
note("×窗口内未离开中枢")
|
||||
break
|
||||
|
||||
# 引擎要求 enter_bi.dir == leave_bi.dir 才比面积,否则背驰无从谈起
|
||||
if e_dir != d:
|
||||
note("×进入段与离开段不同向")
|
||||
cursor = bo_idx + 1
|
||||
continue
|
||||
|
||||
area = 0.0
|
||||
ext = low[bo_idx] if d == -1 else high[bo_idx]
|
||||
ext_idx = bo_idx
|
||||
entry_idx, div_at = None, np.nan
|
||||
for j in range(bo_idx, min(bo_idx + leave_win + 1, n)):
|
||||
h = hist[j]
|
||||
# 与 ChanBI.cal_macd_hist 同口径:只取顺方向一侧,取绝对值
|
||||
if d == -1 and h < 0:
|
||||
area -= h
|
||||
elif d == 1 and h > 0:
|
||||
area += h
|
||||
if (low[j] < ext) if d == -1 else (high[j] > ext):
|
||||
ext, ext_idx = (low[j] if d == -1 else high[j]), j
|
||||
if j < bo_idx + min_leave_bars:
|
||||
continue
|
||||
# 收盘回到中枢内 -> 这不是一次有效的离开,弃掉
|
||||
if zd <= close[j] <= zg:
|
||||
note("×离开段收盘回到中枢内")
|
||||
break
|
||||
if area >= e_area * max_div:
|
||||
continue # 力度未衰减,不是背驰
|
||||
go = close[j] > high[j - 1] if d == -1 else close[j] < low[j - 1]
|
||||
if go:
|
||||
entry_idx, div_at = j, area / e_area
|
||||
break
|
||||
if entry_idx is None:
|
||||
if occ == 1:
|
||||
note("×未在窗口内背驰转向")
|
||||
cursor = bo_idx + 1
|
||||
continue
|
||||
if occ == 1:
|
||||
note("√成交")
|
||||
|
||||
edge = zd if d == -1 else zg
|
||||
# 延伸度用 ATR 归一,才和 step62 的 ext_run 同口径 —— 那里实测它是
|
||||
# 唯一单调区分「趋势末端 vs 趋势中途」的特征(命中率 12%→44%)。
|
||||
# 用相对边界的百分比会混入价格水平,不同币之间不可比。
|
||||
a_ext = atr[ext_idx]
|
||||
rows.append({
|
||||
"entry_idx": entry_idx, "direction": -d,
|
||||
"bo_idx": bo_idx, "ext_idx": ext_idx,
|
||||
"lag": entry_idx - ext_idx,
|
||||
"div": div_at,
|
||||
"ext_atr": (abs(ext - edge) / a_ext
|
||||
if np.isfinite(a_ext) and a_ext > 0 else np.nan),
|
||||
"depth": abs(ext - edge) / edge,
|
||||
"zg": zg, "zd": zd,
|
||||
"width_pct": (zg - zd) / close[bo_idx],
|
||||
"occ": occ, "zone_i": zone_i,
|
||||
})
|
||||
cursor = entry_idx + 1
|
||||
|
||||
out = pd.DataFrame(rows)
|
||||
if out.empty:
|
||||
return out
|
||||
return (out.sort_values(["entry_idx", "occ", "zone_i"])
|
||||
.drop_duplicates("entry_idx", keep="first")
|
||||
.reset_index(drop=True))
|
||||
+8
-134
@@ -1,141 +1,15 @@
|
||||
"""快速三类买卖点:不等笔确认,突破回抽当根即入场。
|
||||
"""快速三类买卖点 —— 实现已移入引擎 `chanlun.analysis.fast_bsp`。
|
||||
|
||||
引擎的 B3/S3 要等 pullback_bi.sure_time(回拉笔被确认),滞后 9~10 根,
|
||||
此时价格已从回抽低点反弹完毕,入场价被吃掉。
|
||||
|
||||
但三买的形态条件本身是实时可判的:
|
||||
中枢已成 -> 收盘突破 zg -> 回抽最低不跌回中枢(low >= zg) -> 重新上行
|
||||
最后一步发生的当根就能下单,滞后 1~2 根。
|
||||
|
||||
全部判定只使用当根及之前的数据,无未来函数。
|
||||
这里只做转发,保证 step 脚本里的 `from lib.fast_bsp3 import find_fast_bsp3` 不用改,
|
||||
同时让回测与 web 图表共用同一份代码。设计说明见引擎模块的 docstring。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
|
||||
def find_fast_bsp3(
|
||||
df: pd.DataFrame,
|
||||
zones: pd.DataFrame,
|
||||
scan: int = 200,
|
||||
pullback_win: int = 30,
|
||||
tol: float = 0.003,
|
||||
max_per_zone: int = 1,
|
||||
diag: dict | None = None,
|
||||
) -> pd.DataFrame:
|
||||
"""扫描每个中枢,找突破后回抽不回中枢的入场点。
|
||||
from chanlun.analysis.fast_bsp import find_fast_bsp3 # noqa: F401,E402
|
||||
|
||||
zones 需含 zg / zd / available_ts,且 available_ts 已是可用时刻。
|
||||
max_per_zone > 1 时,同一中枢在首次入场后继续往后找二次、三次突破回抽,
|
||||
用来检验「趋势里同一中枢反复给机会」是否值得做。
|
||||
|
||||
返回列:
|
||||
entry_idx 实时可下单的K线
|
||||
direction +1 三买 / -1 三卖
|
||||
bo_idx 突破根
|
||||
pb_idx 回抽极值根
|
||||
lag entry_idx - bo_idx
|
||||
depth 回抽深度(相对中枢边界,负值表示曾插入中枢)
|
||||
occ 这是该中枢的第几次入场
|
||||
"""
|
||||
if zones.empty:
|
||||
return pd.DataFrame()
|
||||
|
||||
ts = df["timestamp"].to_numpy()
|
||||
close = df["close"].to_numpy(dtype=float)
|
||||
high = df["high"].to_numpy(dtype=float)
|
||||
low = df["low"].to_numpy(dtype=float)
|
||||
n = len(df)
|
||||
rows = []
|
||||
|
||||
def note(key: str) -> None:
|
||||
if diag is not None:
|
||||
diag[key] = diag.get(key, 0) + 1
|
||||
|
||||
for zone_i, (_, z) in enumerate(zones.iterrows()):
|
||||
note("中枢总数")
|
||||
zg, zd = float(z["zg"]), float(z["zd"])
|
||||
if zg <= zd:
|
||||
note("×无效中枢")
|
||||
continue
|
||||
start = int(np.searchsorted(ts, z["available_ts"], side="left"))
|
||||
if start >= n - 2:
|
||||
note("×中枢太靠后")
|
||||
continue
|
||||
# 允许多次入场时按比例放宽扫描窗口,否则后几次机会会被窗口截断
|
||||
scan_end = min(start + scan * max_per_zone, n)
|
||||
cursor = start
|
||||
|
||||
for occ in range(1, max_per_zone + 1):
|
||||
if cursor >= n - 2:
|
||||
break
|
||||
# 第一步:找突破。要求突破前确实待在中枢内,避免把远处的价格当突破。
|
||||
was_inside = False
|
||||
bo_idx, d = None, 0
|
||||
for j in range(cursor, scan_end):
|
||||
c = close[j]
|
||||
if zd <= c <= zg:
|
||||
was_inside = True
|
||||
continue
|
||||
if not was_inside:
|
||||
continue
|
||||
bo_idx, d = j, (1 if c > zg else -1)
|
||||
break
|
||||
if bo_idx is None:
|
||||
if occ == 1:
|
||||
note("×窗口内未突破")
|
||||
break
|
||||
|
||||
edge = zg if d == 1 else zd
|
||||
# 第二步:突破后监控回抽,回抽不跌回中枢且重新顺势 -> 入场
|
||||
touched = False
|
||||
pb_idx = None
|
||||
pb_ext = None
|
||||
entry_idx = None
|
||||
fell_back = False
|
||||
for j in range(bo_idx + 1, min(bo_idx + pullback_win + 1, n)):
|
||||
# 收盘跌回中枢 -> 突破失效
|
||||
if zd <= close[j] <= zg:
|
||||
fell_back = True
|
||||
break
|
||||
# 回抽触及边界附近(允许 tol 的毛刺)
|
||||
near = (low[j] <= edge * (1 + tol)) if d == 1 else (high[j] >= edge * (1 - tol))
|
||||
if near:
|
||||
touched = True
|
||||
ext = low[j] if d == 1 else high[j]
|
||||
if pb_ext is None or ((ext < pb_ext) if d == 1 else (ext > pb_ext)):
|
||||
pb_ext, pb_idx = ext, j
|
||||
continue
|
||||
# 回抽后重新顺势:收盘创出前一根之上(三买)/ 之下(三卖)
|
||||
if touched and pb_idx is not None:
|
||||
go = close[j] > high[j - 1] if d == 1 else close[j] < low[j - 1]
|
||||
if go:
|
||||
entry_idx = j
|
||||
break
|
||||
if entry_idx is None or pb_ext is None:
|
||||
if occ == 1:
|
||||
note("×突破后跌回中枢" if fell_back
|
||||
else "×回抽未触及边界" if not touched
|
||||
else "×触及边界但未转强")
|
||||
# 这次突破没走成,从突破点之后继续找下一次
|
||||
cursor = bo_idx + 1
|
||||
continue
|
||||
if occ == 1:
|
||||
note("√成交")
|
||||
|
||||
# 回抽深度:>0 表示未插入中枢,越大表示回抽越浅
|
||||
depth = (pb_ext - zg) / zg if d == 1 else (zd - pb_ext) / zd
|
||||
rows.append({
|
||||
"entry_idx": entry_idx, "direction": d,
|
||||
"bo_idx": bo_idx, "pb_idx": pb_idx,
|
||||
"lag": entry_idx - bo_idx,
|
||||
"depth": depth,
|
||||
"zg": zg, "zd": zd,
|
||||
"width_pct": (zg - zd) / close[bo_idx],
|
||||
"occ": occ,
|
||||
"zone_i": zone_i,
|
||||
})
|
||||
cursor = entry_idx + 1
|
||||
|
||||
return pd.DataFrame(rows)
|
||||
__all__ = ["find_fast_bsp3"]
|
||||
|
||||
@@ -17,43 +17,8 @@ import pandas as pd
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
|
||||
from chanlun import TF_DF
|
||||
|
||||
|
||||
def build_htf_zones(df_htf: pd.DataFrame, tf: str, chan: TF_DF | None = None) -> pd.DataFrame:
|
||||
"""算 pure 笔中枢,返回带生效时间的区间表。
|
||||
|
||||
available_ts —— 该中枢最早可被使用的时间戳(其确认时刻)。
|
||||
传入已构建好的 chan 可避免重复跑一遍 pipeline(大数据集上省一半时间)。
|
||||
"""
|
||||
if chan is None:
|
||||
chan = TF_DF(df_htf, 1, tf)
|
||||
zs_list = chan.cal_bi_zs_list_pure(chan.bi_list)
|
||||
# 用引擎自己的 dataframe 对齐,避免调用方传入的 df 与引擎内部行数不一致
|
||||
src = chan.dataframe if getattr(chan, "dataframe", None) is not None else df_htf
|
||||
ts_of = dict(zip(src["date"].dt.strftime("%Y-%m-%d %H:%M:%S"), src["timestamp"]))
|
||||
|
||||
rows = []
|
||||
for zs in zs_list:
|
||||
bis = getattr(zs, "bi_list", [])
|
||||
if not bis:
|
||||
continue
|
||||
# 中枢可用时刻:构成它的最后一笔被确认之时
|
||||
last_bi = bis[-1]
|
||||
sure_key = str(getattr(last_bi, "sure_time", "") or "")
|
||||
end_key = str(getattr(last_bi, "end_time", "") or "")
|
||||
avail = ts_of.get(sure_key) or ts_of.get(end_key)
|
||||
if avail is None:
|
||||
continue
|
||||
start_key = str(bis[0].start_time)
|
||||
rows.append({
|
||||
"zg": float(zs.zg), "zd": float(zs.zd),
|
||||
"gg": float(getattr(zs, "gg", zs.zg)), "dd": float(getattr(zs, "dd", zs.zd)),
|
||||
"start_ts": ts_of.get(start_key, avail),
|
||||
"available_ts": int(avail),
|
||||
})
|
||||
out = pd.DataFrame(rows)
|
||||
return out.sort_values("available_ts").reset_index(drop=True) if not out.empty else out
|
||||
# build_htf_zones 已移入引擎,与 web 共用同一份实现;annotate_position 仍是研究专用
|
||||
from chanlun.analysis.fast_bsp import build_htf_zones # noqa: F401
|
||||
|
||||
|
||||
def annotate_position(
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
"""影子测量要对照的那条线:逐币滑点预算、ATR 门控、lag 阈值。
|
||||
|
||||
记录与报表由 `research/live/shadow_hb.py` / `shadow_report.py` 负责,
|
||||
**这里只放从回测推出来的常数和判据**。分工的理由是这些数会变——
|
||||
2026-08-27 一天之内预算就动了四次(3.91 → 11.06 → 14.25 → 15.19bp),
|
||||
费率也从 3/1bp 改成 2/0.8bp。写死在 live 侧的任何一份都会静默过期。
|
||||
|
||||
## 预算的定义
|
||||
|
||||
余量 = (毛均收益 − 实际手续费) / taker 名义额
|
||||
|
||||
分母是 **taker 名义额**(入场 1.0 + 非止盈出场的部分),所以这个数的含义是
|
||||
**每条 taker 腿能承受多少 bp 滑点**。止盈挂限价是 maker、不吃滑点,
|
||||
不能混进分母——否则会被那 60% 零滑点的止盈腿稀释,低估真实成本。
|
||||
|
||||
对照时用入场滑点即可(入场必然是 taker),无需等出场腿。
|
||||
|
||||
## 口径必须一致,否则数字不可比
|
||||
|
||||
预算算在这套口径上,live 侧的信号路径要对齐,缺一项数就不作数:
|
||||
|
||||
滤网 同向(h1_agree)**+ 中枢阶梯**。只有同向会显著偏低,见 §1.4
|
||||
门控 ATR ≥ ATR_GATE_BP
|
||||
出场 SL 2.0 / 3 ATR 减半 / 剩余半仓止损保持 2.0 / 目标 8 ATR / 48 根
|
||||
费率 taker 2.0bp、maker 0.8bp(返佣后)
|
||||
|
||||
> 曾经的 `毛均收益 − 6.0bp` 是错的:6bp 假设双边 taker 且原始费率 0.12%,
|
||||
> 而实际是原始 taker 0.040%、返 50% 后 2.0bp,且止盈是 maker。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
|
||||
from .exit_model import FEE_TAKER, FEE_MAKER
|
||||
|
||||
# ATR 门控。阈值是**费率的函数**,不是市场常数——低 ATR 的信号毛质量其实
|
||||
# 更好(毛R 1.16 vs 高 ATR 桶的 0.94),断崖只在扣费之后出现。见 §5.3。
|
||||
ATR_GATE_BP = 8.0
|
||||
|
||||
# lag 探针。补丁后实测 506~642ms,理论下限约 500ms。
|
||||
# 超阈值要**停止开新仓**,不能只打日志——退化是静默的,不崩也不报错。
|
||||
LAG_ALARM_MS = 800.0
|
||||
LAG_WINDOW = 30
|
||||
|
||||
# 逐币滑点预算(bp / taker 腿)。取 2026 年——那是七年里 ATR 最低的一年,
|
||||
# 也是当前所处的环境,**不要用全样本平均值**,低波动是现状不是尾部情形。
|
||||
BUDGET_BP = {
|
||||
"BTC": 8.58, # 全期 14.61。ATR 中位 2026 仅 6.5bp,11 个币里最低,
|
||||
# 门控要刷掉 58.5% 的信号 —— **不建议作为 1m 交易标的**,
|
||||
# 但流动性最好,适合做滑点测量的乐观边界
|
||||
"BNB": 15.48, # 全期 22.11。2026 门控只保留 25.2%
|
||||
"ETH": 20.64, # 全期 20.80
|
||||
"SOL": 16.83, # 全期 28.29
|
||||
"LINK": 9.18, # 全期 26.24。2026 波动压缩得厉害
|
||||
"LTC": 9.63, # 全期 23.63
|
||||
"AVAX": 15.02, # 全期 29.00
|
||||
"XRP": 17.04, # 全期 24.52
|
||||
"DOGE": 18.55, # 全期 28.48
|
||||
"ADA": 22.28, # 全期 23.53。2026 唯一没被波动压缩的
|
||||
"TRX": np.nan, # 2026 门控后只剩 4.2% 的信号,当前环境基本不能做
|
||||
}
|
||||
|
||||
# 组合口径(8 个样本外币合计)。单币样本薄时用这个判整体。
|
||||
BUDGET_PORTFOLIO_2026 = 15.19
|
||||
BUDGET_PORTFOLIO_RECENT = 19.17 # 2025+2026
|
||||
BUDGET_PORTFOLIO_FULL = 25.39 # 全样本外
|
||||
|
||||
# 腿 → 是否 taker。这不是日志字段,**这就是成本模型本身**:
|
||||
# 止盈挂限价是 maker 不吃滑点,止损是 stop-market、超时是市价,都吃。
|
||||
# 止损做不成 maker——stop-limit 在急跌里可能不成交,损失远大于省下的费。
|
||||
LEG_IS_TAKER = {
|
||||
"entry": True,
|
||||
"scale": False, # 3 ATR 减半,限价
|
||||
"runner_tp": False, # 剩余半仓到 8 ATR,限价
|
||||
"stop": True, # stop-market
|
||||
"timeout": True, # 48 根到点市价平
|
||||
}
|
||||
|
||||
|
||||
def gate_threshold_bp(fee_taker: float = FEE_TAKER) -> float:
|
||||
"""换费率档位时的粗略指引,**不是** ATR_GATE_BP 的定义式。
|
||||
|
||||
拟合自 taker 0/1.5/3/6bp 上实测的最优阈值 5/8/9/12bp。当前费率下
|
||||
公式给 7.2 而实测峰值是 8——不必调和:均R 在 5~12bp 之间只在
|
||||
0.972~0.987 之间动,**曲线是平的,精确值不重要**。
|
||||
要记住的是阈值随费率上移,换 VIP 档 / 换交易所要重扫,别抄 8bp。
|
||||
"""
|
||||
return 5.0 + 1.1 * fee_taker * 1e4
|
||||
|
||||
|
||||
def fee_rate(leg: str) -> float:
|
||||
return FEE_TAKER if LEG_IS_TAKER[leg] else FEE_MAKER
|
||||
|
||||
|
||||
def budget_of(symbol: str) -> float:
|
||||
"""取该币的预算。未收录或当前环境不可做的返回 nan。"""
|
||||
return BUDGET_BP.get(symbol.upper(), np.nan)
|
||||
|
||||
|
||||
def lag_healthy(recent_lag_ms, alarm_ms: float = LAG_ALARM_MS,
|
||||
window: int = LAG_WINDOW) -> bool:
|
||||
"""滚动中位数是否还在阈值内。False 时应告警并**停止开新仓**。
|
||||
|
||||
用中位数而非均值:单根的网络抖动不该触发停机,持续退化才该。
|
||||
"""
|
||||
x = np.asarray(recent_lag_ms, dtype=float)[-window:]
|
||||
if len(x) < window:
|
||||
return True
|
||||
return bool(np.median(x) <= alarm_ms)
|
||||
|
||||
|
||||
def verdict(measured_slip_bp: float, symbol: str | None = None,
|
||||
budget_bp: float | None = None) -> str:
|
||||
"""把实测滑点翻成 §6 第 4 步的四档判断。
|
||||
|
||||
`measured_slip_bp` 用**每条 taker 腿**的中位滑点,不是每笔的总滑点。
|
||||
"""
|
||||
if budget_bp is None:
|
||||
budget_bp = budget_of(symbol) if symbol else BUDGET_PORTFOLIO_2026
|
||||
if not np.isfinite(budget_bp):
|
||||
return "该币当前环境预算不足,不作交易标的"
|
||||
if measured_slip_bp <= 5:
|
||||
return "可行,进真实 dry run"
|
||||
if measured_slip_bp <= 10:
|
||||
return "可行但仅中高波动期,门控阈值要上调"
|
||||
if measured_slip_bp <= budget_bp:
|
||||
return "仅 2021 型牛市能做,常态不要开"
|
||||
return "放弃该腿"
|
||||
@@ -0,0 +1,186 @@
|
||||
"""最关键的一步:逐笔清单不可复现,总体期望还在不在。
|
||||
|
||||
signal_sensitivity 证明 0.25bp 的数据扰动就能换掉一半信号。这本身不判死刑——
|
||||
趋势跟随策略允许「成交的具体是哪几笔」随机,只要总体期望稳定就仍可交易。
|
||||
但如果扰动后 PF 与毛均收益也跟着塌,那回测测的就是噪声。
|
||||
|
||||
两种结果对应完全不同的下一步:
|
||||
总体稳定 -> 改成「Bitget 原生信号测滑点 + Bitget 原生回测基线」,主线继续
|
||||
总体也塌 -> 滑点根本不是瓶颈,1m 腿的问题在信号定义本身,影子交易器白写
|
||||
|
||||
口径与 step23 一致:SL/TP/MAX_BARS = 1.5/3.0/48,ATR 取信号根,
|
||||
入场为信号次根开盘价(entry_delay=1),成本 4bp 手续费 + 1bp 滑点。
|
||||
3.91bp 的滑点预算就是从「毛均收益 +0.0991%」推出来的,所以毛均收益是主看指标。
|
||||
|
||||
输出 out/aggregate_robustness.csv。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
|
||||
os.environ.setdefault(v, "1")
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
RESEARCH = HERE.parent
|
||||
sys.path.insert(0, str(RESEARCH))
|
||||
sys.path.insert(0, str(RESEARCH.parent))
|
||||
sys.path.insert(0, str(HERE))
|
||||
pd.set_option("display.width", 260)
|
||||
|
||||
from signal_sensitivity import TICK, perturb # noqa: E402
|
||||
|
||||
SYMS = ("BTC", "ETH", "SOL")
|
||||
LEVELS = (0.0, 0.5, 1.0, 2.0)
|
||||
SL, TP, MAX_BARS = 1.5, 3.0, 48
|
||||
FEE, SLIP = 0.0004, 0.0001
|
||||
|
||||
|
||||
def run_once(df_l: pd.DataFrame, df_h: pd.DataFrame) -> pd.DataFrame:
|
||||
"""跑完整管线并逐笔模拟,返回交易表。"""
|
||||
from chanlun import TF_DF
|
||||
from lib.breakout import run_trades
|
||||
from lib.fast_bsp3 import find_fast_bsp3
|
||||
from lib.fx_signal import extract_fx_signals, signals_to_frame
|
||||
from lib.nested_bsp import attach_htf_context, htf_fx_timeline
|
||||
from lib.nested_level import build_htf_zones
|
||||
|
||||
chan_l = TF_DF(df_l, 1, "1m")
|
||||
cdf = chan_l.dataframe
|
||||
zones = build_htf_zones(cdf, "1m", chan=chan_l)
|
||||
if zones.empty:
|
||||
return pd.DataFrame()
|
||||
sig = find_fast_bsp3(cdf, zones.reset_index(drop=True))
|
||||
if sig.empty:
|
||||
return pd.DataFrame()
|
||||
|
||||
chan_h = TF_DF(df_h, 1, "5m")
|
||||
hdf = chan_h.dataframe
|
||||
tl = htf_fx_timeline(signals_to_frame(extract_fx_signals(chan_h, hdf)), hdf)
|
||||
full = attach_htf_context(sig, cdf, tl, "h1")
|
||||
fin = full[full["h1_agree"] == 1]
|
||||
if fin.empty:
|
||||
return pd.DataFrame()
|
||||
|
||||
entries = list(zip(fin["entry_idx"].astype(int), fin["direction"].astype(int)))
|
||||
return run_trades(cdf, entries, SL, TP, MAX_BARS,
|
||||
fee=FEE + SLIP, entry_delay=1)
|
||||
|
||||
|
||||
def stats(tr: pd.DataFrame) -> dict:
|
||||
if tr.empty:
|
||||
return {"笔数": 0}
|
||||
g = tr["gross"].to_numpy(dtype=float)
|
||||
n = tr["ret"].to_numpy(dtype=float)
|
||||
win, loss = n[n > 0].sum(), -n[n < 0].sum()
|
||||
return {
|
||||
"笔数": len(tr),
|
||||
"胜率": f"{(n > 0).mean() * 100:.1f}%",
|
||||
"毛均收益": f"{g.mean() * 100:+.4f}%",
|
||||
"净均收益": f"{n.mean() * 100:+.4f}%",
|
||||
"PF": round(win / loss, 2) if loss > 0 else np.inf,
|
||||
"t值": round(n.mean() / n.std(ddof=1) * np.sqrt(len(n)), 2) if len(n) > 1 else np.nan,
|
||||
"滑点余量bp": round(g.mean() * 1e4 - 6.0, 2),
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--symbols", default="BTC,ETH,SOL")
|
||||
ap.add_argument("--bars", type=int, default=200_000, help="每币用多少根 1m")
|
||||
ap.add_argument("--seeds", type=int, default=2)
|
||||
ap.add_argument("--levels", default=None,
|
||||
help="逗号分隔的噪声档(bp);只给 0 就是纯基线复现")
|
||||
ap.add_argument("--tag", default="", help="产物文件名后缀")
|
||||
args = ap.parse_args()
|
||||
|
||||
levels = (tuple(float(x) for x in args.levels.split(","))
|
||||
if args.levels else LEVELS)
|
||||
|
||||
from lib.data import load_local
|
||||
|
||||
syms = [s.strip() for s in args.symbols.split(",")]
|
||||
print(f"[总体稳健性] {syms} · 每币 {args.bars} 根 1m "
|
||||
f"({args.bars / 1440:.0f} 天) · 噪声档 {levels} bp\n", flush=True)
|
||||
|
||||
rows = []
|
||||
for sym in syms:
|
||||
df_l = load_local(f"{sym}/USDT:USDT", "1m")
|
||||
df_h = load_local(f"{sym}/USDT:USDT", "5m")
|
||||
if df_l is None or df_h is None:
|
||||
print(f"{sym}: 本地无数据,跳过")
|
||||
continue
|
||||
df_l = df_l.tail(args.bars).reset_index(drop=True)
|
||||
lo = int(df_l["timestamp"].iloc[0])
|
||||
df_h = df_h[df_h.timestamp >= lo].reset_index(drop=True)
|
||||
print(f"── {sym} {len(df_l)} 根 1m / {len(df_h)} 根 5m "
|
||||
f"{df_l['date'].iloc[0]:%Y-%m-%d} ~ {df_l['date'].iloc[-1]:%Y-%m-%d}",
|
||||
flush=True)
|
||||
|
||||
for bp in levels:
|
||||
for k in range(1 if bp == 0 else args.seeds):
|
||||
t0 = time.perf_counter()
|
||||
tr = run_once(perturb(df_l, bp, TICK[sym], 2000 + k), df_h)
|
||||
s = stats(tr)
|
||||
rows.append({"品种": sym, "噪声bp": bp, "种子": k, **s})
|
||||
print(f" 噪声 {bp:>4.2f}bp 种子{k}: " +
|
||||
" · ".join(f"{k2} {v}" for k2, v in s.items()) +
|
||||
f" [{time.perf_counter() - t0:.0f}s]", flush=True)
|
||||
|
||||
if not rows:
|
||||
print("无结果")
|
||||
return
|
||||
|
||||
tb = pd.DataFrame(rows)
|
||||
print("\n" + "=" * 130)
|
||||
print("########## 1. 逐币 × 噪声档 ##########")
|
||||
print(tb.to_string(index=False))
|
||||
|
||||
print("\n########## 2. 三币合并(同噪声档取均值)##########")
|
||||
num = tb.copy()
|
||||
num["毛均bp"] = num["毛均收益"].str.rstrip("%").astype(float) * 100
|
||||
num["净均bp"] = num["净均收益"].str.rstrip("%").astype(float) * 100
|
||||
num["胜率_"] = num["胜率"].str.rstrip("%").astype(float)
|
||||
agg = num.groupby("噪声bp").agg(
|
||||
笔数=("笔数", "mean"), 胜率=("胜率_", "mean"),
|
||||
毛均bp=("毛均bp", "mean"), 净均bp=("净均bp", "mean"),
|
||||
PF=("PF", "mean"), t值=("t值", "mean")).round(2)
|
||||
agg["滑点余量bp"] = (agg["毛均bp"] - 6.0).round(2)
|
||||
print(agg.to_string())
|
||||
|
||||
print("\n########## 结论 ##########")
|
||||
print(" step23 的 1m 基线(3 币 3578 笔 / 2.28 年):毛均 9.91bp · PF 2.31 · "
|
||||
"t 19.92 · 余量 3.91bp")
|
||||
if 0.0 not in agg.index:
|
||||
print(" 本次未跑无噪声档,无法给出相对基线的比例")
|
||||
return
|
||||
base = agg.loc[0.0]
|
||||
print(f" 无噪声基线:毛均 {base['毛均bp']:.2f}bp · PF {base['PF']:.2f} · "
|
||||
f"t {base['t值']:.2f} · 滑点余量 {base['滑点余量bp']:.2f}bp")
|
||||
for bp in levels[1:]:
|
||||
if bp not in agg.index:
|
||||
continue
|
||||
r = agg.loc[bp]
|
||||
print(f" 噪声 {bp}bp:毛均 {r['毛均bp']:.2f}bp "
|
||||
f"({r['毛均bp'] / base['毛均bp'] * 100:.0f}% of 基线) · "
|
||||
f"PF {r['PF']:.2f} · t {r['t值']:.2f} · 余量 {r['滑点余量bp']:.2f}bp")
|
||||
print("\n 毛均与 PF 若基本持平 → 逐笔身份随机但总体期望稳定,主线继续,")
|
||||
print(" 但必须换成 Bitget 原生回测基线,Binance 的逐笔清单不可用于对照。")
|
||||
print(" 若毛均随噪声单调下滑 → 回测吃的是数据噪声,滑点不是瓶颈。")
|
||||
|
||||
out = RESEARCH / "out" / f"aggregate_robustness{args.tag}.csv"
|
||||
tb.to_csv(out, index=False)
|
||||
print(f"\n产物写入 {out}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,204 @@
|
||||
"""前置测量一:本机算力——影子交易器的计算延迟下限。
|
||||
|
||||
step39 在 Mac 上量到 2000 根窗口单次 0.20s。这台机器只有 2 vCPU 且单核更慢,
|
||||
而计算延迟直接吃 1m 腿仅 3.9bp 的滑点预算,所以必须在写影子交易器之前实测。
|
||||
|
||||
量四件事:
|
||||
1m 侧 TF_DF(2000根) + build_htf_zones + find_fast_bsp3
|
||||
5m 侧 TF_DF(800根) + 分型 + 时间线(只在 5m 收盘那根变,可摊薄到 1/5)
|
||||
串行 3 币 最后一个币要等多久 —— 这就是它的下单延迟
|
||||
并行 3 币 2 核跑 3 进程会互相抢核,未必比串行快
|
||||
|
||||
输出 out/bench_compute.csv,判据是与 3.9bp 预算对应的漂移:本机 1m 波动
|
||||
BTC 6.5bp/分钟,按 √t 折算,1 秒延迟约 0.8bp、5 秒约 1.9bp。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import warnings
|
||||
from concurrent.futures import ProcessPoolExecutor
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
|
||||
os.environ.setdefault(v, "1")
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
RESEARCH = HERE.parent
|
||||
sys.path.insert(0, str(RESEARCH))
|
||||
sys.path.insert(0, str(RESEARCH.parent))
|
||||
pd.set_option("display.width", 240)
|
||||
|
||||
LTF, HTF = "1m", "5m"
|
||||
WIN_LTF = 2000
|
||||
WIN_HTF = max(WIN_LTF // 5, 800)
|
||||
SYMS = ("BTC", "ETH", "SOL")
|
||||
|
||||
|
||||
def _signal_once(sl_ltf: pd.DataFrame) -> tuple[int, float]:
|
||||
"""1m 侧:切片重建结构与信号。与 step39 的 _pit_once 同一套调用。"""
|
||||
from chanlun import TF_DF
|
||||
from lib.fast_bsp3 import find_fast_bsp3
|
||||
from lib.nested_level import build_htf_zones
|
||||
|
||||
t0 = time.perf_counter()
|
||||
chan = TF_DF(sl_ltf, 1, LTF)
|
||||
cdf = chan.dataframe
|
||||
zones = build_htf_zones(cdf, LTF, chan=chan)
|
||||
n_sig = 0
|
||||
if not zones.empty:
|
||||
sig = find_fast_bsp3(cdf, zones.reset_index(drop=True))
|
||||
n_sig = len(sig)
|
||||
return n_sig, time.perf_counter() - t0
|
||||
|
||||
|
||||
def _agree_once(sl_htf: pd.DataFrame) -> tuple[int, float]:
|
||||
"""5m 侧:切片重建分型时间线。与 step39 的 _pit_agree 同一套调用。"""
|
||||
from chanlun import TF_DF
|
||||
from lib.fx_signal import extract_fx_signals, signals_to_frame
|
||||
from lib.nested_bsp import htf_fx_timeline
|
||||
|
||||
t0 = time.perf_counter()
|
||||
chan = TF_DF(sl_htf, 1, HTF)
|
||||
tl = htf_fx_timeline(signals_to_frame(extract_fx_signals(chan, chan.dataframe)),
|
||||
chan.dataframe)
|
||||
return len(tl), time.perf_counter() - t0
|
||||
|
||||
|
||||
def _load(sym: str) -> tuple[pd.DataFrame, pd.DataFrame]:
|
||||
from lib.data import load_local
|
||||
|
||||
pair = f"{sym}/USDT:USDT"
|
||||
df_l = load_local(pair, LTF)
|
||||
df_h = load_local(pair, HTF)
|
||||
if df_l is None or df_h is None:
|
||||
raise SystemExit(f"{sym} 本地数据缺失,本机只有 BTC/ETH/SOL")
|
||||
return df_l, df_h
|
||||
|
||||
|
||||
def _slices(sym: str, reps: int, seed: int) -> list[tuple[pd.DataFrame, pd.DataFrame]]:
|
||||
"""预先切好窗口。读 feather 是本机 IO,实盘数据来自 WS 缓冲,不计入延迟。"""
|
||||
df_l, df_h = _load(sym)
|
||||
rng = np.random.default_rng(seed)
|
||||
# 从末段随机取窗口,避开数据头部(指标预热)与尾部(不足一窗)
|
||||
lo, hi = max(WIN_LTF, len(df_l) - 200_000), len(df_l) - 1
|
||||
ltf_ts = df_l["timestamp"].to_numpy()
|
||||
htf_ts = df_h["timestamp"].to_numpy()
|
||||
|
||||
out = []
|
||||
for i in rng.integers(lo, hi, size=reps):
|
||||
i = int(i)
|
||||
sl_l = df_l.iloc[i - WIN_LTF + 1: i + 1].reset_index(drop=True)
|
||||
# 5m 只喂到不晚于该 1m 根的部分,与实盘一致
|
||||
h_end = int(np.searchsorted(htf_ts, ltf_ts[i], side="right"))
|
||||
sl_h = (df_h.iloc[h_end - WIN_HTF: h_end].reset_index(drop=True)
|
||||
if h_end >= WIN_HTF else None)
|
||||
out.append((sl_l, sl_h))
|
||||
return out
|
||||
|
||||
|
||||
def bench_pair(task: tuple) -> dict:
|
||||
"""算一根:1m 信号 + 5m 时间线。在子进程里跑,import 都在函数内。"""
|
||||
import warnings as _w
|
||||
_w.filterwarnings("ignore")
|
||||
sys.path.insert(0, str(RESEARCH))
|
||||
sys.path.insert(0, str(RESEARCH.parent))
|
||||
|
||||
sym, sl_l, sl_h = task
|
||||
_, dt_l = _signal_once(sl_l)
|
||||
dt_h = np.nan if sl_h is None else _agree_once(sl_h)[1]
|
||||
return {"sym": sym, "ltf_s": dt_l, "htf_s": dt_h}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--reps", type=int, default=15, help="每币采样窗口数")
|
||||
ap.add_argument("--rounds", type=int, default=5, help="串行/并行各跑几轮")
|
||||
args = ap.parse_args()
|
||||
|
||||
print(f"[本机算力] {os.cpu_count()} 逻辑核 · 1m 窗口 {WIN_LTF} 根 · "
|
||||
f"5m 窗口 {WIN_HTF} 根 · 每币 {args.reps} 次\n", flush=True)
|
||||
|
||||
print("########## 1. 单币单次耗时 ##########", flush=True)
|
||||
prepared = {s: _slices(s, args.reps, 7) for s in SYMS}
|
||||
per = []
|
||||
for s in SYMS:
|
||||
d = pd.DataFrame([bench_pair((s, l, h)) for l, h in prepared[s]])
|
||||
per.append(d)
|
||||
print(f" {s}: 1m {d['ltf_s'].median():.3f}s (P95 {d['ltf_s'].quantile(.95):.3f}s)"
|
||||
f" · 5m {d['htf_s'].median():.3f}s", flush=True)
|
||||
allp = pd.concat(per, ignore_index=True)
|
||||
|
||||
m_ltf = allp["ltf_s"].median()
|
||||
m_htf = allp["htf_s"].median()
|
||||
# 5m 侧只在每 5 根 1m 里变一次,其余 4 根可复用缓存
|
||||
amort = m_ltf + m_htf / 5
|
||||
print(f"\n 三币合并中位:1m {m_ltf:.3f}s · 5m {m_htf:.3f}s")
|
||||
print(f" 单币每根摊薄成本 {amort:.3f}s(5m 每 5 根才重算一次)")
|
||||
print(f" 对比 Mac 的 0.20s:本机慢 {m_ltf / 0.20:.1f} 倍")
|
||||
|
||||
print("\n########## 2. 三币串行 vs 并行(最后一个币的下单延迟)##########",
|
||||
flush=True)
|
||||
print(" 只计算子耗时:读数据不计入,进程池常驻不计启动开销", flush=True)
|
||||
ser, par2, par3 = [], [], []
|
||||
pools = {w: ProcessPoolExecutor(max_workers=w) for w in (2, 3)}
|
||||
try:
|
||||
# 先各跑一次把子进程的 import 预热掉,否则首轮全是模块加载时间
|
||||
for w, ex in pools.items():
|
||||
list(ex.map(bench_pair, [(s, *prepared[s][0]) for s in SYMS]))
|
||||
|
||||
for k in range(args.rounds):
|
||||
batch = [(s, *prepared[s][k % args.reps]) for s in SYMS]
|
||||
t0 = time.perf_counter()
|
||||
for t in batch:
|
||||
bench_pair(t)
|
||||
ser.append(time.perf_counter() - t0)
|
||||
|
||||
for w, bag in ((2, par2), (3, par3)):
|
||||
t0 = time.perf_counter()
|
||||
list(pools[w].map(bench_pair, batch))
|
||||
bag.append(time.perf_counter() - t0)
|
||||
print(f" 轮 {k + 1}: 串行 {ser[-1]:.2f}s · 并行2 {par2[-1]:.2f}s · "
|
||||
f"并行3 {par3[-1]:.2f}s", flush=True)
|
||||
finally:
|
||||
for ex in pools.values():
|
||||
ex.shutdown()
|
||||
|
||||
print("\n########## 3. 延迟折算成 bp(3.9bp 预算的参照)##########")
|
||||
# 各币 1m 收益标准差,用 √t 把延迟折成价格漂移的一个标准差
|
||||
vol = {}
|
||||
for s in SYMS:
|
||||
df_l, _ = _load(s)
|
||||
r = np.diff(np.log(df_l["close"].to_numpy(dtype=float)[-400_000:]))
|
||||
vol[s] = float(np.std(r) * 1e4)
|
||||
rows = []
|
||||
for name, xs in (("串行", ser), ("并行2", par2), ("并行3", par3)):
|
||||
d = float(np.median(xs))
|
||||
rows.append({"方案": name, "三币总耗时": f"{d:.2f}s",
|
||||
**{f"{s} 漂移1σ": f"{vol[s] * np.sqrt(d / 60):.2f}bp"
|
||||
for s in SYMS}})
|
||||
tb = pd.DataFrame(rows)
|
||||
print(tb.to_string(index=False))
|
||||
print(f"\n 1m 波动实测:" + " · ".join(f"{s} {vol[s]:.1f}bp/分钟" for s in SYMS))
|
||||
print(" 漂移 1σ 是随机部分;入场时价格正朝信号方向跑,系统性追价另计。")
|
||||
|
||||
out = RESEARCH / "out" / "bench_compute.csv"
|
||||
pd.DataFrame({
|
||||
"指标": ["cpu核数", "1m窗口", "5m窗口", "1m中位s", "1m_P95s", "5m中位s",
|
||||
"单币摊薄s", "串行3币s", "并行2_3币s", "并行3_3币s"],
|
||||
"值": [os.cpu_count(), WIN_LTF, WIN_HTF, round(m_ltf, 3),
|
||||
round(allp["ltf_s"].quantile(.95), 3), round(m_htf, 3),
|
||||
round(amort, 3), round(float(np.median(ser)), 2),
|
||||
round(float(np.median(par2)), 2), round(float(np.median(par3)), 2)],
|
||||
}).to_csv(out, index=False)
|
||||
print(f"\n产物写入 {out}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,120 @@
|
||||
"""Bitget 原生回测基线:影子交易器要对照的那个数。
|
||||
|
||||
aggregate_robustness 证明总体期望在数据扰动下稳定,但逐笔清单不可复现。
|
||||
所以滑点不能逐笔对照 Binance 回测,只能对照「同一 venue 上的总体毛均收益」。
|
||||
这一步就是把那个数算出来。
|
||||
|
||||
跑法与 Binance 侧完全对齐:同一时间窗、同样 300k 根 1m、同一套 lib/ 代码、
|
||||
同样 SL/TP/MAX_BARS = 1.5/3.0/48 与 entry_delay=1,只换数据源。
|
||||
|
||||
特别关注 SOL:Bitget 的 tick 是 0.001、Binance 是 0.01,粗 10 倍会让
|
||||
`close > high[j-1]` 大量平局不触发。Bitget 上信号多出 74%,多出来的是否
|
||||
同样赚钱,只有跑一遍才知道——这不是随机扰动,是系统性差异。
|
||||
|
||||
输出 out/bitget_baseline.csv。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
|
||||
os.environ.setdefault(v, "1")
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
RESEARCH = HERE.parent
|
||||
sys.path.insert(0, str(RESEARCH))
|
||||
sys.path.insert(0, str(RESEARCH.parent))
|
||||
sys.path.insert(0, str(HERE))
|
||||
pd.set_option("display.width", 260)
|
||||
|
||||
from aggregate_robustness import run_once, stats # noqa: E402
|
||||
from venue_parity import fetch_bitget # noqa: E402
|
||||
|
||||
# Binance 侧在同一窗口的实测值,来自 out/aggregate_robustness_*.csv 的 0bp 档
|
||||
BINANCE_REF = {"BTC": 6.11, "ETH": 12.31, "SOL": 10.01}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--symbols", default="BTC,ETH,SOL")
|
||||
ap.add_argument("--days", type=int, default=210,
|
||||
help="与 Binance 侧的 300k 根(208 天)对齐")
|
||||
ap.add_argument("--refresh", action="store_true")
|
||||
ap.add_argument("--tag", default="", help="产物文件名后缀,避免多次调用互相覆盖")
|
||||
args = ap.parse_args()
|
||||
|
||||
from lib.data import load_local
|
||||
|
||||
syms = [s.strip() for s in args.symbols.split(",")]
|
||||
print(f"[Bitget 原生基线] {syms} · 近 {args.days} 天 1m\n", flush=True)
|
||||
|
||||
rows = []
|
||||
for sym in syms:
|
||||
t0 = time.perf_counter()
|
||||
bg_l = fetch_bitget(sym, "1m", args.days, args.refresh)
|
||||
bg_h = fetch_bitget(sym, "5m", args.days, args.refresh)
|
||||
print(f"── {sym} {len(bg_l)} 根 1m / {len(bg_h)} 根 5m "
|
||||
f"{bg_l['date'].iloc[0]:%Y-%m-%d} ~ {bg_l['date'].iloc[-1]:%Y-%m-%d}"
|
||||
f" [拉取 {time.perf_counter() - t0:.0f}s]", flush=True)
|
||||
|
||||
t1 = time.perf_counter()
|
||||
tr = run_once(bg_l, bg_h)
|
||||
s = stats(tr)
|
||||
rows.append({"品种": sym, "venue": "Bitget", **s})
|
||||
print(f" Bitget : " + " · ".join(f"{k} {v}" for k, v in s.items())
|
||||
+ f" [{time.perf_counter() - t1:.0f}s]", flush=True)
|
||||
|
||||
# 同窗口的 Binance 对照,直接重算一遍,避免口径漂移
|
||||
lo, hi = int(bg_l.timestamp.min()), int(bg_l.timestamp.max())
|
||||
bn_l = load_local(f"{sym}/USDT:USDT", "1m")
|
||||
bn_h = load_local(f"{sym}/USDT:USDT", "5m")
|
||||
if bn_l is None:
|
||||
continue
|
||||
bn_l = bn_l[(bn_l.timestamp >= lo) & (bn_l.timestamp <= hi)].reset_index(drop=True)
|
||||
bn_h = bn_h[(bn_h.timestamp >= lo) & (bn_h.timestamp <= hi)].reset_index(drop=True)
|
||||
t1 = time.perf_counter()
|
||||
s2 = stats(run_once(bn_l, bn_h))
|
||||
rows.append({"品种": sym, "venue": "Binance", **s2})
|
||||
print(f" Binance: " + " · ".join(f"{k} {v}" for k, v in s2.items())
|
||||
+ f" [{time.perf_counter() - t1:.0f}s]", flush=True)
|
||||
|
||||
if not rows:
|
||||
print("无结果")
|
||||
return
|
||||
|
||||
tb = pd.DataFrame(rows)
|
||||
tb["毛均bp"] = tb["毛均收益"].str.rstrip("%").astype(float) * 100
|
||||
print("\n" + "=" * 120)
|
||||
print("########## 同窗口 · 同代码 · 只换数据源 ##########")
|
||||
print(tb.to_string(index=False))
|
||||
|
||||
print("\n########## 毛均收益对照(bp)##########")
|
||||
piv = tb.pivot_table(index="品种", columns="venue", values="毛均bp")
|
||||
piv["差额"] = (piv.get("Bitget", np.nan) - piv.get("Binance", np.nan)).round(2)
|
||||
cnt = tb.pivot_table(index="品种", columns="venue", values="笔数")
|
||||
piv["Bitget笔数"] = cnt.get("Bitget")
|
||||
piv["Binance笔数"] = cnt.get("Binance")
|
||||
piv["Bitget余量bp"] = (piv.get("Bitget", np.nan) - 6.0).round(2)
|
||||
print(piv.round(2).to_string())
|
||||
|
||||
print("\n########## 结论 ##########")
|
||||
print(" Bitget余量 = 该 venue 自己的毛均收益 − 6bp 双边费率(VIP1 taker + API 返50%)。")
|
||||
print(" 这就是影子交易器测出的滑点要去比的那条线,逐笔清单不参与比较。")
|
||||
print(" 两家毛均若接近 → 总体口径可迁移;若差很多 → 以 Bitget 的为准。")
|
||||
|
||||
out = RESEARCH / "out" / f"bitget_baseline{args.tag}.csv"
|
||||
tb.to_csv(out, index=False)
|
||||
print(f"\n产物写入 {out}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,211 @@
|
||||
"""对比两个(或多个)采集站点的数据差异。
|
||||
|
||||
## 该比什么(2026-08-28 实测修正)
|
||||
|
||||
原以为该比地理位置,实测下来不是。腾讯云新加坡的延迟构成:
|
||||
|
||||
数据到达 506ms 其中网络仅 2ms 往返(ws.bitget.com 是 CloudFront 边缘)
|
||||
信号计算 646ms 本机 2 核,三币同时收盘还要排队
|
||||
合计 1315ms
|
||||
|
||||
换机房只能动那 2ms。**主指标是 compute_ms,不是 lag_data_ms。**
|
||||
|
||||
lag_data_ms 仍然要看,但作用是**自检**:两站应当几乎相同;若差很多,先怀疑
|
||||
时钟没对齐——那比网络差异的可能性大得多。
|
||||
|
||||
## 判读前必须先看的两件事
|
||||
|
||||
1. **时钟。** 两台机器的时钟偏移差多少,延迟对比就凭空差多少,且不报错。
|
||||
run_meta_*.json 里有各站启动时的 chrony 偏移,先确认都在 10ms 内。
|
||||
2. **同期。** 只比两站都有数据的那些 kline_ts。不取交集的话,比的可能是
|
||||
不同时段的市场状态,而延迟对市场活跃度是敏感的。
|
||||
|
||||
## 用法
|
||||
|
||||
把各站的 research/out/ 收到一处(文件名相同会覆盖,所以先按站点改名或
|
||||
分目录放),然后:
|
||||
|
||||
python research/live/compare_sites.py --glob 'collected/*/shadow_latency.csv'
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import glob
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
|
||||
def load(patterns: list[str]) -> pd.DataFrame:
|
||||
paths: list[str] = []
|
||||
for p in patterns:
|
||||
paths.extend(sorted(glob.glob(p)))
|
||||
if not paths:
|
||||
raise SystemExit(f"没有匹配到文件:{patterns}")
|
||||
frames = []
|
||||
for p in paths:
|
||||
df = pd.read_csv(p)
|
||||
if "site" not in df.columns:
|
||||
raise SystemExit(
|
||||
f"{p} 没有 site 列。这是 2026-08-28 之前采的旧数据,"
|
||||
f"无法确定来源,不能用于跨地对比")
|
||||
df["_src"] = p
|
||||
frames.append(df)
|
||||
out = pd.concat(frames, ignore_index=True)
|
||||
print(f"读入 {len(paths)} 个文件、{len(out):,} 行、"
|
||||
f"站点 {sorted(out['site'].unique())}")
|
||||
return out
|
||||
|
||||
|
||||
def show_meta(out_dirs: list[Path]) -> None:
|
||||
print("\n########## 一、运行元数据 ##########")
|
||||
metas = []
|
||||
for d in out_dirs:
|
||||
metas.extend(sorted(d.glob("run_meta_*.json")))
|
||||
if not metas:
|
||||
print(" 没找到 run_meta_*.json。时钟偏移与代码版本无法核对——")
|
||||
print(" 两站数据若有差异,分不清是地理位置还是环境不同造成的")
|
||||
return
|
||||
rows = []
|
||||
for m in metas:
|
||||
try:
|
||||
rows.append(json.loads(m.read_text()))
|
||||
except Exception as e:
|
||||
print(f" {m.name} 读取失败:{e!r}")
|
||||
if not rows:
|
||||
return
|
||||
df = pd.DataFrame(rows)
|
||||
# net 是嵌套 dict,摊平出关心的几项
|
||||
if "net" in df.columns:
|
||||
for k in ("icmp_min_ms", "tcp_connect_min_ms", "ttfb_min_ms",
|
||||
"self_org", "edge_org"):
|
||||
df[k] = df["net"].apply(
|
||||
lambda v, k=k: v.get(k) if isinstance(v, dict) else None)
|
||||
keep = [c for c in ("site", "clock_offset_ms", "nproc", "cpu_model",
|
||||
"icmp_min_ms", "ttfb_min_ms", "git_commit",
|
||||
"git_dirty", "image_digest", "mem_gb", "tz",
|
||||
"started_utc") if c in df.columns]
|
||||
print(df[keep].to_string(index=False))
|
||||
if "icmp_min_ms" in df and df["icmp_min_ms"].notna().any():
|
||||
x = df["icmp_min_ms"].astype(float)
|
||||
print(f"\n 到 Bitget 边缘的往返:{x.min():.1f}~{x.max():.1f}ms。"
|
||||
f"站间极差 {x.max() - x.min():.1f}ms 就是换机房的全部空间")
|
||||
if "clock_offset_ms" in df and df["clock_offset_ms"].notna().any():
|
||||
o = df["clock_offset_ms"].astype(float)
|
||||
spread = float(o.max() - o.min())
|
||||
flag = "" if spread < 5 else " ⚠ 这个差会直接叠加到延迟对比上"
|
||||
print(f"\n 站点间时钟偏移极差 {spread:.3f}ms{flag}")
|
||||
if "git_commit" in df and df["git_commit"].nunique() > 1:
|
||||
print(" ⚠ 各站代码版本不同,差异可能来自代码而非地理位置")
|
||||
if "git_dirty" in df and df["git_dirty"].any():
|
||||
print(" ⚠ 有站点带未提交改动,无法复现")
|
||||
|
||||
|
||||
def compare_latency(df: pd.DataFrame, col: str = "lag_data_ms") -> None:
|
||||
"""延迟对比。只取各站都有的 kline_ts,避免比到不同时段。"""
|
||||
if col not in df.columns:
|
||||
print(f"\n没有 {col} 列")
|
||||
return
|
||||
sites = sorted(df["site"].unique())
|
||||
if len(sites) < 2:
|
||||
print(f"\n只有一个站点({sites[0]}),无从对比。"
|
||||
f"等第二台机器的数据到齐")
|
||||
return
|
||||
|
||||
label = {"compute_ms": "信号计算耗时(主指标,取决于 CPU)",
|
||||
"lag_data_ms": "数据到达延迟(自检项,两站应当接近)",
|
||||
"lag_signal_ms": "合计到可下单"}.get(col, col)
|
||||
print(f"\n########## {label}({col}) ##########")
|
||||
print("\n 全量(各站各自的样本,时段可能不同)")
|
||||
for s in sites:
|
||||
x = df[df["site"] == s][col].dropna().astype(float)
|
||||
print(f" {s:<16} n={len(x):>6} 中位 {x.median():>7.0f}ms "
|
||||
f"P90 {np.percentile(x, 90):>7.0f}ms "
|
||||
f"P99 {np.percentile(x, 99):>7.0f}ms")
|
||||
|
||||
# 取交集:同一根 K 线在各站都有记录
|
||||
key = ["sym", "kline_ts"]
|
||||
piv = df.pivot_table(index=key, columns="site", values=col,
|
||||
aggfunc="first")
|
||||
both = piv.dropna()
|
||||
if both.empty:
|
||||
print("\n 各站没有共同的 K 线。可能是采集时段不重叠,")
|
||||
print(" 或 kline_ts 对不上(先查两站时区与时钟)")
|
||||
return
|
||||
print(f"\n 同根对比({len(both):,} 根 K 线,各站都有)")
|
||||
for s in sites:
|
||||
x = both[s].astype(float)
|
||||
print(f" {s:<16} 中位 {x.median():>7.0f}ms "
|
||||
f"P90 {np.percentile(x, 90):>7.0f}ms")
|
||||
base = sites[0]
|
||||
for s in sites[1:]:
|
||||
d = (both[s] - both[base]).astype(float)
|
||||
# 配对差的符号检验:同根配对消掉了市场状态,比两个中位数相减干净
|
||||
n_pos = int((d > 0).sum())
|
||||
print(f"\n {s} − {base}:中位差 {d.median():+.0f}ms "
|
||||
f"· 均值差 {d.mean():+.0f}ms")
|
||||
print(f" {s} 更慢的根占 {n_pos / len(d) * 100:.1f}%"
|
||||
f"(50% 表示无系统性差异)")
|
||||
for sym in sorted(both.index.get_level_values("sym").unique()):
|
||||
ds = d.xs(sym, level="sym")
|
||||
print(f" {sym:<5} 中位差 {ds.median():+7.0f}ms (n={len(ds)})")
|
||||
|
||||
|
||||
def compare_drift(patterns: list[str]) -> None:
|
||||
"""漂移对比。延迟差若能兑换成漂移差,才是钱上的差别。"""
|
||||
paths: list[str] = []
|
||||
for p in patterns:
|
||||
paths.extend(sorted(glob.glob(p)))
|
||||
if not paths:
|
||||
return
|
||||
frames = []
|
||||
for p in paths:
|
||||
d = pd.read_csv(p)
|
||||
if "site" in d.columns:
|
||||
frames.append(d)
|
||||
if not frames:
|
||||
return
|
||||
df = pd.concat(frames, ignore_index=True)
|
||||
if df["site"].nunique() < 2:
|
||||
return
|
||||
print("\n########## 三、延迟漂移(无条件,每根都记) ##########")
|
||||
for label in sorted(df["delay_label"].dropna().unique()):
|
||||
sub = df[df["delay_label"] == label]
|
||||
line = f" {label:>7}"
|
||||
for s in sorted(sub["site"].unique()):
|
||||
x = sub[sub["site"] == s]["drift_bp_long"].dropna().astype(float)
|
||||
if len(x):
|
||||
line += f" · {s} {x.abs().median():.3f}bp(n={len(x)})"
|
||||
print(line)
|
||||
print("\n 同一个固定延迟点上,两站的漂移应当几乎相同——漂移是市场性质,")
|
||||
print(" 与机器位置无关。若差异明显,先查时钟与采集时段是否对齐")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--glob", action="append", default=None,
|
||||
help="shadow_latency.csv 的路径模式,可给多次")
|
||||
ap.add_argument("--drift-glob", action="append", default=None)
|
||||
ap.add_argument("--meta-dir", action="append", default=None)
|
||||
a = ap.parse_args()
|
||||
|
||||
lat = a.glob or ["research/out/shadow_latency.csv",
|
||||
"collected/*/shadow_latency.csv"]
|
||||
drf = a.drift_glob or ["research/out/shadow_drift.csv",
|
||||
"collected/*/shadow_drift.csv"]
|
||||
metas = [Path(p) for p in (a.meta_dir or ["research/out", "collected"])]
|
||||
|
||||
show_meta([p for p in metas if p.is_dir()])
|
||||
df = load(lat)
|
||||
# compute_ms 是主指标:它是延迟里唯一有大幅改善空间的一项(646ms vs
|
||||
# 网络的 2ms)。lag_data_ms 放后面,作用是自检两站是否可比
|
||||
compare_latency(df, "compute_ms")
|
||||
compare_latency(df, "lag_data_ms")
|
||||
compare_latency(df, "lag_signal_ms")
|
||||
compare_drift(drf)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,149 @@
|
||||
# 影子采集器部署
|
||||
|
||||
在第二台机器上跑一套完全相同的采集,用来看不同地理位置的数据差异。
|
||||
|
||||
## 先看这一节:延迟的瓶颈不在机房
|
||||
|
||||
滑点里最大的一项是**延迟漂移**,所以延迟越低越好。但延迟拆开之后,可优化的
|
||||
地方和直觉不一样。腾讯云新加坡实测(2026-08-28):
|
||||
|
||||
| 构成 | 中位 | 随机房位置变化吗 |
|
||||
| --- | --- | --- |
|
||||
| 数据到达(交易所推送 + 回源 + 网络) | 506ms | 只有网络那段,**2ms** |
|
||||
| 信号计算(本机 CPU) | **646ms** | 不变,取决于 CPU |
|
||||
| 合计到可下单 | 1315ms | |
|
||||
|
||||
`ws.bitget.com` 解析出来是 **CloudFront**(`dxotqhr62n6z4.cloudfront.net`),
|
||||
落在新加坡的 AS16509(Amazon)。所以我们连的是 AWS 的 CDN 边缘,不是 Bitget
|
||||
自己的机房。从腾讯云新加坡到这个边缘:
|
||||
|
||||
ICMP 往返 2.1ms
|
||||
TCP 握手 3.3ms
|
||||
TLS 完成 9.0ms
|
||||
首字节 87.8ms ← 减去 TLS 的 9ms,约 79ms 是 CloudFront 回源开销
|
||||
|
||||
回源那 79ms 和交易所自己的推送节奏,不管我们坐在哪都一样。而随位置变化的
|
||||
只有那 2ms 往返。**换机房的全部空间是 1~2ms,占总延迟 1315ms 的 0.1%。**
|
||||
|
||||
真正的大头是本机 646ms 的信号计算:每根 K 线要在 2000 根 1m 加 800 根 5m 上
|
||||
重建缠论结构,三个币同时收盘而本机只有 2 核、2 个计算进程,第三个币还要排队。
|
||||
|
||||
**所以第二台机器该测的是 CPU 规格,不是地理位置。** 看 `compute_ms`,不是
|
||||
`lag_data_ms`。3 个币至少要 3 个 worker,`--workers` 给到核数减一。
|
||||
|
||||
## 三个必须一致,一个必须不同
|
||||
|
||||
必须一致,否则差异分不清是地理位置还是环境造成的:
|
||||
|
||||
- **代码版本**(`git_commit`)——同一个 commit
|
||||
- **镜像摘要**(`image_digest`)——同一个 hummingbot 镜像
|
||||
- **时钟**——两台都同步到 NTP,偏移都在 10ms 内
|
||||
|
||||
必须不同:
|
||||
|
||||
- **`SHADOW_SITE`**——写进每一行数据,是合并后区分来源的唯一依据
|
||||
|
||||
`start.sh` 会把这四项连同内核、核数、内存一起写进
|
||||
`research/out/run_meta_<site>.json`。两地数据对不上时先看这个文件。
|
||||
|
||||
## 时钟为什么是硬门槛
|
||||
|
||||
所有延迟数字都是「本地时钟 − 交易所 K 线收盘时间戳」。时钟偏 50ms,全部
|
||||
延迟就同向偏 50ms,而且**不会有任何报错**——只会让跨地对比得出一个干净、
|
||||
自信、且完全错误的结论。所以 `start.sh` 在时钟未同步或偏移超阈值时直接
|
||||
拒绝启动,而不是打个警告了事。
|
||||
|
||||
## 步骤
|
||||
|
||||
在新机器上:
|
||||
|
||||
```bash
|
||||
git clone ssh://jack@git.jackyu66.com:2222/jack/chan.git
|
||||
cd chan && git checkout chan
|
||||
|
||||
bash research/live/deploy/setup.sh # 装 docker + chrony,拉镜像
|
||||
|
||||
# 站点名带上机型,因为要比的是 CPU 而不是位置
|
||||
SHADOW_SITE=aws-sg-c7a4x WORKERS=3 bash research/live/deploy/start.sh
|
||||
```
|
||||
|
||||
`WORKERS` 按核数减一给。3 个币同时收盘,worker 少于 3 就会排队,而排队时间
|
||||
直接计入 `compute_ms`。本机 2 核只能给 2,这本身就是 646ms 里的一部分。
|
||||
|
||||
确认健康:
|
||||
|
||||
```bash
|
||||
bash research/live/deploy/status.sh
|
||||
```
|
||||
|
||||
启动日志里应当看到:
|
||||
|
||||
```
|
||||
[补丁] 覆盖生效:基类取首元素 … 本地取末元素 …
|
||||
成交流已挂 ['BTC', 'ETH', 'SOL']
|
||||
就绪 3.0s · 1m [2001, 2001, 2001] 根 · 5m [801, 801, 801] 根
|
||||
```
|
||||
|
||||
第一行尤其重要。上游 Bitget 连接器的换根解析有 bug(只取多根消息的首元素),
|
||||
补丁把它修掉拿回约 1.06 秒。补丁若失效是静默的——不崩不报错,只是延迟悄悄
|
||||
退回 1.4 秒,所以启动时做了断言。
|
||||
|
||||
## 对比
|
||||
|
||||
把两站的 `research/out/` 收到一处(同名文件会覆盖,所以分目录放):
|
||||
|
||||
```bash
|
||||
mkdir -p collected/sg collected/aws
|
||||
rsync -av sg-box:chan/research/out/ collected/sg/
|
||||
rsync -av aws-box:chan/research/out/ collected/aws/
|
||||
|
||||
python research/live/compare_sites.py \
|
||||
--glob 'collected/*/shadow_latency.csv' \
|
||||
--drift-glob 'collected/*/shadow_drift.csv' \
|
||||
--meta-dir collected/sg --meta-dir collected/aws
|
||||
```
|
||||
|
||||
对比脚本做两件事值得说明:
|
||||
|
||||
- **只取各站都有的 K 线**做配对比较。不取交集就可能在比不同时段,而延迟对
|
||||
市场活跃度敏感。
|
||||
- 报**配对差的符号占比**而不只是两个中位数相减。同根配对消掉了市场状态,
|
||||
「A 比 B 慢的根占多少」比「两个中位数差多少」更能说明有无系统性差异。
|
||||
|
||||
判读上有一条自检:**同一个固定延迟点上,两站的漂移应当几乎相同**——漂移是
|
||||
市场性质,与机器位置无关。若漂移也差很多,先怀疑时钟或时段没对齐,而不是
|
||||
急着下结论。
|
||||
|
||||
同理,`lag_data_ms` 两站也应当几乎相同(网络那段只有 2ms 空间)。真正该出现
|
||||
差异的是 `compute_ms`。如果 `lag_data_ms` 差很多,先查时钟——比查网络更可能。
|
||||
|
||||
⚠ 但先读下面「资源占用」一节:`compute_ms` 的差异几乎全部来自单核性能,而
|
||||
现役机型之间单核差距很小。**跨站点比 CPU 这件事本身收益有限**,本节流程保留
|
||||
是为了比网络与时钟,不建议为了比 CPU 单独开机器。
|
||||
|
||||
## 资源占用
|
||||
|
||||
本机实测(2 vCPU EPYC 9K65 / 3 个币 / 2 worker):内存 **410MiB**,CPU 均值
|
||||
1~3%。盘口与成交流落盘约 15MB/天(gzip),一周在百 MB 内。
|
||||
|
||||
内存和平均 CPU 都不是约束。约束是**单根 K 线的计算延迟**,而它是纯单线程的:
|
||||
|
||||
```
|
||||
compute_ms 中位 764ms 父进程测的墙钟,含排队
|
||||
queue_ms 中位 3ms 等空闲 worker
|
||||
inner_ms 中位 695ms 进程内真正在算
|
||||
```
|
||||
|
||||
`queue_ms` 只有 3ms,说明 **2 个 worker 跑 3 个币并不排队**——三个币的收盘消息
|
||||
错峰到达(SOL 最晚,排 66ms),没有真正的并发争抢。
|
||||
|
||||
**这条结论直接否掉了「换更强机器」这个方向。** 加核只能压 queue_ms,而它已经
|
||||
是 3ms;695ms 全在单线程里,取决于单核性能。t3a.medium(Zen 1,2017)单核比
|
||||
本机 Zen 5 慢 1.8~2 倍,换过去 compute_ms 会涨到 1200ms 以上。c7a / c6a 这类
|
||||
现代机型单核与本机相当,也换不到东西。
|
||||
|
||||
要压这 695ms 只有算法一条路:现在每分钟把 2001 根从头算一遍,其中 2000 根的
|
||||
结构与上一分钟完全相同。
|
||||
|
||||
`--workers 2` 是因为信号计算走独立进程池、不能阻塞事件循环。币数超过 worker
|
||||
数才会看到 queue_ms 上来;届时加 worker 有效,加到与币数相等即可。
|
||||
Executable
+69
@@ -0,0 +1,69 @@
|
||||
#!/usr/bin/env bash
|
||||
# 量本机到 Bitget 端点的网络距离,输出 JSON。start.sh 会把它并进运行元数据。
|
||||
#
|
||||
# 为什么要单独量:K 线到达延迟(lag_data_ms)是「交易所推送节奏 + 回源 + 网络」
|
||||
# 三者之和,中位 506ms。其中只有最后一段随机房位置变化,而实测它只有 2ms
|
||||
# 往返——用到达延迟去比两个机房,等于用公斤秤称克。这个探测把可变的那一段
|
||||
# 单独拿出来。
|
||||
#
|
||||
# 还有一件事值得知道:ws.bitget.com 解析出来是 CloudFront(AWS 的 CDN 边缘),
|
||||
# 不是 Bitget 自己的机房。所以「离交易所近」实际是「离 CloudFront 边缘近」。
|
||||
set -uo pipefail
|
||||
|
||||
WS_HOST="${WS_HOST:-ws.bitget.com}"
|
||||
N="${N:-10}"
|
||||
|
||||
resolved="$(getent hosts "$WS_HOST" 2>/dev/null | head -1 || true)"
|
||||
edge_ip="$(awk '{print $1}' <<<"$resolved")"
|
||||
cname="$(awk '{print $2}' <<<"$resolved")"
|
||||
|
||||
icmp_min=null; icmp_avg=null
|
||||
if out="$(ping -c "$N" -q -W 2 "$WS_HOST" 2>/dev/null)"; then
|
||||
stats="$(grep -oE 'rtt min/avg/max/mdev = [0-9./]+' <<<"$out" | awk '{print $NF}')"
|
||||
if [[ -n "$stats" ]]; then
|
||||
icmp_min="$(cut -d/ -f1 <<<"$stats")"
|
||||
icmp_avg="$(cut -d/ -f2 <<<"$stats")"
|
||||
fi
|
||||
fi
|
||||
|
||||
# TCP 握手是一个完整往返,比 ICMP 更能代表实际连接路径(有些网络对 ICMP 降级)
|
||||
tcp_min=null; tls_min=null; ttfb_min=null
|
||||
if command -v curl >/dev/null 2>&1; then
|
||||
vals="$(for _ in $(seq "$N"); do
|
||||
curl -s -o /dev/null --max-time 8 \
|
||||
-w '%{time_connect} %{time_appconnect} %{time_starttransfer}\n' \
|
||||
"https://$WS_HOST/v2/ws/public" 2>/dev/null
|
||||
done)"
|
||||
if [[ -n "$vals" ]]; then
|
||||
tcp_min="$(awk '{print $1*1000}' <<<"$vals" | sort -n | head -1)"
|
||||
tls_min="$(awk '{print $2*1000}' <<<"$vals" | sort -n | head -1)"
|
||||
ttfb_min="$(awk '{print $3*1000}' <<<"$vals" | sort -n | head -1)"
|
||||
fi
|
||||
fi
|
||||
|
||||
org="unknown"
|
||||
if [[ -n "$edge_ip" ]]; then
|
||||
org="$(curl -s --max-time 6 "https://ipinfo.io/$edge_ip/json" 2>/dev/null \
|
||||
| awk -F'"' '/"org"/{print $4}')"
|
||||
[[ -n "$org" ]] || org="unknown"
|
||||
fi
|
||||
self_org="$(curl -s --max-time 6 https://ipinfo.io/json 2>/dev/null \
|
||||
| awk -F'"' '/"org"/{print $4}')"
|
||||
self_city="$(curl -s --max-time 6 https://ipinfo.io/json 2>/dev/null \
|
||||
| awk -F'"' '/"city"/{print $4}')"
|
||||
|
||||
cat <<EOF
|
||||
{
|
||||
"ws_host": "$WS_HOST",
|
||||
"edge_ip": "${edge_ip:-unknown}",
|
||||
"edge_cname": "${cname:-unknown}",
|
||||
"edge_org": "$org",
|
||||
"self_org": "${self_org:-unknown}",
|
||||
"self_city": "${self_city:-unknown}",
|
||||
"icmp_min_ms": ${icmp_min:-null},
|
||||
"icmp_avg_ms": ${icmp_avg:-null},
|
||||
"tcp_connect_min_ms": ${tcp_min:-null},
|
||||
"tls_done_min_ms": ${tls_min:-null},
|
||||
"ttfb_min_ms": ${ttfb_min:-null}
|
||||
}
|
||||
EOF
|
||||
Executable
+87
@@ -0,0 +1,87 @@
|
||||
#!/usr/bin/env bash
|
||||
# 影子采集器的机器初始化。幂等,可重复跑。
|
||||
#
|
||||
# 用途:在另一台机器(如 AWS)上部署一套完全相同的采集,用来看不同地理位置
|
||||
# 的数据采集有无差异。要比的主要是延迟——「本地接收 − K线收盘」这个量直接
|
||||
# 取决于机器到交易所的网络距离。
|
||||
#
|
||||
# 时钟同步是硬前置条件,不是可选项。所有延迟数字都是本地时钟减交易所时间戳,
|
||||
# 时钟偏 50ms 就等于所有延迟凭空多(或少)50ms,而且不会有任何报错。所以这里
|
||||
# 装并启用 NTP,start.sh 里还会再校验一次、不合格拒绝启动。
|
||||
#
|
||||
# bash research/live/deploy/setup.sh
|
||||
set -euo pipefail
|
||||
|
||||
IMAGE="${SHADOW_IMAGE:-hummingbot/hummingbot:latest}"
|
||||
|
||||
say() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
|
||||
|
||||
say "系统信息"
|
||||
uname -a
|
||||
echo "内存:$(free -g | awk '/^Mem:/{print $2"GB 总 / "$7"GB 可用"}')"
|
||||
echo "CPU:$(nproc) 核"
|
||||
|
||||
say "安装 docker"
|
||||
if command -v docker >/dev/null 2>&1; then
|
||||
echo "已有 docker $(docker --version)"
|
||||
else
|
||||
if command -v apt-get >/dev/null 2>&1; then
|
||||
sudo apt-get update -qq
|
||||
sudo apt-get install -y -qq ca-certificates curl gnupg
|
||||
sudo install -m 0755 -d /etc/apt/keyrings
|
||||
curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
|
||||
| sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
|
||||
sudo chmod a+r /etc/apt/keyrings/docker.gpg
|
||||
. /etc/os-release
|
||||
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
|
||||
https://download.docker.com/linux/ubuntu ${VERSION_CODENAME} stable" \
|
||||
| sudo tee /etc/apt/sources.list.d/docker.list >/dev/null
|
||||
sudo apt-get update -qq
|
||||
sudo apt-get install -y -qq docker-ce docker-ce-cli containerd.io
|
||||
elif command -v dnf >/dev/null 2>&1; then
|
||||
# Amazon Linux 2023
|
||||
sudo dnf install -y -q docker
|
||||
sudo systemctl enable --now docker
|
||||
else
|
||||
echo "不认识的包管理器,请手动装 docker" >&2
|
||||
exit 1
|
||||
fi
|
||||
sudo usermod -aG docker "$USER" || true
|
||||
echo "已装 docker。若本次 shell 无权限,重新登录后再跑 start.sh"
|
||||
fi
|
||||
|
||||
say "启用时钟同步(延迟测量的前置条件)"
|
||||
if command -v chronyc >/dev/null 2>&1; then
|
||||
echo "已有 chrony"
|
||||
elif command -v apt-get >/dev/null 2>&1; then
|
||||
sudo apt-get install -y -qq chrony
|
||||
elif command -v dnf >/dev/null 2>&1; then
|
||||
sudo dnf install -y -q chrony
|
||||
fi
|
||||
sudo systemctl enable --now chrony 2>/dev/null \
|
||||
|| sudo systemctl enable --now chronyd 2>/dev/null || true
|
||||
sleep 3
|
||||
if command -v chronyc >/dev/null 2>&1; then
|
||||
chronyc tracking | grep -E 'Reference ID|System time|Last offset' || true
|
||||
else
|
||||
timedatectl 2>/dev/null | grep -i synchron || true
|
||||
fi
|
||||
|
||||
say "拉镜像 $IMAGE"
|
||||
docker pull "$IMAGE"
|
||||
docker image inspect "$IMAGE" --format '摘要 {{index .RepoDigests 0}}' 2>/dev/null || true
|
||||
|
||||
say "准备输出目录"
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
|
||||
mkdir -p "$REPO_ROOT/research/out"
|
||||
echo "$REPO_ROOT/research/out"
|
||||
|
||||
say "完成"
|
||||
cat <<'EOF'
|
||||
下一步:
|
||||
|
||||
SHADOW_SITE=aws-tokyo bash research/live/deploy/start.sh
|
||||
|
||||
SHADOW_SITE 必须显式给且两台机器不能相同——它会写进每一行数据,
|
||||
是之后区分数据来源的唯一依据。
|
||||
EOF
|
||||
Executable
+157
@@ -0,0 +1,157 @@
|
||||
#!/usr/bin/env bash
|
||||
# 启动影子采集器。跑之前先 setup.sh。
|
||||
#
|
||||
# SHADOW_SITE=aws-tokyo bash research/live/deploy/start.sh
|
||||
# SHADOW_SITE=aws-tokyo HOURS=168 WORKERS=2 bash research/live/deploy/start.sh
|
||||
#
|
||||
# 为什么 SHADOW_SITE 必填:它写进每一行数据,是两台机器的数据合起来之后
|
||||
# 唯一的来源区分。缺了就只能靠文件路径猜,一合并就分不清了。
|
||||
#
|
||||
# 为什么时钟不同步就拒绝启动:所有延迟数字都是「本地时钟 − 交易所 K 线收盘
|
||||
# 时间戳」。时钟偏 50ms,全部延迟就凭空偏 50ms,而这**不会有任何报错**,
|
||||
# 只会让跨地对比得出一个完全错误的结论。这类静默错误必须在启动就挡掉。
|
||||
set -euo pipefail
|
||||
|
||||
NAME="${NAME:-shadow}"
|
||||
# lean 模式让 TF_DF 只构建到中枢,跳过线段/走势中枢/MACD 状态机。等价性由
|
||||
# verify_lean_parity.py 在影子这条路径上逐根验过(180 窗口 / 90 命中零分歧)。
|
||||
# 设 0 可退回 full,用来复量两模式的耗时差。
|
||||
SHADOW_LEAN="${SHADOW_LEAN:-1}"
|
||||
# 币池。默认三个流动性最好的做滑点测量;十币池是实际要交易的那批(TRX 剔除,
|
||||
# ATR 门控几乎全刷掉)。
|
||||
#
|
||||
# 币数超过核数时排队会成为主项:所有币同一秒收盘。此时**加 worker 没用**
|
||||
# ——CPU 密集的活,worker 超过核数不增吞吐,只会把等待从 queue_ms 挪到
|
||||
# inner_ms。增量路径(SHADOW_INCR=1,默认开)已把清空压到 300~560ms。
|
||||
#
|
||||
# 再往下压的顺序见 HANDOFF §5.72(口径对齐后的实测):单币 inner 100ms ≈
|
||||
# 信号链 30ms + 追加 2.81 根 37ms + 2 核争抢 33ms。争抢只有 1.49x,所以
|
||||
# **加核收益有限**;最便宜的一刀是按币绑定 worker(现在 symbol 随机落
|
||||
# worker,每份缓存都漏掉对方处理过的根,于是人人要追 2.81 根而非 1 根)。
|
||||
SYMS="${SYMS:-BTC,ETH,SOL}"
|
||||
IMAGE="${SHADOW_IMAGE:-hummingbot/hummingbot:latest}"
|
||||
HOURS="${HOURS:-168}"
|
||||
WORKERS="${WORKERS:-2}"
|
||||
MAX_OFFSET_MS="${MAX_OFFSET_MS:-10}"
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
|
||||
OUT="$REPO_ROOT/research/out"
|
||||
# 信号总线的宿主机目录。**刻意放在仓库外**:这是交给实盘执行器(另一台机)的
|
||||
# 交接点,而仓库会被 git checkout/clean 动。执行器那台用 ssh tail 拉这个文件,
|
||||
# 所以它也不能在容器内部,必须挂出来
|
||||
BUS_DIR="${BUS_DIR:-$HOME/chan-live/state}"
|
||||
|
||||
die() { printf '\033[31m错误:%s\033[0m\n' "$*" >&2; exit 1; }
|
||||
say() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
|
||||
|
||||
[[ -n "${SHADOW_SITE:-}" ]] || die "必须设 SHADOW_SITE,例如 SHADOW_SITE=aws-tokyo。
|
||||
它写进每一行数据,是跨地对比时区分来源的唯一依据。"
|
||||
|
||||
say "站点 $SHADOW_SITE"
|
||||
|
||||
mkdir -p "$BUS_DIR"
|
||||
# 容器内是 root,写出来的总线文件宿主机上归 root。执行器那台用普通用户
|
||||
# ssh 过来 tail,所以目录要可进入、文件要可读
|
||||
chmod 755 "$BUS_DIR" 2>/dev/null || true
|
||||
echo "信号总线:$BUS_DIR/signals_live.jsonl(容器内挂成 /bus)"
|
||||
|
||||
say "校验时钟同步"
|
||||
offset_ms=""
|
||||
if command -v chronyc >/dev/null 2>&1; then
|
||||
if ! chronyc tracking >/dev/null 2>&1; then
|
||||
die "chrony 没在跑。先 sudo systemctl start chrony(或 chronyd)"
|
||||
fi
|
||||
# System time 那行形如 "0.000058703 seconds fast of NTP time"
|
||||
line="$(chronyc tracking | grep '^System time' || true)"
|
||||
secs="$(awk '{print $4}' <<<"$line")"
|
||||
offset_ms="$(awk -v s="$secs" 'BEGIN{printf "%.3f", s*1000}')"
|
||||
echo "$line"
|
||||
leap="$(chronyc tracking | awk -F': *' '/Leap status/{print $2}')"
|
||||
[[ "$leap" == "Normal" ]] || die "chrony leap status = $leap,尚未收敛。等几分钟再试"
|
||||
over="$(awk -v o="$offset_ms" -v m="$MAX_OFFSET_MS" 'BEGIN{print (o>m)?1:0}')"
|
||||
[[ "$over" == "0" ]] || die "时钟偏移 ${offset_ms}ms 超过阈值 ${MAX_OFFSET_MS}ms。
|
||||
所有延迟测量都会同向偏这么多且不报错,跨地对比会得出错误结论。
|
||||
先等 chrony 收敛,或调 MAX_OFFSET_MS(不建议)。"
|
||||
echo "偏移 ${offset_ms}ms,在 ${MAX_OFFSET_MS}ms 阈值内"
|
||||
elif command -v timedatectl >/dev/null 2>&1; then
|
||||
timedatectl | grep -qi 'synchronized: yes' \
|
||||
|| die "系统时钟未同步。装 chrony:见 setup.sh"
|
||||
echo "timedatectl 报已同步(无 chronyc,拿不到具体偏移)"
|
||||
else
|
||||
die "既无 chronyc 也无 timedatectl,无法确认时钟。装 chrony 后再启动"
|
||||
fi
|
||||
|
||||
say "检查镜像"
|
||||
docker image inspect "$IMAGE" >/dev/null 2>&1 || die "没有镜像 $IMAGE,先跑 setup.sh"
|
||||
digest="$(docker image inspect "$IMAGE" --format '{{if .RepoDigests}}{{index .RepoDigests 0}}{{end}}' 2>/dev/null || true)"
|
||||
|
||||
say "停掉旧容器"
|
||||
docker rm -f "$NAME" >/dev/null 2>&1 || true
|
||||
|
||||
mkdir -p "$OUT"
|
||||
|
||||
say "量网络距离"
|
||||
netprobe="$(bash "$(dirname "${BASH_SOURCE[0]}")/netprobe.sh" 2>/dev/null || echo '{}')"
|
||||
echo "$netprobe" | grep -E 'edge_org|self_org|self_city|icmp_min_ms|tcp_connect' || true
|
||||
|
||||
# 运行元数据。两地数据对不上时,先看这个文件——镜像摘要、代码版本、时钟偏移
|
||||
# 三者任一不同都足以解释差异,不必去猜。网络探测并在这里,因为「换机房能省
|
||||
# 多少」这个问题只有它能回答(到达延迟里网络只占约 2ms,用它比等于用公斤秤称克)。
|
||||
meta="$OUT/run_meta_${SHADOW_SITE}.json"
|
||||
cat >"$meta" <<EOF
|
||||
{
|
||||
"site": "$SHADOW_SITE",
|
||||
"hostname": "$(hostname)",
|
||||
"started_utc": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
|
||||
"tz": "$(date +%Z%z)",
|
||||
"clock_offset_ms": ${offset_ms:-null},
|
||||
"git_commit": "$(git -C "$REPO_ROOT" rev-parse --short HEAD 2>/dev/null || echo unknown)",
|
||||
"git_dirty": $(git -C "$REPO_ROOT" diff --quiet 2>/dev/null && echo false || echo true),
|
||||
"image": "$IMAGE",
|
||||
"image_digest": "${digest:-unknown}",
|
||||
"hours": $HOURS,
|
||||
"workers": $WORKERS,
|
||||
"kernel": "$(uname -r)",
|
||||
"nproc": $(nproc),
|
||||
"cpu_model": "$(awk -F': ' '/model name/{print $2; exit}' /proc/cpuinfo 2>/dev/null || echo unknown)",
|
||||
"mem_gb": $(free -g | awk '/^Mem:/{print $2}'),
|
||||
"net": $netprobe
|
||||
}
|
||||
EOF
|
||||
echo "元数据已写 $meta"
|
||||
|
||||
say "启动容器 $NAME"
|
||||
docker run -d --name "$NAME" -w /home/hummingbot \
|
||||
--restart unless-stopped \
|
||||
-e PYTHONPATH=/home/hummingbot:/repo/research:/repo/research/live:/repo \
|
||||
-e SHADOW_SITE="$SHADOW_SITE" \
|
||||
-e SHADOW_LEAN="$SHADOW_LEAN" \
|
||||
-e SHADOW_INCR="${SHADOW_INCR:-1}" \
|
||||
-e TG_TOKEN="${TG_TOKEN:-}" \
|
||||
-e TG_CHAT="${TG_CHAT:-}" \
|
||||
-e TG_NOTIONAL="${TG_NOTIONAL:-500}" \
|
||||
-e TG_LEVERAGE="${TG_LEVERAGE:-10}" \
|
||||
-e TG_STALE_S="${TG_STALE_S:-90}" \
|
||||
-e SIGNAL_BUS=/bus/signals_live.jsonl \
|
||||
-v "$REPO_ROOT:/repo:ro" \
|
||||
-v "$OUT:/out" \
|
||||
-v "$BUS_DIR:/bus" \
|
||||
--entrypoint /opt/conda/envs/hummingbot/bin/python \
|
||||
"$IMAGE" /repo/research/live/shadow_hb.py \
|
||||
--hours "$HOURS" --workers "$WORKERS" --syms "$SYMS" >/dev/null
|
||||
|
||||
echo "已启动。等启动自检(补丁断言 + 历史回填,约 60 秒)…"
|
||||
sleep 45
|
||||
docker logs "$NAME" 2>&1 | tail -12
|
||||
|
||||
cat <<EOF
|
||||
|
||||
看日志: docker logs -f $NAME
|
||||
看状态: bash research/live/deploy/status.sh
|
||||
停止: docker rm -f $NAME
|
||||
|
||||
启动日志里应能看到:
|
||||
[补丁] 覆盖生效 —— 换根解析补丁有效(缺了会静默慢 1.06 秒)
|
||||
成交流已挂 [...] —— maker 成交率要用
|
||||
就绪 … 根 —— 历史回填完成
|
||||
EOF
|
||||
Executable
+67
@@ -0,0 +1,67 @@
|
||||
#!/usr/bin/env bash
|
||||
# 采集器健康速查。跨地部署时两台都跑一遍,对着看。
|
||||
set -uo pipefail
|
||||
|
||||
NAME="${NAME:-shadow}"
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
|
||||
OUT="$REPO_ROOT/research/out"
|
||||
|
||||
say() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
|
||||
|
||||
say "容器"
|
||||
docker ps -a --filter "name=^${NAME}$" \
|
||||
--format 'table {{.Names}}\t{{.Status}}\t{{.RunningFor}}' || true
|
||||
|
||||
say "时钟"
|
||||
if command -v chronyc >/dev/null 2>&1; then
|
||||
chronyc tracking | grep -E 'System time|Last offset|Leap status'
|
||||
fi
|
||||
|
||||
say "最近心跳"
|
||||
docker logs "$NAME" 2>&1 | grep '\[心跳\]' | tail -3 || echo "还没到第一次心跳(每 5 分钟一次)"
|
||||
|
||||
say "告警与异常"
|
||||
docker logs "$NAME" 2>&1 \
|
||||
| grep -E '⚠|错误|失效|损坏|停滞|Traceback|退化' | tail -10 \
|
||||
|| echo "无"
|
||||
|
||||
say "输出规模"
|
||||
for f in shadow_latency.csv shadow_drift.csv shadow_signals.csv; do
|
||||
p="$OUT/$f"
|
||||
if [[ -s "$p" ]]; then
|
||||
printf ' %-22s %8d 行\n' "$f" "$(( $(wc -l <"$p") - 1 ))"
|
||||
elif [[ -f "$p" ]]; then
|
||||
# 已建但表头还没冲刷:signals 只在有信号时才 flush,门控后每天仅 4~6 个
|
||||
printf ' %-22s %8s\n' "$f" "0(待首条)"
|
||||
else
|
||||
printf ' %-22s %8s\n' "$f" "无"
|
||||
fi
|
||||
done
|
||||
for f in shadow_books.jsonl.gz shadow_tape.jsonl.gz; do
|
||||
p="$OUT/$f"
|
||||
[[ -f "$p" ]] && printf ' %-22s %8s\n' "$f" "$(du -h "$p" | cut -f1)" \
|
||||
|| printf ' %-22s %8s\n' "$f" "无"
|
||||
done
|
||||
|
||||
say "各站点行数(确认 site 列生效)"
|
||||
p="$OUT/shadow_latency.csv"
|
||||
if [[ -f "$p" ]]; then
|
||||
awk -F, 'NR>1{c[$1]++} END{for(s in c) printf " %-16s %8d 行\n", s, c[s]}' "$p"
|
||||
else
|
||||
echo " 尚无数据"
|
||||
fi
|
||||
|
||||
say "到达延迟中位(本站,跨地对比的主指标)"
|
||||
# 按列名取下标,不写死数字:加了 site 列之后字段整体右移过一次,
|
||||
# 写死 $8 会静默变成读 lag_signal_ms
|
||||
if [[ -s "$p" ]]; then
|
||||
for sym in BTC ETH SOL; do
|
||||
med=$(awk -F, -v s="$sym" '
|
||||
NR==1 { for (i=1;i<=NF;i++) { if ($i=="sym") si=i; if ($i=="lag_data_ms") li=i } ; next }
|
||||
$si==s && $li!="" { print $li }' "$p" | sort -n | awk '
|
||||
{ v[NR]=$1 } END { if (NR) printf "%.0f %d", v[int((NR+1)/2)], NR }')
|
||||
[[ -n "$med" ]] && printf ' %-5s 中位 %6sms (n=%s)\n' "$sym" ${med} \
|
||||
|| printf ' %-5s 尚无数据\n' "$sym"
|
||||
done
|
||||
echo " 参考:本机(新加坡)补丁后实测 350~650ms,理论下限约 500ms"
|
||||
fi
|
||||
@@ -0,0 +1,38 @@
|
||||
# 复制成 tg.env 再填。tg.env 已在 .gitignore 里,不会被提交。
|
||||
#
|
||||
# 拿 token:Telegram 里找 @BotFather → /newbot → 按提示起名
|
||||
# 拿 chat id:给你的 bot 随便发一句,然后打开
|
||||
# https://api.telegram.org/bot<TOKEN>/getUpdates
|
||||
# 返回的 result[0].message.chat.id 就是
|
||||
export TG_TOKEN=""
|
||||
export TG_CHAT=""
|
||||
# 每笔名义额(USDT)。杠杆**不改**手续费与滑点(都按名义额收),所以抬名义额
|
||||
# 是有真实成本的;抬它的唯一理由是压掉步长取整:实测最差币的偏差
|
||||
# 100U → 6.7%(SOL)、500U → 1.8%、1000U → 0.6%。
|
||||
export TG_NOTIONAL="500"
|
||||
# 杠杆只影响占用保证金,不影响名义敞口/手续费/滑点/盈亏绝对值。
|
||||
# 名义 500 在 10x 下占 50 USDT 保证金;止损在 2 ATR ≈ 0.2%,而 10x 强平约需
|
||||
# 逆向 10% = 100 个 ATR,差 50 倍,所以这里的杠杆几乎不引入强平风险。
|
||||
# 交易所侧记得设**逐仓**,让每笔最大损失被保证金封住。
|
||||
export TG_LEVERAGE="10"
|
||||
# 距「参考价成立」超过这么多秒就标为已失效。参考价是次根开盘价,
|
||||
# 过了就不是回测那个成交价了
|
||||
export TG_STALE_S="90"
|
||||
|
||||
# ── 自动化实盘(live_exec.py)──────────────────────────────────
|
||||
# 只读+交易权限,**不要开提币权限**
|
||||
export BITGET_API_KEY=""
|
||||
export BITGET_API_SECRET=""
|
||||
export BITGET_PASSPHRASE=""
|
||||
# 名义额与杠杆。理由同 TG_NOTIONAL:抬名义额是为了压步长取整,不是为了赚更多
|
||||
export LIVE_NOTIONAL="500"
|
||||
export LIVE_LEVERAGE="10"
|
||||
# 硬约束。这三条封住的是「代价不随仓位缩小」的那几类故障:
|
||||
# MAX_OPEN 失控下单(单笔小但笔数无界)
|
||||
# MAX_DAY 同上,日维度
|
||||
# MAX_DAY_LOSS 策略真的不行但没人盯着
|
||||
export LIVE_MAX_OPEN="3"
|
||||
export LIVE_MAX_DAY="15"
|
||||
export LIVE_MAX_DAY_LOSS="20"
|
||||
# 信号超过这么久就不做。参考成交价是次根开盘价,过期后跑的不是回测那个价
|
||||
export LIVE_STALE_S="20"
|
||||
@@ -0,0 +1,116 @@
|
||||
"""把 1030ms 归因到具体环节。
|
||||
|
||||
前面的测量已经排除两个可能:
|
||||
· Bitget 换根首条推送就是 update,Hummingbot 没有因丢弃 snapshot 而等待
|
||||
· 容器内同进程里,Hummingbot 的 feed 只比原始 WS 慢 2~6ms,处理开销可忽略
|
||||
|
||||
剩下的变量是「位置」和「客户端」。本脚本对同一批 K 线三方对齐:
|
||||
|
||||
host_ccxt 宿主机 ccxt.pro watch_ohlcv latency_ccxt.csv
|
||||
host_raw 宿主机 原始 aiohttp WS probe_raw_host.csv
|
||||
cont_raw 容器内 原始 aiohttp WS probe_hb_vs_raw.csv
|
||||
cont_hb 容器内 Hummingbot candles feed probe_hb_vs_raw.csv
|
||||
|
||||
据此可分离两件事:
|
||||
cont_raw − host_raw 容器网络的代价(同一份代码,只换位置)
|
||||
host_ccxt − host_raw ccxt.pro 客户端的差异(同一位置,只换客户端)
|
||||
|
||||
.venv/bin/python research/live/latency_attribute.py
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
OUT = Path(__file__).resolve().parents[1] / "out"
|
||||
SYMS = ("BTC", "ETH", "SOL")
|
||||
|
||||
|
||||
def load() -> pd.DataFrame | None:
|
||||
parts = []
|
||||
|
||||
f = OUT / "latency_ccxt.csv"
|
||||
if f.exists():
|
||||
d = pd.read_csv(f)
|
||||
if not d.empty:
|
||||
parts.append(d[["kline_ts", "sym", "t_data_ms"]]
|
||||
.rename(columns={"t_data_ms": "host_ccxt"}))
|
||||
|
||||
f = OUT / "probe_raw_host.csv"
|
||||
if f.exists():
|
||||
d = pd.read_csv(f)
|
||||
if not d.empty:
|
||||
parts.append(d[["kline_ts", "sym", "raw_ms"]]
|
||||
.rename(columns={"raw_ms": "host_raw"}))
|
||||
|
||||
f = OUT / "probe_hb_vs_raw.csv"
|
||||
if f.exists():
|
||||
d = pd.read_csv(f)
|
||||
if not d.empty:
|
||||
parts.append(d[["kline_ts", "sym", "raw_ms", "hb_ms"]]
|
||||
.rename(columns={"raw_ms": "cont_raw",
|
||||
"hb_ms": "cont_hb"}))
|
||||
|
||||
if not parts:
|
||||
return None
|
||||
m = parts[0]
|
||||
for p in parts[1:]:
|
||||
m = m.merge(p, on=["kline_ts", "sym"], how="outer")
|
||||
return m
|
||||
|
||||
|
||||
def main() -> None:
|
||||
m = load()
|
||||
if m is None or m.empty:
|
||||
print("四路数据均缺失")
|
||||
return
|
||||
|
||||
cols = [c for c in ("host_ccxt", "host_raw", "cont_raw", "cont_hb")
|
||||
if c in m.columns]
|
||||
print(f"各路样本数(重叠前):")
|
||||
for c in cols:
|
||||
print(f" {c}: {int(m[c].notna().sum())}")
|
||||
|
||||
print("\n########## 各路 t_data − t_close(ms,中位)##########")
|
||||
print(f"{'币':<5}" + "".join(f"{c:>11}" for c in cols))
|
||||
for s in SYMS:
|
||||
g = m[m["sym"] == s]
|
||||
line = f"{s:<5}"
|
||||
for c in cols:
|
||||
v = (g[c] - g["kline_ts"]).dropna()
|
||||
line += f"{np.median(v):>11.0f}" if len(v) else f"{'—':>11}"
|
||||
print(line)
|
||||
|
||||
# 只在四路都有的 K 线上做差,避免不同子集的中位数互相错位
|
||||
full = m.dropna(subset=cols)
|
||||
print(f"\n########## 归因(仅四路齐全的 {len(full)} 根)##########")
|
||||
if full.empty:
|
||||
print(" 无四路齐全的 K 线;检查三个采集窗口是否重叠")
|
||||
return
|
||||
pairs = []
|
||||
if "cont_raw" in cols and "host_raw" in cols:
|
||||
pairs.append(("容器网络代价", "cont_raw", "host_raw"))
|
||||
if "host_ccxt" in cols and "host_raw" in cols:
|
||||
pairs.append(("ccxt.pro 客户端差异", "host_ccxt", "host_raw"))
|
||||
if "cont_hb" in cols and "cont_raw" in cols:
|
||||
pairs.append(("Hummingbot 处理开销", "cont_hb", "cont_raw"))
|
||||
if "cont_hb" in cols and "host_ccxt" in cols:
|
||||
pairs.append(("合计:容器 HB vs 宿主 ccxt", "cont_hb", "host_ccxt"))
|
||||
|
||||
print(f"{'环节':<26}" + "".join(f"{s:>9}" for s in SYMS))
|
||||
for name, a, b in pairs:
|
||||
line = f"{name:<26}"
|
||||
for s in SYMS:
|
||||
g = full[full["sym"] == s]
|
||||
if g.empty:
|
||||
line += f"{'—':>9}"
|
||||
continue
|
||||
line += f"{np.median(g[a] - g[b]):>9.0f}"
|
||||
print(line)
|
||||
print("\n(单位 ms,正数表示前者更慢)")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,120 @@
|
||||
"""延迟基准 A:ccxt.pro 侧的数据到手时刻。
|
||||
|
||||
要测的事件只有一个:**我们在什么时候得知第 N 根 1m 已经收盘**。
|
||||
它等价于 t_data − t_close,是下单延迟里最先发生、也往往最大的一段。
|
||||
|
||||
与 latency_hummingbot.py 测的是同一个事件,两边跑同一段时间才可比,
|
||||
所以两个脚本都按整分钟对齐输出,事后按 K 线时间戳 join。
|
||||
|
||||
判据:ETH 的 Bitget 原生滑点余量只有 4.02bp、SOL 2.92bp,而本机 1m 波动
|
||||
ETH 8.6bp/分钟、SOL 9.6bp/分钟。按 √t 折算,1 秒延迟就是 1.1~1.2bp 的
|
||||
随机漂移,且入场方向上还有系统性追价。所以这个数值本身就可能决定生死。
|
||||
|
||||
输出 out/latency_ccxt.csv,每根一行。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import csv
|
||||
import signal
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
RESEARCH = HERE.parent
|
||||
sys.path.insert(0, str(RESEARCH))
|
||||
sys.path.insert(0, str(RESEARCH.parent))
|
||||
|
||||
SYMS = ("BTC", "ETH", "SOL")
|
||||
OUT = RESEARCH / "out" / "latency_ccxt.csv"
|
||||
PERIOD_MS = 60_000
|
||||
|
||||
_stop = False
|
||||
|
||||
|
||||
def _on_signal(*_):
|
||||
global _stop
|
||||
_stop = True
|
||||
|
||||
|
||||
async def watch(ex, sym: str, writer, fh, stats: dict) -> None:
|
||||
"""watch_ohlcv 每次推送都带整段最近 K 线,靠时间戳前进判断上一根已收盘。"""
|
||||
pair = f"{sym}/USDT:USDT"
|
||||
last_ts = None
|
||||
while not _stop:
|
||||
try:
|
||||
o = await ex.watch_ohlcv(pair, "1m")
|
||||
except Exception as e:
|
||||
print(f" {sym} watch 异常 {type(e).__name__}: {e}", flush=True)
|
||||
await asyncio.sleep(1)
|
||||
continue
|
||||
if not o:
|
||||
continue
|
||||
now_ms = int(time.time() * 1000)
|
||||
newest = int(o[-1][0])
|
||||
if last_ts is None:
|
||||
last_ts = newest
|
||||
continue
|
||||
if newest > last_ts:
|
||||
# newest 是刚开始的那根,故 last_ts 那根在 newest 时刻收盘
|
||||
closed_ts = newest
|
||||
lag_ms = now_ms - closed_ts
|
||||
writer.writerow({"kline_ts": closed_ts, "sym": sym,
|
||||
"t_data_ms": now_ms, "lag_ms": lag_ms})
|
||||
fh.flush()
|
||||
stats.setdefault(sym, []).append(lag_ms)
|
||||
n = len(stats[sym])
|
||||
if n % 5 == 1:
|
||||
med = sorted(stats[sym])[n // 2]
|
||||
print(f" {sym}: 第 {n} 根,本次 lag {lag_ms}ms,中位 {med}ms",
|
||||
flush=True)
|
||||
last_ts = newest
|
||||
|
||||
|
||||
async def main_async(minutes: int) -> None:
|
||||
import ccxt.pro as ccxtpro
|
||||
|
||||
ex = ccxtpro.bitget({"options": {"defaultType": "swap"},
|
||||
"enableRateLimit": True})
|
||||
OUT.parent.mkdir(parents=True, exist_ok=True)
|
||||
fh = OUT.open("w", newline="")
|
||||
writer = csv.DictWriter(fh, fieldnames=["kline_ts", "sym", "t_data_ms", "lag_ms"])
|
||||
writer.writeheader()
|
||||
stats: dict = {}
|
||||
|
||||
print(f"[ccxt.pro 延迟] {SYMS} · 计划跑 {minutes} 分钟 · 输出 {OUT.name}",
|
||||
flush=True)
|
||||
tasks = [asyncio.create_task(watch(ex, s, writer, fh, stats)) for s in SYMS]
|
||||
deadline = time.time() + minutes * 60
|
||||
while time.time() < deadline and not _stop:
|
||||
await asyncio.sleep(1)
|
||||
for t in tasks:
|
||||
t.cancel()
|
||||
await asyncio.gather(*tasks, return_exceptions=True)
|
||||
await ex.close()
|
||||
fh.close()
|
||||
|
||||
print("\n########## t_data − t_close(ms)##########")
|
||||
for s in SYMS:
|
||||
v = sorted(stats.get(s, []))
|
||||
if not v:
|
||||
print(f" {s}: 无样本")
|
||||
continue
|
||||
print(f" {s}: n={len(v)} 中位 {v[len(v) // 2]}ms "
|
||||
f"P90 {v[int(len(v) * .9)]}ms 最大 {v[-1]}ms")
|
||||
print(f"\n产物写入 {OUT}")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--minutes", type=int, default=20)
|
||||
args = ap.parse_args()
|
||||
signal.signal(signal.SIGINT, _on_signal)
|
||||
signal.signal(signal.SIGTERM, _on_signal)
|
||||
asyncio.run(main_async(args.minutes))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,125 @@
|
||||
"""把两侧 t_data − t_close 对齐,并折算成 bp,与滑点余量对照。
|
||||
|
||||
为什么要折算成 bp 才有意义:延迟本身不花钱,花钱的是延迟期间价格的漂移。
|
||||
按随机游走,t 秒的价格标准差是 σ_1m · √(t/60),其中 σ_1m 是本币 1m 收益
|
||||
的标准差。入场方向上还有系统性追价(信号触发往往伴随同向动量),所以随机
|
||||
漂移只是下限,真实成本更高——这也是为什么最终仍要用真实盘口测滑点。
|
||||
|
||||
预算从 `lib/shadow_budget` import,不在这里写死。曾经写死的
|
||||
`{BTC: -0.13, ETH: 4.02, SOL: 2.92}` 是错的——那是 `bitget_baseline.py`
|
||||
按「毛均 − 6bp 双边 taker」算的,六处口径叠加(费率档位记高、余量没除
|
||||
taker 名义额、用了 5m~30m 的出场参数、只有同向没有阶梯与 ATR 门控)。
|
||||
正确值是 8.58 / 20.64 / 16.83,ETH 差了五倍。
|
||||
|
||||
.venv/bin/python research/live/latency_compare.py
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
RESEARCH = HERE.parent
|
||||
sys.path.insert(0, str(RESEARCH))
|
||||
|
||||
from lib.shadow_budget import budget_of # noqa: E402
|
||||
|
||||
OUT = RESEARCH / "out"
|
||||
SYMS = ("BTC", "ETH", "SOL")
|
||||
|
||||
|
||||
def vol_bp_per_min() -> dict[str, float]:
|
||||
"""从 Bitget 缓存算 1m 收益标准差,单位 bp。"""
|
||||
out = {}
|
||||
for s in SYMS:
|
||||
f = HERE / "cache" / f"bitget_{s}_1m_30d.feather"
|
||||
if not f.exists():
|
||||
f = HERE / "cache" / f"bitget_{s}_1m_210d.feather"
|
||||
if not f.exists():
|
||||
continue
|
||||
df = pd.read_feather(f)
|
||||
r = np.log(df["close"].to_numpy(dtype=float))
|
||||
out[s] = float(np.nanstd(np.diff(r)) * 1e4)
|
||||
return out
|
||||
|
||||
|
||||
def drift_bp(lag_ms: float, vol: float) -> float:
|
||||
"""随机游走下,lag 毫秒对应的价格漂移标准差(bp)。"""
|
||||
return vol * np.sqrt(max(lag_ms, 0) / 60_000.0)
|
||||
|
||||
|
||||
def load(tag: str) -> pd.DataFrame | None:
|
||||
f = OUT / f"latency_{tag}.csv"
|
||||
if not f.exists():
|
||||
return None
|
||||
df = pd.read_csv(f)
|
||||
return df if not df.empty else None
|
||||
|
||||
|
||||
def describe(df: pd.DataFrame, name: str, vols: dict) -> None:
|
||||
print(f"\n########## {name}:t_data − t_close ##########")
|
||||
print(f"{'币':<5}{'n':>5}{'中位ms':>9}{'P90ms':>9}{'最大ms':>9}"
|
||||
f"{'中位漂移bp':>12}{'余量bp':>9}{'占余量':>9}")
|
||||
for s in SYMS:
|
||||
v = df[df["sym"] == s]["lag_ms"].to_numpy(dtype=float)
|
||||
if not len(v):
|
||||
continue
|
||||
med, p90 = float(np.median(v)), float(np.percentile(v, 90))
|
||||
vol = vols.get(s)
|
||||
d = drift_bp(med, vol) if vol else float("nan")
|
||||
b = budget_of(s)
|
||||
share = f"{d / b * 100:.0f}%" if np.isfinite(b) and b > 0 else "—"
|
||||
print(f"{s:<5}{len(v):>5}{med:>9.0f}{p90:>9.0f}{v.max():>9.0f}"
|
||||
f"{d:>12.2f}{b:>9.2f}{share:>9}")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
vols = vol_bp_per_min()
|
||||
print("1m 收益标准差(bp/分钟,Bitget 缓存实测):")
|
||||
for s, v in vols.items():
|
||||
print(f" {s}: {v:.2f}")
|
||||
|
||||
a, b = load("ccxt"), load("hummingbot")
|
||||
if a is None or b is None:
|
||||
print(f"\n数据未就绪:ccxt={'有' if a is not None else '无'} "
|
||||
f"hummingbot={'有' if b is not None else '无'}")
|
||||
if a is not None:
|
||||
describe(a, "ccxt.pro", vols)
|
||||
if b is not None:
|
||||
describe(b, "Hummingbot", vols)
|
||||
return
|
||||
|
||||
describe(a, "ccxt.pro", vols)
|
||||
describe(b, "Hummingbot", vols)
|
||||
|
||||
# 逐根配对才能消掉「不同分钟市场活跃度不同」的干扰
|
||||
m = a.merge(b, on=["kline_ts", "sym"], suffixes=("_ccxt", "_hb"))
|
||||
print(f"\n########## 逐根配对(重叠 {len(m)} 根)##########")
|
||||
if m.empty:
|
||||
print(" 两侧无重叠 K 线,无法配对;检查采集时间窗是否错开")
|
||||
return
|
||||
print(f"{'币':<5}{'n':>5}{'ccxt中位':>10}{'HB中位':>10}"
|
||||
f"{'差值中位':>10}{'HB更慢占比':>12}{'差值→bp':>10}")
|
||||
for s in SYMS:
|
||||
g = m[m["sym"] == s]
|
||||
if g.empty:
|
||||
continue
|
||||
d = (g["lag_ms_hb"] - g["lag_ms_ccxt"]).to_numpy(dtype=float)
|
||||
vol = vols.get(s)
|
||||
# 差值转 bp:比较两条路径各自漂移的差,而非直接对差值开方
|
||||
extra = (drift_bp(float(np.median(g["lag_ms_hb"])), vol)
|
||||
- drift_bp(float(np.median(g["lag_ms_ccxt"])), vol)) if vol else float("nan")
|
||||
print(f"{s:<5}{len(g):>5}{np.median(g['lag_ms_ccxt']):>10.0f}"
|
||||
f"{np.median(g['lag_ms_hb']):>10.0f}{np.median(d):>10.0f}"
|
||||
f"{(d > 0).mean() * 100:>11.0f}%{extra:>10.2f}")
|
||||
|
||||
print("\n判读:差值 → bp 若显著小于余量,说明选哪个运行时不影响结论,"
|
||||
"可直接用 Hummingbot(生产路径一致);若接近或超过余量,"
|
||||
"则运行时本身就是成本项,需要单独优化或放弃 1m。")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,117 @@
|
||||
"""延迟基准 B:Hummingbot 侧的数据到手时刻。
|
||||
|
||||
在 Hummingbot 容器内运行,测的事件与 latency_ccxt.py 完全相同:
|
||||
**我们在什么时候得知第 N 根 1m 已经收盘**。
|
||||
|
||||
为什么必须在 Hummingbot 里测而不是复用 ccxt 的数字:如果最终执行走
|
||||
Hummingbot,那决定成交价的是它的 WS 处理与事件循环延迟。用别的运行时测出
|
||||
的滑点,换到 Hummingbot 上就不成立了。
|
||||
|
||||
用 max_records=20 让历史回填快速完成——这里只关心增量推送的时刻,不需要
|
||||
2000 根窗口。
|
||||
|
||||
输出 out/latency_hummingbot.csv,字段与 A 侧一致,事后按 kline_ts join。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import csv
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
SYMS = ("BTC", "ETH", "SOL")
|
||||
CONNECTOR = "bitget_perpetual"
|
||||
OUT = Path("/out/latency_hummingbot.csv")
|
||||
|
||||
|
||||
async def main_async(minutes: int) -> None:
|
||||
from hummingbot.data_feed.candles_feed.candles_factory import CandlesFactory
|
||||
from hummingbot.data_feed.candles_feed.data_types import CandlesConfig
|
||||
|
||||
feeds = {}
|
||||
for s in SYMS:
|
||||
cfg = CandlesConfig(connector=CONNECTOR, trading_pair=f"{s}-USDT",
|
||||
interval="1m", max_records=20)
|
||||
feeds[s] = CandlesFactory.get_candle(cfg)
|
||||
|
||||
print(f"[Hummingbot 延迟] {SYMS} · connector={CONNECTOR} · "
|
||||
f"计划跑 {minutes} 分钟", flush=True)
|
||||
for s, f in feeds.items():
|
||||
if hasattr(f, "start"):
|
||||
f.start()
|
||||
else:
|
||||
await f.start_network()
|
||||
print(f" {s} 已启动订阅", flush=True)
|
||||
|
||||
# 等历史回填完成,否则 deque 尾部时间戳还在跳变
|
||||
t0 = time.time()
|
||||
while time.time() - t0 < 120:
|
||||
if all(f.ready for f in feeds.values()):
|
||||
break
|
||||
await asyncio.sleep(0.5)
|
||||
ready = {s: f.ready for s, f in feeds.items()}
|
||||
print(f" 回填状态 {ready}({time.time() - t0:.1f}s)", flush=True)
|
||||
|
||||
OUT.parent.mkdir(parents=True, exist_ok=True)
|
||||
fh = OUT.open("w", newline="")
|
||||
writer = csv.DictWriter(fh, fieldnames=["kline_ts", "sym", "t_data_ms", "lag_ms"])
|
||||
writer.writeheader()
|
||||
|
||||
last = {}
|
||||
for s, f in feeds.items():
|
||||
c = f._candles
|
||||
last[s] = int(c[-1][0]) if len(c) else None
|
||||
|
||||
stats: dict = {}
|
||||
deadline = time.time() + minutes * 60
|
||||
while time.time() < deadline:
|
||||
for s, f in feeds.items():
|
||||
c = f._candles
|
||||
if not len(c):
|
||||
continue
|
||||
newest = int(c[-1][0])
|
||||
if last[s] is None:
|
||||
last[s] = newest
|
||||
continue
|
||||
if newest > last[s]:
|
||||
now_ms = int(time.time() * 1000)
|
||||
# Hummingbot 的时间戳是秒,统一成毫秒后再与本地钟相减
|
||||
closed_ms = newest * 1000 if newest < 1e12 else newest
|
||||
lag_ms = now_ms - closed_ms
|
||||
writer.writerow({"kline_ts": closed_ms, "sym": s,
|
||||
"t_data_ms": now_ms, "lag_ms": lag_ms})
|
||||
fh.flush()
|
||||
stats.setdefault(s, []).append(lag_ms)
|
||||
n = len(stats[s])
|
||||
if n % 5 == 1:
|
||||
med = sorted(stats[s])[n // 2]
|
||||
print(f" {s}: 第 {n} 根,本次 lag {lag_ms}ms,中位 {med}ms",
|
||||
flush=True)
|
||||
last[s] = newest
|
||||
await asyncio.sleep(0.01)
|
||||
|
||||
for f in feeds.values():
|
||||
f.stop()
|
||||
fh.close()
|
||||
|
||||
print("\n########## t_data − t_close(ms)##########")
|
||||
for s in SYMS:
|
||||
v = sorted(stats.get(s, []))
|
||||
if not v:
|
||||
print(f" {s}: 无样本")
|
||||
continue
|
||||
print(f" {s}: n={len(v)} 中位 {v[len(v) // 2]}ms "
|
||||
f"P90 {v[int(len(v) * .9)]}ms 最大 {v[-1]}ms")
|
||||
print(f"\n产物写入 {OUT}")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--minutes", type=int, default=20)
|
||||
args = ap.parse_args()
|
||||
asyncio.run(main_async(args.minutes))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,191 @@
|
||||
"""环境等价性验证:同一份切片,在 .venv 与 Hummingbot 容器里必须算出同一组信号。
|
||||
|
||||
为什么要单独验这件事:容器里是 Python 3.13.14 + pandas 3.0.5 + numpy 2.4.6,
|
||||
本机 .venv 是 Python 3.14.4 + pandas 3.0.5 + numpy 2.5.2。pandas 同版本,
|
||||
numpy 差一个小版本。信号已经被证明对 0.25bp 的数据扰动极度敏感(扰动会换掉
|
||||
一半信号),所以浮点或 groupby 顺序上的任何细微差异都可能改变信号集合——
|
||||
必须实测,不能推断。
|
||||
|
||||
用 --dump 先从 .venv 导出切片成 CSV,两个环境再读同一个 CSV,
|
||||
这样数据来源差异为零,比出来的就是纯计算差异。
|
||||
|
||||
.venv/bin/python research/live/parity_env.py --dump # 导出切片
|
||||
.venv/bin/python research/live/parity_env.py --run # 本机计算
|
||||
docker run ... /app/parity_env.py --run --tag container # 容器计算
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
|
||||
os.environ.setdefault(v, "1")
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
RESEARCH = HERE.parent
|
||||
sys.path.insert(0, str(RESEARCH))
|
||||
sys.path.insert(0, str(RESEARCH.parent))
|
||||
|
||||
LTF, HTF = "1m", "5m"
|
||||
WINDOW = 2000 # step39 定下的窗口:命中率在此饱和
|
||||
HTF_WINDOW = 800
|
||||
SYMS = ("BTC", "ETH", "SOL")
|
||||
|
||||
|
||||
def slice_dir() -> Path:
|
||||
"""容器里挂在 /out,本机是 research/out。"""
|
||||
p = Path("/out")
|
||||
return p if p.is_dir() else RESEARCH / "out"
|
||||
|
||||
|
||||
def dump() -> None:
|
||||
"""从 Bitget 缓存导出末尾切片,供两个环境共用。
|
||||
|
||||
用 Bitget 而非 Binance 的数据:目标场地就是 Bitget,且这批缓存正是
|
||||
bitget_baseline.py 算余量时用的同一份,口径可直接对上。
|
||||
"""
|
||||
d = slice_dir() / "parity_slices"
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
cache = HERE / "cache"
|
||||
for s in SYMS:
|
||||
for tf, n in ((LTF, WINDOW), (HTF, HTF_WINDOW)):
|
||||
src = cache / f"bitget_{s}_{tf}_30d.feather"
|
||||
if not src.exists():
|
||||
src = cache / f"bitget_{s}_{tf}_210d.feather"
|
||||
if not src.exists():
|
||||
print(f" {s} {tf}: 无缓存 {src.name},跳过")
|
||||
continue
|
||||
df = pd.read_feather(src)
|
||||
if df is None or df.empty:
|
||||
print(f" {s} {tf}: 缓存为空,跳过")
|
||||
continue
|
||||
out = df.tail(n).reset_index(drop=True)
|
||||
f = d / f"{s}_{tf}.csv"
|
||||
# 用 float 全精度写出,避免导出环节就引入舍入差异
|
||||
out.to_csv(f, index=False, float_format="%.10f")
|
||||
print(f" {s} {tf}: {len(out)} 根 -> {f.name}")
|
||||
print(f"\n切片目录 {d}")
|
||||
|
||||
|
||||
def signals(df_ltf: pd.DataFrame, df_htf: pd.DataFrame) -> dict:
|
||||
"""复用 step39 的时点重建口径,返回信号下标与耗时。"""
|
||||
from chanlun import TF_DF
|
||||
from lib.fast_bsp3 import find_fast_bsp3
|
||||
from lib.nested_level import build_htf_zones
|
||||
|
||||
t0 = time.perf_counter()
|
||||
chan = TF_DF(df_ltf, 1, LTF)
|
||||
cdf = chan.dataframe
|
||||
zones = build_htf_zones(cdf, LTF, chan=chan)
|
||||
raw: list[int] = []
|
||||
if not zones.empty:
|
||||
sig = find_fast_bsp3(cdf, zones.reset_index(drop=True))
|
||||
if sig is not None and not sig.empty:
|
||||
col = "idx" if "idx" in sig.columns else sig.columns[0]
|
||||
raw = sorted(int(x) for x in sig[col].to_numpy())
|
||||
dt = time.perf_counter() - t0
|
||||
|
||||
# 中枢边界是信号定义的核心中间量,一并指纹化:
|
||||
# 若信号相同但中枢不同,说明差异只是被过滤掉了,仍是隐患
|
||||
zsig = None
|
||||
if not zones.empty:
|
||||
num = zones.select_dtypes(include=[np.number])
|
||||
zsig = float(np.nansum(num.to_numpy(dtype=float)))
|
||||
|
||||
return {"n_bars": int(len(df_ltf)), "n_signals": len(raw),
|
||||
"signal_idx": raw, "zone_checksum": zsig,
|
||||
"n_zones": int(len(zones)), "compute_s": round(dt, 4)}
|
||||
|
||||
|
||||
def run(tag: str) -> None:
|
||||
d = slice_dir() / "parity_slices"
|
||||
res = {"tag": tag,
|
||||
"python": sys.version.split()[0],
|
||||
"pandas": pd.__version__,
|
||||
"numpy": np.__version__}
|
||||
per = {}
|
||||
for s in SYMS:
|
||||
f_ltf, f_htf = d / f"{s}_{LTF}.csv", d / f"{s}_{HTF}.csv"
|
||||
if not f_ltf.exists():
|
||||
print(f" {s}: 缺切片 {f_ltf}")
|
||||
continue
|
||||
df_ltf = pd.read_csv(f_ltf)
|
||||
df_htf = pd.read_csv(f_htf) if f_htf.exists() else pd.DataFrame()
|
||||
# date 存的是时间戳字符串,chanlun 的 kline builder 要真 datetime;
|
||||
# timestamp 是毫秒整数,保持数值不动
|
||||
for df in (df_ltf, df_htf):
|
||||
if not df.empty and "date" in df.columns:
|
||||
df["date"] = pd.to_datetime(df["date"], utc=True)
|
||||
r = signals(df_ltf, df_htf)
|
||||
per[s] = r
|
||||
print(f" {s}: {r['n_signals']} 信号 · {r['n_zones']} 中枢 · "
|
||||
f"{r['compute_s']}s · zone_checksum={r['zone_checksum']}")
|
||||
res["per_symbol"] = per
|
||||
|
||||
out = slice_dir() / f"parity_env_{tag}.json"
|
||||
out.write_text(json.dumps(res, indent=2, ensure_ascii=False))
|
||||
print(f"\n[{tag}] python {res['python']} pandas {res['pandas']} "
|
||||
f"numpy {res['numpy']}")
|
||||
print(f"产物写入 {out}")
|
||||
|
||||
|
||||
def compare(a: str, b: str) -> None:
|
||||
d = slice_dir()
|
||||
ra = json.loads((d / f"parity_env_{a}.json").read_text())
|
||||
rb = json.loads((d / f"parity_env_{b}.json").read_text())
|
||||
print(f"{a}: python {ra['python']} pandas {ra['pandas']} numpy {ra['numpy']}")
|
||||
print(f"{b}: python {rb['python']} pandas {rb['pandas']} numpy {rb['numpy']}")
|
||||
print("\n########## 信号集合是否逐一相同 ##########")
|
||||
ok = True
|
||||
for s in SYMS:
|
||||
pa, pb = ra["per_symbol"].get(s), rb["per_symbol"].get(s)
|
||||
if not pa or not pb:
|
||||
print(f" {s}: 缺结果,跳过")
|
||||
continue
|
||||
same_sig = pa["signal_idx"] == pb["signal_idx"]
|
||||
same_zone = pa["n_zones"] == pb["n_zones"]
|
||||
# checksum 是浮点求和,允许相对 1e-9 的差;超出即为真实分歧
|
||||
za, zb = pa["zone_checksum"], pb["zone_checksum"]
|
||||
same_cs = (za is None and zb is None) or (
|
||||
za is not None and zb is not None
|
||||
and abs(za - zb) <= 1e-9 * max(1.0, abs(za)))
|
||||
ok = ok and same_sig and same_zone and same_cs
|
||||
print(f" {s}: 信号 {'一致' if same_sig else '不一致'}"
|
||||
f"({pa['n_signals']} vs {pb['n_signals']})· "
|
||||
f"中枢 {'一致' if same_zone else '不一致'}"
|
||||
f"({pa['n_zones']} vs {pb['n_zones']})· "
|
||||
f"checksum {'一致' if same_cs else '不一致'}")
|
||||
if not same_sig:
|
||||
sa, sb = set(pa["signal_idx"]), set(pb["signal_idx"])
|
||||
print(f" 仅 {a} 有: {sorted(sa - sb)[:10]}")
|
||||
print(f" 仅 {b} 有: {sorted(sb - sa)[:10]}")
|
||||
print(f" 耗时 {pa['compute_s']}s vs {pb['compute_s']}s")
|
||||
print(f"\n结论:{'两环境等价,可直接在容器内算信号' if ok else '存在分歧,需把信号计算固定在单一环境'}")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--dump", action="store_true")
|
||||
ap.add_argument("--run", action="store_true")
|
||||
ap.add_argument("--tag", default="venv")
|
||||
ap.add_argument("--compare", nargs=2, metavar=("A", "B"))
|
||||
args = ap.parse_args()
|
||||
if args.dump:
|
||||
dump()
|
||||
if args.run:
|
||||
run(args.tag)
|
||||
if args.compare:
|
||||
compare(*args.compare)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,126 @@
|
||||
"""修正 Hummingbot bitget_perpetual candles feed 的换根延迟。
|
||||
|
||||
上游 _parse_websocket_message 里是:
|
||||
|
||||
candle = data["data"][0]
|
||||
|
||||
而 Bitget 在换根时会推一条带两根的消息 [上一根, 新一根]。取 [0] 拿到的是
|
||||
上一根,其时间戳与 deque 尾部相同,于是只做了原地更新;新一根要等下一条
|
||||
单元素消息才进入 deque——实测晚约 1.1 秒。
|
||||
|
||||
不能简单改成 [-1]:那样上一根的收盘价就永远停在换根前约 1 秒的那次推送上。
|
||||
1m 信号对 0.25bp 的扰动都会换掉一半(见 signal_sensitivity.py),收盘价
|
||||
偏一个 tick 是不能接受的。所以这里把**除最后一根外的元素就地写回 deque**,
|
||||
再把最后一根交给基类走正常的 append 流程。
|
||||
|
||||
已向上游反馈前,本地用子类覆盖,不改动镜像。
|
||||
|
||||
## 为什么必须有启动断言
|
||||
|
||||
子类覆盖的失效方式是**静默**的:上游若把 `_parse_websocket_message` 改名、
|
||||
或改走别的钩子,我们的覆盖就成了死代码,行情悄悄退回慢 1.06 秒,不崩、
|
||||
不报错、不留日志,只会让收益慢慢变差,几周后才从统计里看出来。
|
||||
|
||||
`assert_patch_effective()` 不做名字检查——名字对不上未必失效,名字对得上
|
||||
也未必生效。它喂一条合成的两元素消息,直接验证行为:基类返回首元素(bug
|
||||
仍在、覆盖仍有必要),子类返回末元素(覆盖确实生效)。再加一条源码检查
|
||||
确认基类的收包循环还在调这个钩子。任一不满足就在启动时抛错。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import inspect
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
import numpy as np
|
||||
|
||||
from hummingbot.data_feed.candles_feed.bitget_perpetual_candles import (
|
||||
BitgetPerpetualCandles,
|
||||
)
|
||||
from hummingbot.data_feed.candles_feed.candles_base import CandlesBase
|
||||
|
||||
|
||||
def _row_to_dict(row: list, ensure_s) -> Dict[str, Any]:
|
||||
return {"timestamp": ensure_s(int(row[0])),
|
||||
"open": float(row[1]), "high": float(row[2]),
|
||||
"low": float(row[3]), "close": float(row[4]),
|
||||
"volume": float(row[5]), "quote_asset_volume": float(row[6]),
|
||||
"n_trades": 0., "taker_buy_base_volume": 0.,
|
||||
"taker_buy_quote_volume": 0.}
|
||||
|
||||
|
||||
class PatchedBitgetPerpetualCandles(BitgetPerpetualCandles):
|
||||
"""与上游唯一的差别:一条消息里的多根 K 线全部处理,而非只取第一根。"""
|
||||
|
||||
def _parse_websocket_message(self, data: dict) -> Optional[Dict[str, Any]]:
|
||||
if data == "pong":
|
||||
return None
|
||||
if not (data and data.get("data") and data.get("action") == "update"):
|
||||
return None
|
||||
|
||||
rows = data["data"]
|
||||
# 前面的元素都是已收盘 K 线的最终值:就地覆盖,保住真实收盘价
|
||||
for row in rows[:-1]:
|
||||
d = _row_to_dict(row, self.ensure_timestamp_in_seconds)
|
||||
self._overwrite_existing(d)
|
||||
# 最后一根交给基类:时间戳更大就 append,相同就原地更新
|
||||
return _row_to_dict(rows[-1], self.ensure_timestamp_in_seconds)
|
||||
|
||||
def _overwrite_existing(self, d: Dict[str, Any]) -> None:
|
||||
if not len(self._candles):
|
||||
return
|
||||
ts = int(d["timestamp"])
|
||||
if int(self._candles[-1][0]) != ts:
|
||||
return
|
||||
self._candles[-1] = np.array(
|
||||
[d["timestamp"], d["open"], d["high"], d["low"], d["close"],
|
||||
d["volume"], d["quote_asset_volume"], d["n_trades"],
|
||||
d["taker_buy_base_volume"], d["taker_buy_quote_volume"]]
|
||||
).astype(float)
|
||||
|
||||
|
||||
# 换根时 Bitget 推的就是这个形状:[上一根, 新一根]
|
||||
_PROBE = {
|
||||
"action": "update",
|
||||
"arg": {"instType": "USDT-FUTURES", "channel": "candle1m",
|
||||
"instId": "BTCUSDT"},
|
||||
"data": [
|
||||
["1700000040000", "1", "1", "1", "1", "1", "1", "1"],
|
||||
["1700000100000", "2", "2", "2", "2", "2", "2", "2"],
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def assert_patch_effective() -> None:
|
||||
"""启动即验证覆盖真的生效,否则抛错。让静默失效变成启动失败。"""
|
||||
src = inspect.getsource(CandlesBase._process_websocket_messages_task)
|
||||
if "_parse_websocket_message" not in src:
|
||||
raise RuntimeError(
|
||||
"上游收包循环已不再调用 _parse_websocket_message,"
|
||||
"patched_candles 的覆盖失效。需重新定位钩子后再启动。")
|
||||
|
||||
stock = BitgetPerpetualCandles("BTC-USDT", "1m", 20)
|
||||
ours = PatchedBitgetPerpetualCandles("BTC-USDT", "1m", 20)
|
||||
got_stock = stock._parse_websocket_message(_PROBE)
|
||||
got_ours = ours._parse_websocket_message(_PROBE)
|
||||
|
||||
head_ts = stock.ensure_timestamp_in_seconds(int(_PROBE["data"][0][0]))
|
||||
tail_ts = stock.ensure_timestamp_in_seconds(int(_PROBE["data"][-1][0]))
|
||||
|
||||
if not got_ours or int(got_ours["timestamp"]) != int(tail_ts):
|
||||
raise RuntimeError(
|
||||
f"覆盖未生效:子类返回 {got_ours and got_ours.get('timestamp')},"
|
||||
f"应为末元素 {tail_ts}。")
|
||||
if got_stock and int(got_stock["timestamp"]) == int(tail_ts):
|
||||
# 上游自己修好了。此时覆盖无害但已多余,明确说出来,免得以后
|
||||
# 有人以为那 1.06 秒还是靠我们拿回来的
|
||||
print(" [补丁] 上游已自行修正换根解析,本地覆盖现为冗余,可移除",
|
||||
flush=True)
|
||||
elif not got_stock or int(got_stock["timestamp"]) != int(head_ts):
|
||||
raise RuntimeError(
|
||||
f"基类行为与预期不符:返回 "
|
||||
f"{got_stock and got_stock.get('timestamp')},"
|
||||
f"既非首元素 {head_ts} 也非末元素 {tail_ts}。"
|
||||
f"上游改了解析逻辑,补丁的前提需重新确认。")
|
||||
else:
|
||||
print(f" [补丁] 覆盖生效:基类取首元素 {int(head_ts)}、"
|
||||
f"本地取末元素 {int(tail_ts)}", flush=True)
|
||||
@@ -0,0 +1,152 @@
|
||||
"""止盈位被首次触及那一根,价格穿透了多深。
|
||||
|
||||
为什么要这个数:影子成交流显示,限价单若正好落在某根的最高价,该价位之上
|
||||
的主动买成交额只有几十到几千美元——对十万量级的仓位等于不成交。但那是
|
||||
最坏情形。真实成交率取决于**止盈位被穿透了多深**:若价格一路冲过目标,
|
||||
成交没问题;若只是上影线点一下就回落,就成交不了。
|
||||
|
||||
这个分布不需要再采数据,历史 K 线里就有:给定入场价与 ATR,目标位是
|
||||
`entry + T×ATR`,找到首次 `high ≥ target` 的那根,穿透深度就是
|
||||
`high − target`。把它折成「占该根价格区间的比例」,就能直接对上成交流
|
||||
那条「≥ 限价的成交额 vs 限价在区间中的位置」曲线。
|
||||
|
||||
口径与 lib/exit_model.walk_exits 对齐:入场取信号次根开盘价,ATR 取信号
|
||||
根的 Wilder ATR-14,上限 48 根。
|
||||
|
||||
**取样方式的局限**:这里用全体 K 线做候选入场点,而非真实的三滤网信号。
|
||||
真实信号是按结构条件挑出来的,入场时刻可能与波动率状态相关。以 ATR 归一
|
||||
后的穿透深度对波动率状态应当不敏感,但要精确到信号级别,得重跑一次
|
||||
step42 的缠论链路(单币约 370s、峰值 24.5GB)。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
SCALE_AT = 3.0 # 分批减仓位
|
||||
RUNNER = 8.0 # 剩余半仓目标
|
||||
MAX_BARS = 48
|
||||
|
||||
|
||||
def wilder_atr(high, low, close, period: int = 14) -> np.ndarray:
|
||||
"""与 chanlun.indicators.ta.ATR 逐位一致的 Wilder ATR。"""
|
||||
n = high.size
|
||||
out = np.full(n, np.nan)
|
||||
if n <= period:
|
||||
return out
|
||||
prev_close = close[:-1]
|
||||
tr = np.maximum.reduce([high[1:] - low[1:],
|
||||
np.abs(high[1:] - prev_close),
|
||||
np.abs(low[1:] - prev_close)])
|
||||
sm = np.empty(tr.size)
|
||||
sm[period - 1] = tr[:period].mean()
|
||||
a = 1.0 / period
|
||||
for k in range(period, tr.size):
|
||||
sm[k] = sm[k - 1] + a * (tr[k] - sm[k - 1])
|
||||
out[period:] = sm[period - 1:]
|
||||
return out
|
||||
|
||||
|
||||
def penetration(df: pd.DataFrame, target_atr: float,
|
||||
stride: int = 1) -> pd.DataFrame:
|
||||
"""对每个候选入场点,求首次触及 `target_atr` 时的穿透深度。
|
||||
|
||||
只统计**触及了**的那些(未触及的属止损或超时出场,不涉及 maker 腿)。
|
||||
"""
|
||||
high = df["high"].to_numpy(float)
|
||||
low = df["low"].to_numpy(float)
|
||||
close = df["close"].to_numpy(float)
|
||||
open_ = df["open"].to_numpy(float)
|
||||
atr = wilder_atr(high, low, close)
|
||||
n = len(df)
|
||||
rows = []
|
||||
for i in range(20, n - MAX_BARS - 2, stride):
|
||||
a = atr[i]
|
||||
if not np.isfinite(a) or a <= 0:
|
||||
continue
|
||||
e = i + 1
|
||||
entry = open_[e]
|
||||
# 多头:目标在上方。空头对称,穿透深度分布按对称性等价,故只算一边
|
||||
target = entry + target_atr * a
|
||||
end = e + MAX_BARS
|
||||
seg_hi = high[e:end + 1]
|
||||
hit = np.flatnonzero(seg_hi >= target)
|
||||
if not hit.size:
|
||||
continue
|
||||
j = e + int(hit[0])
|
||||
rng = high[j] - low[j]
|
||||
if rng <= 0:
|
||||
continue
|
||||
pen = high[j] - target
|
||||
rows.append({
|
||||
"bar": j,
|
||||
# 限价在该根价格区间中的位置:0 = 正好在最高价(最坏),
|
||||
# 1 = 在最低价(该根全部成交都在限价之上)
|
||||
"k": min(1.0, pen / rng),
|
||||
"pen_bp": pen / target * 1e4,
|
||||
"pen_atr": pen / a,
|
||||
"range_bp": rng / target * 1e4,
|
||||
})
|
||||
return pd.DataFrame(rows)
|
||||
|
||||
|
||||
def report(sym: str, df: pd.DataFrame) -> dict:
|
||||
print(f"\n{'=' * 68}\n{sym} 共 {len(df):,} 根 1m")
|
||||
out = {}
|
||||
for tgt, name in ((SCALE_AT, f"减仓位 {SCALE_AT:g}ATR"),
|
||||
(RUNNER, f"目标位 {RUNNER:g}ATR")):
|
||||
p = penetration(df, tgt)
|
||||
if p.empty:
|
||||
print(f" {name}: 无触及样本")
|
||||
continue
|
||||
k = p["k"].to_numpy()
|
||||
print(f"\n {name} · 触及 {len(p):,} 次")
|
||||
print(f" 穿透深度 中位 {p['pen_bp'].median():.2f}bp "
|
||||
f"({p['pen_atr'].median():.2f} ATR) · "
|
||||
f"P25 {p['pen_bp'].quantile(.25):.2f}bp · "
|
||||
f"P75 {p['pen_bp'].quantile(.75):.2f}bp")
|
||||
print(f" 限价在区间中的位置 k(0=正好在最高价,越大越靠下越易成交)")
|
||||
print(f" 中位 {np.median(k):.3f} · P10 {np.percentile(k, 10):.3f}"
|
||||
f" · P25 {np.percentile(k, 25):.3f}"
|
||||
f" · P75 {np.percentile(k, 75):.3f}")
|
||||
for thr in (0.05, 0.10, 0.25, 0.50):
|
||||
print(f" k ≤ {thr:.2f}(限价挤在该根顶部 {thr * 100:.0f}% 内):"
|
||||
f"{float((k <= thr).mean()) * 100:5.1f}% 的触及")
|
||||
out[tgt] = p
|
||||
return out
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--syms", default="BTC,ETH,SOL")
|
||||
ap.add_argument("--cache", default="research/live/cache")
|
||||
ap.add_argument("--save", default="research/out/penetration.csv")
|
||||
a = ap.parse_args()
|
||||
|
||||
root = Path(a.cache)
|
||||
allp = []
|
||||
for sym in a.syms.split(","):
|
||||
cands = sorted(root.glob(f"bitget_{sym}_1m_*.feather"),
|
||||
key=lambda p: p.stat().st_size, reverse=True)
|
||||
if not cands:
|
||||
print(f"{sym}: 找不到 1m 缓存,跳过")
|
||||
continue
|
||||
df = pd.read_feather(cands[0])
|
||||
got = report(sym, df)
|
||||
for tgt, p in got.items():
|
||||
p = p.copy()
|
||||
p["sym"], p["target_atr"] = sym, tgt
|
||||
allp.append(p)
|
||||
if allp:
|
||||
out = pd.concat(allp, ignore_index=True)
|
||||
Path(a.save).parent.mkdir(parents=True, exist_ok=True)
|
||||
out.to_csv(a.save, index=False)
|
||||
print(f"\n已存 {a.save}({len(out):,} 行),"
|
||||
f"供 shadow_depth.py 合并成交量曲线")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,68 @@
|
||||
"""真实 Bitget 盘口在 BOOK_DEPTH 档内能不能吃下各个名义额档位。
|
||||
|
||||
shadow_hb 把吃单换成了框架的 get_vwap_for_volume,深度不足时它返回 nan、
|
||||
整行标 depth_ok=0。所以「档数够不够」直接决定某个仓位档会不会整段丢失,
|
||||
不是个可以事后补救的参数——先量出来再定 BOOK_DEPTH。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
|
||||
import numpy as np
|
||||
|
||||
sys.path.insert(0, "/repo/research/live")
|
||||
|
||||
from shadow_hb import BOOK_DEPTH, NOTIONALS, SYMS, book_from
|
||||
|
||||
|
||||
async def run() -> None:
|
||||
from hummingbot.connector.derivative.bitget_perpetual.bitget_perpetual_derivative import (
|
||||
BitgetPerpetualDerivative,
|
||||
)
|
||||
conn = BitgetPerpetualDerivative(
|
||||
bitget_perpetual_api_key="", bitget_perpetual_secret_key="",
|
||||
bitget_perpetual_passphrase="",
|
||||
trading_pairs=[f"{s}-USDT" for s in SYMS], trading_required=False)
|
||||
await conn.start_network()
|
||||
print(f"连接器已启动,等盘口(档数上限 {BOOK_DEPTH})")
|
||||
for _ in range(60):
|
||||
await asyncio.sleep(1)
|
||||
try:
|
||||
if all(conn.get_order_book(f"{s}-USDT") is not None for s in SYMS):
|
||||
break
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
for s in SYMS:
|
||||
ob = conn.get_order_book(f"{s}-USDT")
|
||||
bids = np.array([(float(r.price), float(r.amount), i)
|
||||
for i, (r, _) in enumerate(
|
||||
zip(ob.bid_entries(), range(BOOK_DEPTH)))])
|
||||
asks = np.array([(float(r.price), float(r.amount), i)
|
||||
for i, (r, _) in enumerate(
|
||||
zip(ob.ask_entries(), range(BOOK_DEPTH)))])
|
||||
mid = (bids[0][0] + asks[0][0]) / 2.0
|
||||
snap = book_from(bids, asks)
|
||||
ask_notional = float((asks[:, 0] * asks[:, 1]).sum())
|
||||
print(f"\n{s} 中价 {mid:.2f} · 取到 {len(asks)} 档 · "
|
||||
f"卖盘 {len(asks)} 档合计 {ask_notional:,.0f} USDT")
|
||||
print(f" 最深一档距中价 "
|
||||
f"{(asks[-1][0] / mid - 1) * 1e4:.1f}bp")
|
||||
for notional in NOTIONALS:
|
||||
base = notional / mid
|
||||
r = snap.get_vwap_for_volume(True, base)
|
||||
px = float(r.result_price)
|
||||
ok = float(r.result_volume) >= base * 0.999
|
||||
if ok:
|
||||
print(f" 名义 {notional:>7,.0f} → {base:.6f} 币 · "
|
||||
f"冲击 {(px / mid - 1) * 1e4:6.2f}bp · 吃得下")
|
||||
else:
|
||||
print(f" 名义 {notional:>7,.0f} → {base:.6f} 币 · "
|
||||
f"深度不足,仅 {r.result_volume:.6f} 币 · "
|
||||
f"这一档会整段丢失")
|
||||
await conn.stop_network()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(run())
|
||||
@@ -0,0 +1,186 @@
|
||||
"""定位 Hummingbot 那 1030ms 究竟出在处理链路还是容器网络。
|
||||
|
||||
前一轮对照有两个变量同时在变:运行时(Hummingbot vs ccxt.pro)和位置
|
||||
(容器内 vs 宿主机)。所以 1030ms 无法归因。原始 WS 探针已否掉「丢弃
|
||||
snapshot」这个猜测——换根首条推送就是 update,Hummingbot 确实收到了。
|
||||
|
||||
本脚本在**容器内同一个进程**里并行跑两条路径,共用一个时钟、一条网络:
|
||||
A. Hummingbot 的 BitgetPerpetualCandles,轮询 _candles 尾部时间戳
|
||||
B. 一条原始 aiohttp WS,直接订阅 candle1m
|
||||
|
||||
两条路径对同一根 K 线各记一个到达时刻,差值即 Hummingbot 处理链路的净开销。
|
||||
若 B 也慢,则是容器网络,与框架无关。
|
||||
|
||||
在容器内运行:
|
||||
docker run --rm -v $PWD:/repo:ro -v $PWD/research/out:/out \
|
||||
-w /home/hummingbot -e PYTHONPATH=/home/hummingbot \
|
||||
--entrypoint /opt/conda/envs/hummingbot/bin/python \
|
||||
hummingbot/hummingbot:latest /repo/research/live/probe_hb_vs_raw.py --minutes 20
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import csv
|
||||
import json
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
SYMS = ("BTC", "ETH", "SOL")
|
||||
WSS = "wss://ws.bitget.com/v2/ws/public"
|
||||
|
||||
|
||||
def out_dir() -> Path:
|
||||
p = Path("/out")
|
||||
return p if p.is_dir() else Path(__file__).resolve().parents[1] / "out"
|
||||
|
||||
|
||||
async def raw_ws(rec: dict, stop: asyncio.Event) -> None:
|
||||
"""B 路径:原始 WS,收到第一条带新时间戳的推送就记时刻。"""
|
||||
import aiohttp
|
||||
|
||||
payload = {"op": "subscribe",
|
||||
"args": [{"instType": "USDT-FUTURES", "channel": "candle1m",
|
||||
"instId": f"{s}USDT"} for s in SYMS]}
|
||||
while not stop.is_set():
|
||||
try:
|
||||
async with aiohttp.ClientSession() as sess, \
|
||||
sess.ws_connect(WSS, heartbeat=20) as ws:
|
||||
await ws.send_str(json.dumps(payload))
|
||||
print(" [raw] 已订阅", flush=True)
|
||||
last: dict[str, int] = {}
|
||||
while not stop.is_set():
|
||||
msg = await ws.receive()
|
||||
if msg.type is not aiohttp.WSMsgType.TEXT:
|
||||
# 关闭类消息必须跳出重连,否则 receive() 会立刻返回造成空转
|
||||
print(f" [raw] 非文本消息 {msg.type},重连", flush=True)
|
||||
break
|
||||
if msg.data == "pong":
|
||||
continue
|
||||
d = json.loads(msg.data)
|
||||
if "data" not in d or "arg" not in d:
|
||||
continue
|
||||
sym = d["arg"]["instId"].replace("USDT", "")
|
||||
kts = int(d["data"][0][0])
|
||||
if last.get(sym) is not None and kts > last[sym]:
|
||||
rec.setdefault((sym, kts), {})["raw_ms"] = \
|
||||
int(time.time() * 1000)
|
||||
last[sym] = max(kts, last.get(sym, 0))
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as e:
|
||||
print(f" [raw] 异常 {type(e).__name__}: {e}", flush=True)
|
||||
# 无论是正常跳出还是异常,重连前都歇一下,避免服务端持续拒绝时打成风暴
|
||||
if not stop.is_set():
|
||||
await asyncio.sleep(1)
|
||||
|
||||
|
||||
async def hb_feed(rec: dict, stop: asyncio.Event) -> None:
|
||||
"""A 路径:Hummingbot candles feed,10ms 轮询 deque 尾部。"""
|
||||
from hummingbot.data_feed.candles_feed.candles_factory import CandlesFactory
|
||||
from hummingbot.data_feed.candles_feed.data_types import CandlesConfig
|
||||
|
||||
feeds = {}
|
||||
for s in SYMS:
|
||||
cfg = CandlesConfig(connector="bitget_perpetual",
|
||||
trading_pair=f"{s}-USDT", interval="1m",
|
||||
max_records=20)
|
||||
f = CandlesFactory.get_candle(cfg)
|
||||
f.start()
|
||||
feeds[s] = f
|
||||
print(" [hb] 已订阅", flush=True)
|
||||
|
||||
t0 = time.time()
|
||||
while time.time() - t0 < 120 and not all(f.ready for f in feeds.values()):
|
||||
await asyncio.sleep(0.5)
|
||||
print(f" [hb] 回填完成 {time.time() - t0:.1f}s", flush=True)
|
||||
|
||||
last = {s: (int(f._candles[-1][0]) if len(f._candles) else None)
|
||||
for s, f in feeds.items()}
|
||||
while not stop.is_set():
|
||||
for s, f in feeds.items():
|
||||
if not len(f._candles):
|
||||
continue
|
||||
newest = int(f._candles[-1][0])
|
||||
if last[s] is not None and newest > last[s]:
|
||||
# HB 的时间戳是秒,统一成毫秒好与原始 WS 的 kline_ts 对齐
|
||||
kts = newest * 1000 if newest < 1e12 else newest
|
||||
rec.setdefault((s, kts), {})["hb_ms"] = int(time.time() * 1000)
|
||||
last[s] = newest
|
||||
await asyncio.sleep(0.01)
|
||||
for f in feeds.values():
|
||||
f.stop()
|
||||
|
||||
|
||||
async def main_async(minutes: int, raw_only: bool = False) -> None:
|
||||
rec: dict = {}
|
||||
stop = asyncio.Event()
|
||||
tasks = [asyncio.create_task(raw_ws(rec, stop))]
|
||||
if not raw_only:
|
||||
tasks.append(asyncio.create_task(hb_feed(rec, stop)))
|
||||
where = "宿主机 raw 单跑" if raw_only else "容器内同进程对照"
|
||||
print(f"[{where}] {SYMS} · 跑 {minutes} 分钟", flush=True)
|
||||
|
||||
deadline = time.time() + minutes * 60
|
||||
reported = set()
|
||||
while time.time() < deadline:
|
||||
await asyncio.sleep(2)
|
||||
for k, v in rec.items():
|
||||
if k in reported or "raw_ms" not in v or "hb_ms" not in v:
|
||||
continue
|
||||
reported.add(k)
|
||||
print(f" {k[0]} @{k[1]}: raw->hb 延后 {v['hb_ms'] - v['raw_ms']}ms",
|
||||
flush=True)
|
||||
stop.set()
|
||||
for t in tasks:
|
||||
t.cancel()
|
||||
await asyncio.gather(*tasks, return_exceptions=True)
|
||||
|
||||
if raw_only:
|
||||
# 只落盘 raw 到达时刻,供与容器侧按 kline_ts 对齐
|
||||
f = out_dir() / "probe_raw_host.csv"
|
||||
with f.open("w", newline="") as fh:
|
||||
w = csv.writer(fh)
|
||||
w.writerow(["sym", "kline_ts", "raw_ms"])
|
||||
for (s, k), v in sorted(rec.items(), key=lambda x: x[0][1]):
|
||||
if "raw_ms" in v:
|
||||
w.writerow([s, k, v["raw_ms"]])
|
||||
print(f"\n宿主机 raw 样本 {sum('raw_ms' in v for v in rec.values())} 根")
|
||||
print(f"产物写入 {f}")
|
||||
return
|
||||
|
||||
f = out_dir() / "probe_hb_vs_raw.csv"
|
||||
both = [(s, k, v) for (s, k), v in rec.items()
|
||||
if "raw_ms" in v and "hb_ms" in v]
|
||||
with f.open("w", newline="") as fh:
|
||||
w = csv.writer(fh)
|
||||
w.writerow(["sym", "kline_ts", "raw_ms", "hb_ms", "delta_ms"])
|
||||
for s, k, v in sorted(both, key=lambda x: x[1]):
|
||||
w.writerow([s, k, v["raw_ms"], v["hb_ms"],
|
||||
v["hb_ms"] - v["raw_ms"]])
|
||||
|
||||
print(f"\n########## 同进程内 raw WS 与 Hummingbot 的到达差 ##########")
|
||||
print(f"(正数 = Hummingbot 更慢;配对 {len(both)} 根)")
|
||||
for s in SYMS:
|
||||
d = sorted(v["hb_ms"] - v["raw_ms"] for ss, _, v in both if ss == s)
|
||||
if not d:
|
||||
print(f" {s}: 无配对样本")
|
||||
continue
|
||||
print(f" {s}: n={len(d)} 中位 {d[len(d) // 2]}ms "
|
||||
f"P90 {d[int(len(d) * .9)]}ms 最小 {d[0]}ms 最大 {d[-1]}ms")
|
||||
print(f"\n判读:中位接近 0 说明 1030ms 来自容器网络或宿主机对照本身;"
|
||||
f"中位接近 1000ms 说明是 Hummingbot 处理链路的净开销。")
|
||||
print(f"产物写入 {f}")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--minutes", type=int, default=20)
|
||||
ap.add_argument("--raw-only", action="store_true",
|
||||
help="不加载 hummingbot,只跑原始 WS(供宿主机对照)")
|
||||
a = ap.parse_args()
|
||||
asyncio.run(main_async(a.minutes, a.raw_only))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,123 @@
|
||||
"""把 `inner_ms` 拆成和本地一致的分档,用来定位两边测不一致的那部分。
|
||||
|
||||
起因:本地量到的构成是 TF_DF 占 ~80%、信号链 ~20%,服务器报的是 chan 构建
|
||||
22ms、信号链 86ms。按机器差(×1.36)也解释不了四倍差距,说明两边测的不是
|
||||
同一件事,或者有个环节只在服务器上贵。
|
||||
|
||||
用同一批窗口跑,比较分档而不是总数。两边都跑一遍再对表:
|
||||
|
||||
python research/live/probe_inner.py --syms BTC,ETH,SOL --repeat 5
|
||||
|
||||
分档口径(与 shadow_signal.compute 的调用顺序一致):
|
||||
rebuild payload → DataFrame
|
||||
ind_ltf/htf 两条腿各自的 add_indicators(TF_DF 内部会做,这里单独计时)
|
||||
chan_ltf/htf TF_DF 构建(lean)
|
||||
zones build_htf_zones
|
||||
ladder add_zone_ladder
|
||||
bsp find_fast_bsp3
|
||||
timeline htf_fx_timeline(5m 分型时间线)
|
||||
attach attach_htf_agree + attach_zone_ladder
|
||||
|
||||
注意 `ind_*` 与 `chan_*` 在真实路径里是合一的(TF_DF 内部调 add_indicators),
|
||||
这里拆开只为定位。总和会略大于实际 inner_ms。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
for _v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
|
||||
os.environ.setdefault(_v, "1")
|
||||
|
||||
HERE = Path(__file__).resolve()
|
||||
sys.path.insert(0, str(HERE.parents[1]))
|
||||
sys.path.insert(0, str(HERE.parents[2]))
|
||||
sys.path.insert(0, str(HERE.parent))
|
||||
|
||||
LTF_BARS, HTF_BARS = 2001, 801
|
||||
|
||||
|
||||
def med(fn, n: int):
|
||||
ts = []
|
||||
out = None
|
||||
for _ in range(n):
|
||||
t = time.perf_counter()
|
||||
out = fn()
|
||||
ts.append((time.perf_counter() - t) * 1000)
|
||||
return float(np.median(ts)), out
|
||||
|
||||
|
||||
def probe(sym: str, repeat: int) -> dict:
|
||||
from chanlun import TF_DF
|
||||
from chanlun.analysis.fast_bsp import (
|
||||
add_zone_ladder, attach_htf_agree, attach_zone_ladder,
|
||||
build_htf_zones, find_fast_bsp3, htf_fx_timeline,
|
||||
)
|
||||
from lib.data import fetch_ohlcv
|
||||
|
||||
dl = fetch_ohlcv(f"{sym}/USDT:USDT", "1m", LTF_BARS * 3).tail(LTF_BARS).reset_index(drop=True)
|
||||
dh = fetch_ohlcv(f"{sym}/USDT:USDT", "5m", HTF_BARS * 3).tail(HTF_BARS).reset_index(drop=True)
|
||||
|
||||
probe_tf = TF_DF(lean=True)
|
||||
r = {"sym": sym}
|
||||
r["ind_ltf"], _ = med(lambda: probe_tf.add_indicators(dl.copy()), repeat)
|
||||
r["ind_htf"], _ = med(lambda: probe_tf.add_indicators(dh.copy()), repeat)
|
||||
r["chan_ltf"], cl = med(lambda: TF_DF(dl, 1, "1m", lean=True), repeat)
|
||||
r["chan_htf"], ch = med(lambda: TF_DF(dh, 1, "5m", lean=True), repeat)
|
||||
cdf = cl.dataframe
|
||||
r["zones"], z = med(lambda: build_htf_zones(cdf, "1m", chan=cl), repeat)
|
||||
if z is None or z.empty:
|
||||
r["n_zones"] = 0
|
||||
return r
|
||||
z0 = z.reset_index(drop=True)
|
||||
r["ladder"], zl = med(lambda: add_zone_ladder(z0), repeat)
|
||||
r["bsp"], sg = med(lambda: find_fast_bsp3(cdf, zl), repeat)
|
||||
r["timeline"], tl = med(lambda: htf_fx_timeline(ch, ch.dataframe), repeat)
|
||||
r["attach"], _ = med(
|
||||
lambda: attach_zone_ladder(attach_htf_agree(sg, cdf, tl), zl), repeat)
|
||||
r["n_zones"], r["n_sig"] = len(z0), len(sg)
|
||||
|
||||
# 增量口径:init 一次后追加,看稳态单根成本
|
||||
c = TF_DF(lean=True)
|
||||
c.init_stream(dl.iloc[:-60].reset_index(drop=True), 1, "1m")
|
||||
ts = []
|
||||
for k in range(len(dl) - 60, len(dl)):
|
||||
t = time.perf_counter()
|
||||
c.append_bar(dl.iloc[k])
|
||||
ts.append((time.perf_counter() - t) * 1000)
|
||||
r["append_bar"] = float(np.median(ts[20:]))
|
||||
return r
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--syms", default="BTC,ETH,SOL")
|
||||
ap.add_argument("--repeat", type=int, default=5)
|
||||
args = ap.parse_args()
|
||||
|
||||
rows = [probe(s.strip(), args.repeat) for s in args.syms.split(",") if s.strip()]
|
||||
d = pd.DataFrame(rows).set_index("sym")
|
||||
parts = [c for c in ("ind_ltf", "ind_htf", "chan_ltf", "chan_htf", "zones",
|
||||
"ladder", "bsp", "timeline", "attach") if c in d]
|
||||
d["合计"] = d[parts].sum(axis=1)
|
||||
pd.set_option("display.width", 220)
|
||||
print("\n分档耗时(ms,中位)")
|
||||
print(d[parts + ["合计", "append_bar"]].round(2).to_string())
|
||||
print("\n占比(%)")
|
||||
print((d[parts].div(d["合计"], axis=0) * 100).round(1).to_string())
|
||||
print("\n规模")
|
||||
print(d[[c for c in ("n_zones", "n_sig") if c in d]].to_string())
|
||||
print("\n注:ind_* 与 chan_* 在真实路径里合一(TF_DF 内部调 add_indicators),"
|
||||
"拆开只为定位,合计会略大于实际 inner_ms。")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,96 @@
|
||||
"""100 USDT 的仓位在各币上能不能下出来——下单精度与最小量。
|
||||
|
||||
手工小额实盘的第一个坑不在策略,在交易规则。100 USDT 的仓位要拆成两半
|
||||
(3 ATR 减半 50 USDT、8 ATR 目标 50 USDT),任一半低于最小下单量就下不出去,
|
||||
或者被精度取整到与计划偏差很大的数量。
|
||||
|
||||
取整偏差会直接扭曲收益结构:若 50 USDT 被取整到 40,减半那一腿实际只出了
|
||||
40%,剩余 60% 暴露在 8 ATR 目标上。回测的收益结构假设是 50/50。
|
||||
|
||||
python research/live/probe_rules.py --notional 100
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve()
|
||||
sys.path.insert(0, str(HERE.parents[1]))
|
||||
sys.path.insert(0, str(HERE.parent))
|
||||
|
||||
SYMS = os.environ.get(
|
||||
"SYMS", "BTC,ETH,SOL,BNB,XRP,DOGE,ADA,AVAX,LINK,LTC").split(",")
|
||||
|
||||
|
||||
async def run(notional: float) -> None:
|
||||
from hummingbot.connector.derivative.bitget_perpetual.bitget_perpetual_derivative import ( # noqa: E501
|
||||
BitgetPerpetualDerivative,
|
||||
)
|
||||
|
||||
pairs = [f"{s}-USDT" for s in SYMS]
|
||||
conn = BitgetPerpetualDerivative(
|
||||
bitget_perpetual_api_key="", bitget_perpetual_secret_key="",
|
||||
bitget_perpetual_passphrase="", trading_pairs=pairs,
|
||||
trading_required=False)
|
||||
await conn.start_network()
|
||||
for _ in range(60):
|
||||
await asyncio.sleep(1)
|
||||
if conn.trading_rules and all(p in conn.trading_rules for p in pairs):
|
||||
break
|
||||
|
||||
print(f"仓位 {notional:.0f} USDT · 减半腿 {notional / 2:.0f} USDT\n")
|
||||
print(f" {'币':<6}{'现价':>11}{'最小量':>12}{'量步长':>12}"
|
||||
f"{'最小名义':>10} 减半腿可行性")
|
||||
bad = []
|
||||
for s in SYMS:
|
||||
p = f"{s}-USDT"
|
||||
r = conn.trading_rules.get(p)
|
||||
if r is None:
|
||||
print(f" {s:<6}{'取不到规则':>11}")
|
||||
continue
|
||||
ob = conn.get_order_book(p)
|
||||
px = float((ob.get_price(True) + ob.get_price(False)) / 2) if ob \
|
||||
else float("nan")
|
||||
min_amt = float(r.min_order_size)
|
||||
step = float(r.min_base_amount_increment)
|
||||
min_not = float(r.min_notional_size or 0)
|
||||
|
||||
half_base = (notional / 2) / px
|
||||
# 按步长向下取整——交易所就是这么处理的,向上取会下不出去
|
||||
q = Decimal(str(half_base)) // Decimal(str(step)) * Decimal(str(step))
|
||||
got = float(q)
|
||||
if got < min_amt or (min_not and got * px < min_not):
|
||||
verdict = f"⛔ 下不出(需 ≥ {max(min_amt, min_not / px):.6f})"
|
||||
bad.append(s)
|
||||
else:
|
||||
dev = abs(got * px - notional / 2) / (notional / 2) * 1e4
|
||||
verdict = f"✓ {got:.6f} 币,偏差 {dev:.0f}bp"
|
||||
if dev > 100:
|
||||
verdict += " ⚠ 取整偏差大"
|
||||
bad.append(s)
|
||||
print(f" {s:<6}{px:>11,.4f}{min_amt:>12.6f}{step:>12.6f}"
|
||||
f"{min_not:>10.1f} {verdict}")
|
||||
|
||||
print()
|
||||
if bad:
|
||||
print(f" ⛔ {notional:.0f} USDT 下这些币的减半腿有问题:{','.join(bad)}")
|
||||
print(f" 要么提高仓位,要么这些币不做减半、单腿到 8 ATR 出场——但后者")
|
||||
print(f" 改了回测的收益结构,不能直接套用原预算。")
|
||||
else:
|
||||
print(f" {notional:.0f} USDT 在全部 {len(SYMS)} 个币上都能拆成两半下出。")
|
||||
await conn.stop_network()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--notional", type=float, default=100.0)
|
||||
a = ap.parse_args()
|
||||
asyncio.run(run(a.notional))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,156 @@
|
||||
"""验证 Hummingbot 那 1030ms 的来源:Bitget candle1m 的 snapshot / update 序列。
|
||||
|
||||
Hummingbot 的 bitget_perpetual candles feed 里有这么一句:
|
||||
|
||||
if data and data.get("data") and data["action"] == "update":
|
||||
|
||||
action == "snapshot" 的消息被整条丢弃。若新 K 线的首条推送恰是 snapshot,
|
||||
Hummingbot 就必须等到下一条 update 才知道换根了,代价约等于一个推送间隔。
|
||||
|
||||
本脚本直接连原始 WS,对每条消息记录 action、K 线时间戳、到达时刻,
|
||||
然后针对每次换根回答两件事:
|
||||
1. 首条带新时间戳的消息,action 是什么
|
||||
2. 若是 snapshot,到首条 update 之间隔了多久(= Hummingbot 白等的时间)
|
||||
|
||||
.venv/bin/python research/live/probe_ws_action.py --minutes 6
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import time
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
WSS = "wss://ws.bitget.com/v2/ws/public"
|
||||
SYMS = ("BTCUSDT", "ETHUSDT", "SOLUSDT")
|
||||
OUT = Path(__file__).resolve().parents[1] / "out" / "probe_ws_action.csv"
|
||||
|
||||
|
||||
async def run(minutes: int) -> None:
|
||||
import csv
|
||||
|
||||
import aiohttp
|
||||
|
||||
payload = {"op": "subscribe",
|
||||
"args": [{"instType": "USDT-FUTURES", "channel": "candle1m",
|
||||
"instId": s} for s in SYMS]}
|
||||
|
||||
n_rows = 0
|
||||
# 每个币记录:当前时间戳、该时间戳下已见过的 action 序列
|
||||
cur: dict[str, int] = {}
|
||||
seen: dict[str, list] = defaultdict(list)
|
||||
rollovers: list[dict] = []
|
||||
cnt: dict = defaultdict(lambda: defaultdict(int))
|
||||
|
||||
# 边收边写:进程若被杀,已采到的样本仍在盘上
|
||||
OUT.parent.mkdir(parents=True, exist_ok=True)
|
||||
fh = OUT.open("w", newline="")
|
||||
w = csv.DictWriter(fh, fieldnames=["t_ms", "sym", "action", "kline_ts"])
|
||||
w.writeheader()
|
||||
|
||||
print(f"[原始 WS 探针] {SYMS} · candle1m · 跑 {minutes} 分钟", flush=True)
|
||||
deadline = time.time() + minutes * 60
|
||||
try:
|
||||
while time.time() < deadline:
|
||||
try:
|
||||
async with aiohttp.ClientSession() as sess, \
|
||||
sess.ws_connect(WSS, heartbeat=20) as ws:
|
||||
await ws.send_str(json.dumps(payload))
|
||||
while time.time() < deadline:
|
||||
msg = await ws.receive(timeout=30)
|
||||
# 断开后 receive() 会立刻返回 CLOSED;这里若只 continue
|
||||
# 就成了无等待空转,必须跳出去重连
|
||||
if msg.type is not aiohttp.WSMsgType.TEXT:
|
||||
print(f" 非文本消息 {msg.type},重连", flush=True)
|
||||
break
|
||||
raw = msg.data
|
||||
if raw == "pong":
|
||||
continue
|
||||
try:
|
||||
d = json.loads(raw)
|
||||
except Exception:
|
||||
continue
|
||||
if "data" not in d or "arg" not in d:
|
||||
if d.get("event"):
|
||||
print(f" 事件: {d}", flush=True)
|
||||
continue
|
||||
|
||||
now_ms = int(time.time() * 1000)
|
||||
sym = d["arg"]["instId"]
|
||||
action = d.get("action")
|
||||
kts = int(d["data"][0][0])
|
||||
w.writerow({"t_ms": now_ms, "sym": sym,
|
||||
"action": action, "kline_ts": kts})
|
||||
n_rows += 1
|
||||
cnt[sym][action] += 1
|
||||
if n_rows % 50 == 0:
|
||||
fh.flush()
|
||||
|
||||
prev = cur.get(sym)
|
||||
if prev is None:
|
||||
cur[sym] = kts
|
||||
seen[sym] = [(action, now_ms)]
|
||||
continue
|
||||
if kts > prev:
|
||||
cur[sym] = kts
|
||||
seen[sym] = [(action, now_ms)]
|
||||
rollovers.append({"sym": sym, "kline_ts": kts,
|
||||
"first_action": action,
|
||||
"first_ms": now_ms})
|
||||
print(f" {sym} 换根 -> 首条 action={action}",
|
||||
flush=True)
|
||||
else:
|
||||
seen[sym].append((action, now_ms))
|
||||
# 若首条是 snapshot,找该时间戳下首条 update 的延后量
|
||||
for r in rollovers:
|
||||
if (r["sym"] == sym and r["kline_ts"] == kts
|
||||
and r["first_action"] == "snapshot"
|
||||
and "gap_to_update_ms" not in r
|
||||
and action == "update"):
|
||||
r["gap_to_update_ms"] = now_ms - r["first_ms"]
|
||||
print(f" {sym} snapshot->update 间隔 "
|
||||
f"{r['gap_to_update_ms']}ms", flush=True)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as e:
|
||||
print(f" 连接异常 {type(e).__name__}: {e},1s 后重连", flush=True)
|
||||
# 重连前固定歇一下,避免服务端持续拒绝时打成重连风暴
|
||||
if time.time() < deadline:
|
||||
await asyncio.sleep(1)
|
||||
finally:
|
||||
fh.close()
|
||||
|
||||
print(f"\n########## 消息构成(共 {n_rows} 条)##########")
|
||||
for s in SYMS:
|
||||
print(f" {s}: {dict(cnt[s])}")
|
||||
|
||||
print(f"\n########## 换根时首条消息的 action({len(rollovers)} 次)##########")
|
||||
by = defaultdict(lambda: defaultdict(int))
|
||||
gaps = defaultdict(list)
|
||||
for r in rollovers:
|
||||
by[r["sym"]][r["first_action"]] += 1
|
||||
if "gap_to_update_ms" in r:
|
||||
gaps[r["sym"]].append(r["gap_to_update_ms"])
|
||||
for s in SYMS:
|
||||
g = sorted(gaps[s])
|
||||
med = g[len(g) // 2] if g else None
|
||||
print(f" {s}: 首条 action 分布 {dict(by[s])}"
|
||||
+ (f" · snapshot->update 中位 {med}ms(n={len(g)})" if g else ""))
|
||||
|
||||
all_g = sorted(x for v in gaps.values() for x in v)
|
||||
if all_g:
|
||||
print(f"\n合计 snapshot->update 中位 {all_g[len(all_g) // 2]}ms "
|
||||
f"(n={len(all_g)})——这就是 Hummingbot 因丢弃 snapshot 白等的时间")
|
||||
print(f"\n产物写入 {OUT}")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--minutes", type=int, default=6)
|
||||
asyncio.run(run(ap.parse_args().minutes))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,110 @@
|
||||
"""查 Bitget candle1m 每条推送里到底带几根 K 线、最新的在哪一端。
|
||||
|
||||
归因结果显示 ccxt.pro 比原始 WS 快约 1.4 秒,而两者订阅的是同一条频道。
|
||||
Hummingbot 的解析取 data["data"][0],我的探针也取 [0],两者一致地慢——
|
||||
若 Bitget 一条消息里带多根、最新在末尾,取 [0] 就会系统性落后一个滑动窗口。
|
||||
|
||||
对每条消息记录:元素个数、首末元素的时间戳、以及首末之差。
|
||||
若普遍 len>1 且末元素更新,则 [0] 就是那 1.4 秒的来源,且可用一行覆盖修好。
|
||||
|
||||
.venv/bin/python research/live/probe_ws_payload.py --minutes 4
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import time
|
||||
from collections import Counter, defaultdict
|
||||
|
||||
WSS = "wss://ws.bitget.com/v2/ws/public"
|
||||
SYMS = ("BTCUSDT", "ETHUSDT", "SOLUSDT")
|
||||
|
||||
|
||||
async def run(minutes: int) -> None:
|
||||
import aiohttp
|
||||
|
||||
payload = {"op": "subscribe",
|
||||
"args": [{"instType": "USDT-FUTURES", "channel": "candle1m",
|
||||
"instId": s} for s in SYMS]}
|
||||
|
||||
n_elems = Counter()
|
||||
action_elems = Counter()
|
||||
# 取 [0] 相对取 [-1] 的落后量:同一根 K 线,两种读法各自首次见到的时刻
|
||||
first_seen_head: dict = {}
|
||||
first_seen_tail: dict = {}
|
||||
n_msg = 0
|
||||
|
||||
print(f"[载荷探针] {SYMS} · 跑 {minutes} 分钟", flush=True)
|
||||
deadline = time.time() + minutes * 60
|
||||
while time.time() < deadline:
|
||||
try:
|
||||
async with aiohttp.ClientSession() as sess, \
|
||||
sess.ws_connect(WSS, heartbeat=20) as ws:
|
||||
await ws.send_str(json.dumps(payload))
|
||||
while time.time() < deadline:
|
||||
msg = await ws.receive(timeout=30)
|
||||
# 断开后 receive() 立刻返回 CLOSED,只 continue 会变成空转
|
||||
if msg.type is not aiohttp.WSMsgType.TEXT:
|
||||
print(f" 非文本消息 {msg.type},重连", flush=True)
|
||||
break
|
||||
if msg.data == "pong":
|
||||
continue
|
||||
d = json.loads(msg.data)
|
||||
if "data" not in d or "arg" not in d:
|
||||
continue
|
||||
arr = d["data"]
|
||||
if not arr:
|
||||
continue
|
||||
n_msg += 1
|
||||
sym = d["arg"]["instId"].replace("USDT", "")
|
||||
act = d.get("action")
|
||||
n_elems[len(arr)] += 1
|
||||
action_elems[(act, len(arr))] += 1
|
||||
|
||||
now_ms = int(time.time() * 1000)
|
||||
head_ts, tail_ts = int(arr[0][0]), int(arr[-1][0])
|
||||
first_seen_head.setdefault((sym, head_ts), now_ms)
|
||||
first_seen_tail.setdefault((sym, tail_ts), now_ms)
|
||||
|
||||
if n_msg <= 6 or len(arr) > 1:
|
||||
print(f" {sym} action={act} 元素数={len(arr)} "
|
||||
f"首ts={head_ts} 末ts={tail_ts} "
|
||||
f"跨度={(tail_ts - head_ts) // 1000}s", flush=True)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as e:
|
||||
print(f" 连接异常 {type(e).__name__}: {e},1s 后重连", flush=True)
|
||||
if time.time() < deadline:
|
||||
await asyncio.sleep(1)
|
||||
|
||||
print(f"\n########## 每条消息的元素个数分布(共 {n_msg} 条)##########")
|
||||
for k in sorted(n_elems):
|
||||
print(f" {k} 根: {n_elems[k]} 条")
|
||||
print("\n按 action 拆分:")
|
||||
for k in sorted(action_elems, key=lambda x: (str(x[0]), x[1])):
|
||||
print(f" action={k[0]} 元素数={k[1]}: {action_elems[k]} 条")
|
||||
|
||||
# 同一根 K 线,用 [-1] 读比用 [0] 读早多少
|
||||
lead = defaultdict(list)
|
||||
for (sym, ts), t_tail in first_seen_tail.items():
|
||||
t_head = first_seen_head.get((sym, ts))
|
||||
if t_head is not None:
|
||||
lead[sym].append(t_head - t_tail)
|
||||
print("\n########## 取 [-1] 比取 [0] 提前多少(ms)##########")
|
||||
for s in SYMS:
|
||||
v = sorted(lead[s.replace("USDT", "")])
|
||||
if not v:
|
||||
print(f" {s}: 无配对")
|
||||
continue
|
||||
print(f" {s}: n={len(v)} 中位 {v[len(v) // 2]}ms 最大 {v[-1]}ms")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--minutes", type=int, default=4)
|
||||
asyncio.run(run(ap.parse_args().minutes))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,399 @@
|
||||
"""从落盘的完整盘口与成交流,算资金容量、排队量与单根成交率。
|
||||
|
||||
## 结论先行(2026-08-28)
|
||||
|
||||
主口径仓位 **10 万 USDT**。该规模下两条约束都不绑定:
|
||||
|
||||
冲击 单边 0.01~1.87bp,占预算 0.1~11.1%,冲击反推上限 100~500 万
|
||||
成交率 按逐根累积结算,预算降幅 BTC −0% / ETH −0% / SOL −1.1%
|
||||
|
||||
⚠ 本文件里的 `composite_fill` 曾给出「10 万仓位全额成交率仅 7~63%」这种数,
|
||||
那个口径只算**首次触及那一根**的可成交量,系统性偏悲观,已不作为结论。
|
||||
真实成交率见 lib/exit_fill.py(挂单常驻多根逐步成交)。
|
||||
|
||||
|
||||
这两个数都不该等实盘暴露:
|
||||
|
||||
**容量**。预算 20bp 意味着存在一个资金上限,超过它策略就不工作。既然完整
|
||||
深度已落盘,任意仓位的冲击都能重算——一次采集回答所有资金量级,换个规模
|
||||
不必重测一周。绑定约束是**薄盘时段**而非中位盘口,所以按分位数报。
|
||||
|
||||
**maker 成交率**。回测假设 3ATR / 8ATR 的限价单全额成交。深度回答不了这个
|
||||
问题:深度说的是「现在挂着多少」,成交率问的是「之后打过来多少」。只有
|
||||
成交流能回答,而且买卖必须分开——多头在 3ATR 挂卖出,靠主动买盘成交。
|
||||
|
||||
读 gzip 时必须容忍末尾成员不完整:采集进程还在写,最后一个 gzip 成员没有
|
||||
结尾标记,直接遍历会在文件尾抛 EOFError 而丢掉**全部**已读记录。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import zlib
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
|
||||
def out_dir() -> Path:
|
||||
p = Path("/out")
|
||||
return p if p.is_dir() else Path(__file__).resolve().parents[1] / "out"
|
||||
|
||||
|
||||
GZ_MAGIC = b"\x1f\x8b\x08"
|
||||
|
||||
|
||||
def _members(blob: bytes):
|
||||
"""把可能损坏的多成员 gzip 拆成「逐个成员解压」,坏成员跳过。
|
||||
|
||||
采集进程被 SIGKILL 时,当前 gzip 成员停在 deflate 块中间、没有结尾标记。
|
||||
下一次运行以追加方式写入的新成员就接在这段垃圾字节后面。此时用
|
||||
`gzip.open` 顺序读会在损坏点抛 `invalid block type`,**该点之后的所有
|
||||
数据都读不出来**——包括后续每一轮运行写进去的。曾因此只读出 21 行而
|
||||
误以为样本就那么少,且不报错。
|
||||
|
||||
所以按成员边界扫描:某个成员解压失败,只丢它,然后前进到下一个 magic
|
||||
继续。返回 (解压出的字节, 跳过的成员数)。
|
||||
"""
|
||||
out, skipped, i, n = [], 0, blob.find(GZ_MAGIC), len(blob)
|
||||
while 0 <= i < n:
|
||||
d = zlib.decompressobj(16 + zlib.MAX_WBITS)
|
||||
try:
|
||||
chunk = d.decompress(blob[i:])
|
||||
except zlib.error:
|
||||
chunk = b""
|
||||
if chunk:
|
||||
out.append(chunk)
|
||||
# 成员完整时 unused_data 指向下一成员;否则只能往前找 magic
|
||||
if d.eof and d.unused_data:
|
||||
nxt = n - len(d.unused_data)
|
||||
else:
|
||||
nxt = blob.find(GZ_MAGIC, i + 3)
|
||||
if chunk == b"":
|
||||
skipped += 1
|
||||
i = nxt if nxt > i else -1
|
||||
return b"".join(out), skipped
|
||||
|
||||
|
||||
def read_jsonl_gz(path: Path, quiet: bool = False):
|
||||
"""读 gzip JSONL,跨运行文件汇总,坏成员跳过而非静默截断。
|
||||
|
||||
`path` 既可以是单个文件,也当作前缀用:同目录下 `<stem>.*.jsonl.gz`
|
||||
(每轮运行一个)会一并读入,这样重启不再把历史数据连坐。
|
||||
"""
|
||||
base = path.name.replace(".jsonl.gz", "")
|
||||
files = sorted({*path.parent.glob(f"{base}.*.jsonl.gz"),
|
||||
*([path] if path.exists() else [])})
|
||||
if not files:
|
||||
return
|
||||
for f in files:
|
||||
blob = f.read_bytes()
|
||||
data, skipped = _members(blob)
|
||||
n_ok = n_bad = 0
|
||||
for line in data.split(b"\n"):
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
rec = json.loads(line)
|
||||
except (json.JSONDecodeError, UnicodeDecodeError):
|
||||
n_bad += 1 # 损坏边界上的半行
|
||||
continue
|
||||
n_ok += 1
|
||||
yield rec
|
||||
if not quiet and (skipped or n_bad):
|
||||
print(f" ⚠ {f.name}: 读出 {n_ok:,} 条,"
|
||||
f"跳过 {skipped} 个损坏成员 / {n_bad} 个半行")
|
||||
|
||||
|
||||
def impact_bp(levels: list, notional: float, mid: float) -> float | None:
|
||||
"""吃掉 notional 计价币后的加权均价相对中间价,bp。深度不足返回 None。"""
|
||||
need = notional
|
||||
cost = 0.0
|
||||
qty = 0.0
|
||||
for px, amt in levels:
|
||||
avail = px * amt
|
||||
take = min(avail, need)
|
||||
q = take / px
|
||||
cost += q * px
|
||||
qty += q
|
||||
need -= take
|
||||
if need <= 1e-9:
|
||||
break
|
||||
if need > 1e-9 or qty <= 0:
|
||||
return None
|
||||
return (cost / qty / mid - 1.0) * 1e4
|
||||
|
||||
|
||||
def queue_ahead(books_path: Path, notional: float = 1e5) -> None:
|
||||
"""限价单排在多少量之后。
|
||||
|
||||
这是唯一还没建模的成本项。exit_fill 假定我们能吃到该价位的全部对手方
|
||||
成交量,实际上我们排在该价位既有挂单之后。
|
||||
|
||||
这里量不了 3ATR 处的排队——3ATR 约 30bp,而 50 档盘口只覆盖 1.4~4.5bp,
|
||||
那个价位远在可见深度之外。但价格走到我们的限价时,我们的单就成了盘口
|
||||
最优档附近的一员,所以「最优档通常趴着多少量」是个有用的上界参照。
|
||||
"""
|
||||
per: dict[str, dict[str, list[float]]] = {}
|
||||
for r in read_jsonl_gz(books_path):
|
||||
asks, bids = r["asks"], r["bids"]
|
||||
if not asks or not bids:
|
||||
continue
|
||||
mid = (asks[0][0] + bids[0][0]) / 2.0
|
||||
d = per.setdefault(r["sym"], {"top": [], "b1": [], "b5": []})
|
||||
d["top"].append(asks[0][0] * asks[0][1])
|
||||
for tag, bp in (("b1", 1.0), ("b5", 5.0)):
|
||||
lim = mid * (1 + bp / 1e4)
|
||||
d[tag].append(sum(p * a for p, a in asks if p <= lim))
|
||||
if not per:
|
||||
return
|
||||
print(f"\n\n########## 限价单前方的排队量 ##########")
|
||||
print(f" 主口径仓位 {notional:,.0f} USDT\n")
|
||||
for sym, d in sorted(per.items()):
|
||||
top = np.array(d["top"]); b1 = np.array(d["b1"])
|
||||
b5 = np.array(d["b5"])
|
||||
print(f" {sym} 最优档 {np.median(top):>10,.0f} · "
|
||||
f"1bp 内累计 {np.median(b1):>10,.0f} · "
|
||||
f"5bp 内累计 {np.median(b5):>10,.0f}")
|
||||
print(f" 我们的 {notional:,.0f} 相当于最优档的 "
|
||||
f"{notional / max(np.median(top), 1):.1f} 倍、"
|
||||
f"1bp 内总量的 {notional / max(np.median(b1), 1):.2f} 倍")
|
||||
print("\n 倍数远小于 1 则排队可忽略;接近或超过 1 则我们本身就是那一档的")
|
||||
print(" 主要挂单,exit_fill 的成交量假设需要打折。")
|
||||
print(" SOL 的最优档倍数畸高是 tick 更细所致(Bitget 的 SOL tick 比同类")
|
||||
print(" 细约 10 倍,同样的量摊到 10 倍多的价位上),所以对 SOL 该看 5bp")
|
||||
print(" 档而非最优档;但即便如此它仍是三个币里排队压力最大的一个")
|
||||
|
||||
|
||||
def capacity(books_path: Path, budgets: dict[str, float],
|
||||
pctl: float = 10.0) -> None:
|
||||
"""报各币的深度曲线与「冲击吃掉预算多少」的资金上限。"""
|
||||
grid = [1e4, 2.5e4, 5e4, 1e5, 2e5, 5e5, 1e6, 2e6]
|
||||
per: dict[str, dict[float, list[float]]] = {}
|
||||
n = 0
|
||||
for r in read_jsonl_gz(books_path):
|
||||
asks, bids = r["asks"], r["bids"]
|
||||
if not asks or not bids:
|
||||
continue
|
||||
mid = (asks[0][0] + bids[0][0]) / 2.0
|
||||
d = per.setdefault(r["sym"], {g: [] for g in grid})
|
||||
for g in grid:
|
||||
v = impact_bp(asks, g, mid)
|
||||
d[g].append(np.nan if v is None else v)
|
||||
n += 1
|
||||
|
||||
if not n:
|
||||
print("没有盘口快照,先跑采集")
|
||||
return
|
||||
|
||||
print(f"\n########## 资金容量 ##########")
|
||||
print(f" 基于 {n:,} 份完整盘口快照(单边买入方向)\n")
|
||||
for sym, d in per.items():
|
||||
b = budgets.get(sym)
|
||||
print(f" {sym} 预算 {b:.2f}bp" if b else f" {sym}")
|
||||
print(f" {'名义额':>12} {'冲击中位':>10} {'冲击P90':>10} "
|
||||
f"{'吃满深度率':>10} {'占预算':>8}")
|
||||
for g in grid:
|
||||
a = np.array(d[g], dtype=float)
|
||||
fill = float(np.isfinite(a).mean())
|
||||
if fill == 0:
|
||||
print(f" {g:>12,.0f} {'—— 50 档吃不下 ——':>30}")
|
||||
continue
|
||||
med = float(np.nanmedian(a))
|
||||
p90 = float(np.nanpercentile(a, 90))
|
||||
share = f"{med / b * 100:6.1f}%" if b else " na"
|
||||
print(f" {g:>12,.0f} {med:>10.2f} {p90:>10.2f} "
|
||||
f"{fill * 100:>9.1f}% {share:>8}")
|
||||
if b:
|
||||
# 上限:冲击的 P90(薄盘时段)吃掉预算三成为止。三成是留给
|
||||
# 漂移与价差的余地——它们才是主项,冲击不该独占预算
|
||||
cap = None
|
||||
for g in grid:
|
||||
a = np.array(d[g], dtype=float)
|
||||
if not np.isfinite(a).any():
|
||||
break
|
||||
if float(np.nanpercentile(a, 90)) > b * 0.30:
|
||||
break
|
||||
cap = g
|
||||
if cap is None:
|
||||
print(f" → 连最小档 {grid[0]:,.0f} 的薄盘冲击都超预算三成")
|
||||
else:
|
||||
print(f" → 资金上限约 {cap:,.0f} USDT"
|
||||
f"(薄盘 P90 冲击 ≤ 预算 30%)")
|
||||
print()
|
||||
|
||||
|
||||
def maker_fill(tape_path: Path, mults=(3.0, 8.0),
|
||||
notionals=(5e4, 1e5, 2e5)) -> None:
|
||||
"""限价单挂在离场目标位,本根内有多少主动量打到那里。
|
||||
|
||||
这里只回答「量够不够」。真实成交还要看排队位置——我们的单排在该价位
|
||||
已有挂单之后,所以这是**上界**:量不够则必然不能全成交,量够也未必成交。
|
||||
"""
|
||||
rows = list(read_jsonl_gz(tape_path))
|
||||
if not rows:
|
||||
print("没有成交流数据,先跑采集")
|
||||
return
|
||||
print(f"\n########## maker 腿成交量上界 ##########")
|
||||
print(f" 基于 {len(rows):,} 根的逐价位成交聚合")
|
||||
print(f" 多头在目标位挂卖出,成交靠主动**买**盘,故只计买方向\n")
|
||||
|
||||
# 限价单只能被**价格 ≥ 限价**的主动买成交打到。而止盈位被触及的那一根,
|
||||
# 限价往往就落在该根价格区间的顶部——最高价刚好碰到目标位是最典型的
|
||||
# 情形。所以按「限价距最高价多近」分层:depth=0 表示限价正好在最高价
|
||||
# (只有打在最高价那一档的量算数),depth=0.25 表示限价在区间顶部 25% 处
|
||||
depths = (0.0, 0.10, 0.25, 1.0)
|
||||
per: dict[str, dict[float, list[float]]] = {}
|
||||
for r in rows:
|
||||
buys = {float(p): v for p, v in r["buys"].items()}
|
||||
d = per.setdefault(r["sym"], {k: [] for k in depths})
|
||||
if not buys:
|
||||
for k in depths:
|
||||
d[k].append(0.0)
|
||||
continue
|
||||
hi, lo = max(buys), min(buys)
|
||||
rng = hi - lo
|
||||
for k in depths:
|
||||
floor_px = hi - k * rng
|
||||
d[k].append(sum(p * v for p, v in buys.items() if p >= floor_px))
|
||||
|
||||
for sym, d in per.items():
|
||||
print(f" {sym} ≥ 限价的主动买成交额(USDT),按限价所处位置分层")
|
||||
print(f" {'限价位置':>16} {'中位':>12} {'P25':>12} "
|
||||
+ " ".join(f"{n:>9,.0f}全仓" for n in notionals))
|
||||
for k in depths:
|
||||
a = np.array(d[k], dtype=float)
|
||||
where = ("正好在最高价" if k == 0 else
|
||||
"整根全部成交" if k == 1.0 else
|
||||
f"区间顶部 {k * 100:.0f}%")
|
||||
cells = " ".join(f"{float((a >= n).mean()) * 100:8.1f}%"
|
||||
for n in notionals)
|
||||
print(f" {where:>16} {np.median(a):>12,.0f} "
|
||||
f"{np.percentile(a, 25):>12,.0f} {cells}")
|
||||
print()
|
||||
print(" 「正好在最高价」那一行才是止盈被刚好触及时的真实处境;")
|
||||
print(" 「整根全部成交」是最宽松的上界。两行差多少,就是回测那个")
|
||||
print(" 「限价单全额成交」假设虚了多少。而且这仍未计排队——我们的单")
|
||||
print(" 排在该价位既有挂单之后,所以真实成交率比表里更低")
|
||||
|
||||
|
||||
def tape_shape(tape_path: Path, kgrid: np.ndarray) -> dict[str, np.ndarray]:
|
||||
"""成交流给「形状」:一根的主动买成交额里,有多少比例落在区间顶部 k 之内。
|
||||
|
||||
形状与规模分开是为了绕开成交流样本小的限制——规模(每根成交多少钱)由
|
||||
210 天历史成交量提供,成交流只需给出形状。
|
||||
|
||||
⚠ 只算主动**买**、只自顶部累积,所以这条曲线只适用于**多头**止盈(挂卖
|
||||
出,靠主动买打上来)。空头止盈要用主动卖自底部累积的曲线,二者实测并不
|
||||
对称:主动买在区间顶部 20% 内已占 40%,主动卖在底部 20% 内只有 18%,
|
||||
BTC/ETH 平均偏差 0.21。lib/exit_fill 里两条曲线是分开的(SHAPE_F 与
|
||||
SHAPE_F_SHORT);用同一条会把空头的可成交量按多头分布高估。
|
||||
|
||||
「形状几十根就稳定」这个说法要打折:45 根样本足以看出上面那个方向性差异,
|
||||
但不足以定数值。好在预算对形状不敏感(见 step43 --sensitivity)。
|
||||
"""
|
||||
acc: dict[str, list[np.ndarray]] = {}
|
||||
for r in read_jsonl_gz(tape_path):
|
||||
buys = {float(p): v for p, v in r["buys"].items()}
|
||||
if len(buys) < 2:
|
||||
continue
|
||||
hi, lo = max(buys), min(buys)
|
||||
rng = hi - lo
|
||||
if rng <= 0:
|
||||
continue
|
||||
tot = sum(p * v for p, v in buys.items())
|
||||
if tot <= 0:
|
||||
continue
|
||||
frac = np.array([sum(p * v for p, v in buys.items()
|
||||
if p >= hi - k * rng) / tot for k in kgrid])
|
||||
acc.setdefault(r["sym"], []).append(frac)
|
||||
return {s: np.mean(np.vstack(v), axis=0) for s, v in acc.items() if v}
|
||||
|
||||
|
||||
def composite_fill(tape_path: Path, pen_path: Path, cache: Path,
|
||||
notionals=(5e4, 1e5, 2e5)) -> None:
|
||||
"""把穿透深度分布与成交量曲线合并,出**单根**内的 maker 成交率。
|
||||
|
||||
⚠ 这个数只回答「限价单若仅有首次触及那一根可以成交,能否成交」。真实的
|
||||
挂单是常驻的:它在那儿放最多 48 根,每根都在成交,且价格决定性穿过限价
|
||||
时整根成交量都可用。所以本函数系统性**偏悲观**,不能当作成交率结论。
|
||||
真实成交率见 lib/exit_fill.walk_filled 与 step43_fill_aware_budget.py,
|
||||
那里按逐根累积结算,10 万仓位下预算降幅不足 1%。
|
||||
"""
|
||||
if not pen_path.exists():
|
||||
print("\n没有 penetration.csv,先跑 penetration.py")
|
||||
return
|
||||
kgrid = np.linspace(0.0, 1.0, 51)
|
||||
shape = tape_shape(tape_path, kgrid)
|
||||
if not shape:
|
||||
print("\n成交流样本不足,无法定形状")
|
||||
return
|
||||
pen = pd.read_csv(pen_path)
|
||||
|
||||
print(f"\n\n########## maker 腿真实成交率 ##########")
|
||||
print(f" 穿透深度分布(历史 63 万次触及)× 每根成交额(210 天)")
|
||||
print(f" × 区间内成交分布形状(影子成交流)\n")
|
||||
|
||||
for sym in sorted(shape):
|
||||
cands = sorted(cache.glob(f"bitget_{sym}_1m_*.feather"),
|
||||
key=lambda p: p.stat().st_size, reverse=True)
|
||||
if not cands:
|
||||
continue
|
||||
bars = pd.read_feather(cands[0])
|
||||
# 每根的主动买成交额。取总成交额的一半——买卖大致均衡,且这与
|
||||
# 成交流实测的买卖比一致
|
||||
bar_notional = (bars["volume"].to_numpy(float)
|
||||
* bars["close"].to_numpy(float)) * 0.5
|
||||
bar_notional = bar_notional[np.isfinite(bar_notional)
|
||||
& (bar_notional > 0)]
|
||||
f = shape[sym]
|
||||
for tgt in sorted(pen["target_atr"].unique()):
|
||||
k = pen[(pen["sym"] == sym)
|
||||
& (pen["target_atr"] == tgt)]["k"].to_numpy(float)
|
||||
if not k.size:
|
||||
continue
|
||||
# 独立配对:穿透位置与该根成交额各自抽样。真实触及根多为放量根,
|
||||
# 故此处偏**保守**(低估可成交量)
|
||||
rng = np.random.default_rng(0)
|
||||
m = 200_000
|
||||
ks = rng.choice(k, m)
|
||||
ns = rng.choice(bar_notional, m)
|
||||
avail = np.interp(ks, kgrid, f) * ns
|
||||
print(f" {sym} · 目标 {tgt:g}ATR · 每根主动买额中位 "
|
||||
f"{np.median(bar_notional):,.0f} USDT")
|
||||
for nt in notionals:
|
||||
full = float((avail >= nt).mean())
|
||||
half = float((avail >= nt / 2).mean())
|
||||
print(f" 仓位 {nt:>9,.0f}:全额成交 {full * 100:5.1f}%"
|
||||
f" · 至少半额 {half * 100:5.1f}%"
|
||||
f" · 可成交额中位 {np.median(avail):>10,.0f}")
|
||||
# 成交率反推的资金上限。这才是绑定约束——它比冲击反推的上限
|
||||
# 低一到两个数量级,而后者才是通常被当作「容量」的那个数
|
||||
for want in (0.80, 0.90):
|
||||
cap = float(np.quantile(avail, 1.0 - want))
|
||||
print(f" → 要 {want * 100:.0f}% 的止盈全额成交,"
|
||||
f"仓位须 ≤ {cap:,.0f} USDT")
|
||||
print()
|
||||
print(" 未计排队(我们的单排在该价位既有挂单之后),故仍是上界。")
|
||||
print(" 回测把这些止盈按「全额成交在目标价」计,差多少就是收益虚多少")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--books", default=None)
|
||||
ap.add_argument("--tape", default=None)
|
||||
a = ap.parse_args()
|
||||
d = out_dir()
|
||||
from lib.shadow_budget import BUDGET_BP
|
||||
tape = Path(a.tape) if a.tape else d / "shadow_tape.jsonl.gz"
|
||||
books = Path(a.books) if a.books else d / "shadow_books.jsonl.gz"
|
||||
capacity(books, BUDGET_BP)
|
||||
queue_ahead(books)
|
||||
maker_fill(tape)
|
||||
composite_fill(tape, d / "penetration.csv",
|
||||
Path(__file__).resolve().parent / "cache")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,937 @@
|
||||
"""影子交易器:在 Hummingbot 运行时上测 1m 腿的真实入场滑点。
|
||||
|
||||
不下单。用 Hummingbot 的 Bitget 连接器取真实盘口,按信号方向和仓位吃单深度
|
||||
算出「若此刻市价单进场会成交在哪」,再与回测假设的成交价相减。
|
||||
|
||||
为什么必须跑在 Hummingbot 上而不是自写脚本:要测的是**生产路径**的滑点。
|
||||
决定成交价的是实际执行链路的延迟,换个运行时测出来的数就不作数了。
|
||||
(连接器的换根解析 bug 见 patched_candles.py,已修,拿回约 1.06 秒。)
|
||||
|
||||
为什么不能用 paper trade 的成交:那是 Hummingbot 自己的撮合模型模拟的,
|
||||
测出来是模型行为不是市场行为。
|
||||
|
||||
口径对齐 step42_exit_tp_1m.py:回测假设成交在**信号次根的开盘价**
|
||||
(entry_delay=1),所以基准价就是换根后新一根的 open。滑点为正表示比回测差。
|
||||
滤网(同向 + 中枢阶梯 + ATR 门控)在 shadow_signal.py 里,必须与预算同源。
|
||||
|
||||
### 统计口径三条硬要求
|
||||
|
||||
1. **主口径只用 pass_all 的信号根**。未过滤的照记但只作提前读数——
|
||||
在我们根本不会下单的根上测滑点会把判据算宽
|
||||
2. **条件漂移与无条件漂移分开报**。所以每根 K 线都记一份漂移
|
||||
(shadow_drift.csv),不只信号根。两者的差就是「系统性追价」的大小
|
||||
3. **出场腿按 maker/taker 分开**。止盈挂限价不吃滑点,把那 60% 混进
|
||||
平均值会低估真实成本。出场腿属持仓管理,尚未实现
|
||||
|
||||
### lag 探针:超阈值要停开仓,不能只打日志
|
||||
|
||||
补丁只防得住「上游代码变了」,防不住 Bitget 再改一次消息格式。每根记
|
||||
本地接收 − K 线收盘,近 30 根取中位数,超 800ms 即判该币不健康。
|
||||
要停开仓是因为这种退化是**经济性且静默**的:不崩不报错,只让收益慢慢
|
||||
变差,几周后才从统计里看得出来。影子期不下单,故落到 lag_ok 字段上。
|
||||
|
||||
### 盘口滚动缓冲把延迟变成自变量
|
||||
|
||||
每 100ms 存一份盘口。信号触发后,不只记「我们实际算完时」的滑点,而是回查
|
||||
t_close+0.5s / 1s / 2s / 5s 各一个。这样即使本机算得慢,也能读出「若延迟为
|
||||
X 秒,滑点是多少」,决策不被自身实现拖累。
|
||||
|
||||
### 滑点分解
|
||||
|
||||
延迟漂移 中间价相对次根开盘价的偏移——主项,且入场方向上系统性追价
|
||||
盘口价差 最优价相对中间价
|
||||
深度冲击 吃单加权价相对最优价
|
||||
|
||||
Bitget 永续实测价差仅约 0.01bp、100 档深度,故预期延迟漂移占绝大部分。
|
||||
|
||||
docker run -d --name shadow -w /home/hummingbot \\
|
||||
-e PYTHONPATH=/home/hummingbot:/repo/research:/repo/research/live:/repo \\
|
||||
-v $PWD:/repo:ro -v $PWD/research/out:/out \\
|
||||
--entrypoint /opt/conda/envs/hummingbot/bin/python \\
|
||||
hummingbot/hummingbot:latest /repo/research/live/shadow_hb.py --hours 24
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import csv
|
||||
import gzip
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import socket
|
||||
import sys
|
||||
import time
|
||||
from collections import deque
|
||||
from concurrent.futures import ProcessPoolExecutor
|
||||
from concurrent.futures.process import BrokenProcessPool
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from lib.shadow_budget import LAG_ALARM_MS, LAG_WINDOW, lag_healthy
|
||||
|
||||
# 总线模块住在生产子树 live/ 下。方向是刻意的:**生产不 import 研究侧**,
|
||||
# 研究侧反过来读生产持有的契约。见 live/live_exec.py 文件头
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2] / "live"))
|
||||
import signal_bus # noqa: E402
|
||||
import tg_notify # noqa: E402
|
||||
|
||||
# 站点标识。跨地对比时两台机器的 CSV 要能合起来读,没有这一列就分不清哪行
|
||||
# 来自哪台。默认取主机名,部署脚本会显式传 SHADOW_SITE(如 sg-hetzner)
|
||||
SITE = os.environ.get("SHADOW_SITE") or socket.gethostname()
|
||||
# 判「加 worker 有没有用」必须知道核数:CPU 密集的活,worker 超过核数不增吞吐
|
||||
CORES = os.cpu_count() or 1
|
||||
# 判定要知道增量开没开,否则增量已生效时还会继续推荐「走增量」
|
||||
INCR_ON = os.environ.get("SHADOW_INCR", "1") not in ("0", "", "false")
|
||||
# 少于这么多根就不送去算。缠论要先有分型再有笔再有中枢,几十根出不来中枢,
|
||||
# 送过去只会白占一个计算槽
|
||||
MIN_BARS = 200
|
||||
|
||||
SYMS = ("BTC", "ETH", "SOL")
|
||||
# 多存一根:deque 尾部是尚未收盘的当前根,剔除后正好剩 step39 定下的窗口
|
||||
LTF_BARS, HTF_BARS = 2001, 801 # 有效窗口 2000 / 800,命中率在此饱和
|
||||
BOOK_HZ = 10 # 盘口采样 10Hz
|
||||
BOOK_KEEP_S = 30 # 缓冲保留 30 秒,够回查到 +5s
|
||||
BOOK_TOL_MS = 250 # 回查容差:10Hz 正常 ≤100ms,留些余量
|
||||
BOOK_DEPTH = 50 # 双边各 50 档,实测能撑 78 万~261 万美元
|
||||
DELAYS_S = (0.5, 1.0, 2.0, 5.0) # 回查点
|
||||
# 主口径 10 万名义额(2026-08-28 定:不会有更大资金)。上下各留两档是为了
|
||||
# 读出局部斜率——单点看不出「再大一倍会怎样」。
|
||||
# **真正的答案在 shadow_books.jsonl.gz 里**:完整盘口已落盘,任意资金量级的
|
||||
# 冲击都能离线重算,换规模不必重测,这里的档位只为让 CSV 直接可读
|
||||
NOTIONALS = (25_000.0, 50_000.0, 100_000.0, 200_000.0)
|
||||
NUM_COLS = ["timestamp", "open", "high", "low", "close", "volume"]
|
||||
|
||||
|
||||
def out_dir() -> Path:
|
||||
p = Path("/out")
|
||||
return p if p.is_dir() else Path(__file__).resolve().parents[1] / "out"
|
||||
|
||||
|
||||
RUN_ID = time.strftime("%Y%m%dT%H%M%S", time.gmtime())
|
||||
|
||||
|
||||
def run_path(path: Path) -> Path:
|
||||
"""把 `x.jsonl.gz` 变成本轮专属的 `x.<RUN_ID>.jsonl.gz`。
|
||||
|
||||
gzip 追加流在进程被杀后不可靠(见 BookLog 的说明),分文件是唯一能保证
|
||||
历史数据不被后续运行连坐的办法。读侧 shadow_depth.read_jsonl_gz 会把
|
||||
同前缀的所有文件一并读入,所以分文件对分析是透明的。
|
||||
"""
|
||||
return path.with_name(path.name.replace(".jsonl.gz",
|
||||
f".{RUN_ID}.jsonl.gz"))
|
||||
|
||||
|
||||
def _writer(path: Path, cols: list[str]):
|
||||
"""追加模式打开;表头对不上就先把旧文件归档。
|
||||
|
||||
不校验的话,列一改,DictWriter 会按新顺序把行写到旧表头下面——
|
||||
读出来整片错位,而且没有任何报错。长跑靠追加续命,这个校验是必需的。
|
||||
"""
|
||||
if path.exists() and path.stat().st_size > 0:
|
||||
with path.open(newline="") as fh:
|
||||
old = next(csv.reader(fh), [])
|
||||
if old != cols:
|
||||
arch = path.parent / "archive"
|
||||
arch.mkdir(exist_ok=True)
|
||||
dst = arch / f"{path.stem}_{time.strftime('%Y%m%d_%H%M%S')}.csv"
|
||||
path.rename(dst)
|
||||
print(f" [CSV] {path.name} 表头已变,旧数据归档为 {dst.name}",
|
||||
flush=True)
|
||||
fresh = not path.exists() or path.stat().st_size == 0
|
||||
f = path.open("a", newline="")
|
||||
w = _SiteWriter(csv.DictWriter(f, fieldnames=cols))
|
||||
if fresh:
|
||||
w.writeheader()
|
||||
return f, w
|
||||
|
||||
|
||||
class _SiteWriter:
|
||||
"""DictWriter 的薄包装,自动补上 site 列。
|
||||
|
||||
逐个 writerow 手加 site 有四处,漏一处就是静默的空值,而跨地对比正是靠
|
||||
这一列区分数据来源。在这里注入,漏不掉。
|
||||
"""
|
||||
|
||||
def __init__(self, w: csv.DictWriter) -> None:
|
||||
self._w = w
|
||||
|
||||
def writeheader(self) -> None:
|
||||
self._w.writeheader()
|
||||
|
||||
def writerow(self, row: dict) -> None:
|
||||
row.setdefault("site", SITE)
|
||||
self._w.writerow(row)
|
||||
|
||||
|
||||
class BookLog:
|
||||
"""把完整盘口快照落成 gzip JSONL。
|
||||
|
||||
只记「某几个仓位档的成交价」的话,这批数据的寿命就等于那几个档位的寿命:
|
||||
换一次资金规模就得重跑一周。存完整深度后,任意仓位的冲击都能离线重算,
|
||||
一次采集回答所有资金量级的问题——包括容量上限那个必须现在就算、
|
||||
不该等实盘暴露的数。
|
||||
|
||||
**每轮运行单独一个文件**,不追加到同一个。追加看着更省事,实际很危险:
|
||||
进程被 SIGKILL 时当前 gzip 成员停在 deflate 块中间,下一轮追加的新成员
|
||||
接在这段垃圾字节之后,顺序解压会在损坏点抛 `invalid block type`,该点
|
||||
之后的所有数据——包括后续每一轮写进去的——全都读不出来。已经因此丢过
|
||||
一次。分文件后损坏最多只影响被杀那一轮的尾部。
|
||||
"""
|
||||
|
||||
def __init__(self, path: Path) -> None:
|
||||
self.path = run_path(path)
|
||||
self.fh = gzip.open(self.path, "at", encoding="utf-8")
|
||||
self.n = 0
|
||||
|
||||
def write(self, sym: str, kline_ts: int, label: str, delay_ms: int,
|
||||
target: int, book_ts: int, bids: np.ndarray,
|
||||
asks: np.ndarray) -> None:
|
||||
# 只留价与量两列,update_id 对离线分析没用。round 到 10 位避免
|
||||
# float repr 把文件撑大一倍
|
||||
rec = {"site": SITE, "sym": sym, "kline_ts": kline_ts, "label": label,
|
||||
"delay_ms": delay_ms, "target": target, "book_ts": book_ts,
|
||||
"bids": [[round(float(p), 10), round(float(a), 10)]
|
||||
for p, a, *_ in bids],
|
||||
"asks": [[round(float(p), 10), round(float(a), 10)]
|
||||
for p, a, *_ in asks]}
|
||||
self.fh.write(json.dumps(rec, separators=(",", ":")) + "\n")
|
||||
self.n += 1
|
||||
|
||||
def flush(self) -> None:
|
||||
self.fh.flush()
|
||||
|
||||
def close(self) -> None:
|
||||
try:
|
||||
self.fh.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
class TapeLog:
|
||||
"""按 K 线、按价位聚合成交量,用来判 maker 腿能不能全额成交。
|
||||
|
||||
回测假设 3ATR 和 8ATR 的限价单全额成交。十万量级挂在那里,全成交还是
|
||||
部分成交是完全不同的事——部分成交会把分批出场的收益结构改掉,而这个
|
||||
问题盘口深度回答不了:深度说的是「现在有多少人挂着」,成交率问的是
|
||||
「之后有多少人打过来」。只有成交流能回答。
|
||||
|
||||
**买卖必须分开存。** 多头在 3ATR 挂卖出止盈,成交靠的是主动**买盘**
|
||||
打上来;把双边成交量合在一起会把成交率高估约一倍。
|
||||
|
||||
聚合到「根 × 价位」而不是逐笔:判据是「本根内有多少量在 ≥ 限价处成交」,
|
||||
逐笔的时序对这个判据没有增量信息,而聚合能把体量压下两个数量级。
|
||||
|
||||
每轮运行单独一个文件,理由同 BookLog。
|
||||
"""
|
||||
|
||||
def __init__(self, path: Path) -> None:
|
||||
self.path = run_path(path)
|
||||
self.fh = gzip.open(self.path, "at", encoding="utf-8")
|
||||
# sym -> side('b'/'s') -> price -> 累计基础币量
|
||||
self.acc: dict[str, dict[str, dict[float, float]]] = {}
|
||||
self.n_trades = 0
|
||||
|
||||
def add(self, sym: str, is_buy: bool, price: float, amount: float) -> None:
|
||||
d = self.acc.setdefault(sym, {"b": {}, "s": {}})
|
||||
side = d["b"] if is_buy else d["s"]
|
||||
side[price] = side.get(price, 0.0) + amount
|
||||
self.n_trades += 1
|
||||
|
||||
def flush_bar(self, sym: str, bar_ts: int) -> None:
|
||||
"""一根走完就把这根的聚合结果落盘并清空。"""
|
||||
d = self.acc.get(sym)
|
||||
if not d or (not d["b"] and not d["s"]):
|
||||
return
|
||||
rec = {"site": SITE, "sym": sym, "bar_ts": bar_ts,
|
||||
"buys": {f"{p:.10g}": round(v, 10)
|
||||
for p, v in sorted(d["b"].items())},
|
||||
"sells": {f"{p:.10g}": round(v, 10)
|
||||
for p, v in sorted(d["s"].items())}}
|
||||
self.fh.write(json.dumps(rec, separators=(",", ":")) + "\n")
|
||||
self.fh.flush()
|
||||
self.acc[sym] = {"b": {}, "s": {}}
|
||||
|
||||
def close(self) -> None:
|
||||
try:
|
||||
self.fh.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def hb_to_research(cdf: pd.DataFrame) -> pd.DataFrame:
|
||||
"""Hummingbot 的 candles_df 转成 research/lib/data.py 的列结构。
|
||||
|
||||
HB 的 timestamp 是秒且无 date 列;chanlun 的 kline builder 需要真 datetime,
|
||||
时区跟 lib/data.py 取 Asia/Shanghai,保证与回测同一口径。
|
||||
"""
|
||||
ts_ms = (cdf["timestamp"].astype("int64") * 1000)
|
||||
date = pd.to_datetime(ts_ms, unit="ms", utc=True).dt.tz_convert("Asia/Shanghai")
|
||||
out = pd.DataFrame({"timestamp": ts_ms.astype("int64"), "date": date})
|
||||
for c in ("open", "high", "low", "close", "volume"):
|
||||
out[c] = pd.to_numeric(cdf[c], errors="coerce")
|
||||
return out.dropna().drop_duplicates(subset=["timestamp"]) \
|
||||
.sort_values("timestamp").reset_index(drop=True)
|
||||
|
||||
|
||||
def book_from(bids: np.ndarray, asks: np.ndarray):
|
||||
"""用缓冲里的快照临时搭一个 OrderBook,以便调用框架自带的吃单查询。
|
||||
|
||||
自己手写吃单曾经踩过两个坑,框架版都没有:`get_vwap_for_volume` 返回的
|
||||
是真加权均价(市价单的实际成交价),而 `get_price_for_quote_volume` 返回
|
||||
的是**边际价**,用后者会高估冲击;深度不足时框架返回 nan 而不是一个
|
||||
「看起来很正常」的部分成交均价,靠 query_volume/result_volume 判断。
|
||||
"""
|
||||
from hummingbot.core.data_type.order_book import OrderBook
|
||||
ob = OrderBook()
|
||||
ob.apply_numpy_snapshot(bids, asks)
|
||||
return ob
|
||||
|
||||
|
||||
class BookBuffer:
|
||||
"""每币一份滚动盘口。按时间戳回查,取第一个不早于目标时刻的快照。
|
||||
|
||||
回查必须有容差上界。10Hz 下正常落在目标后 100ms 内(所有延迟点同向
|
||||
偏约 +50ms,不影响曲线形状),但采样一旦卡顿,标着「0.5s」的那行可能
|
||||
用的是 +3s 的盘口——数据看不出异常,判读却已经错了。超容差宁可丢弃,
|
||||
并且把快照实际时刻写进 CSV,让这件事事后可查。
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.buf: dict[str, deque] = {s: deque() for s in SYMS}
|
||||
self.n_stale = 0 # 因超容差被丢弃的回查次数
|
||||
|
||||
def push(self, sym: str, t_ms: int, bids: list, asks: list) -> None:
|
||||
d = self.buf[sym]
|
||||
d.append((t_ms, bids, asks))
|
||||
cutoff = t_ms - BOOK_KEEP_S * 1000
|
||||
while d and d[0][0] < cutoff:
|
||||
d.popleft()
|
||||
|
||||
def at(self, sym: str, t_ms: int) -> tuple | None:
|
||||
for snap in self.buf[sym]:
|
||||
if snap[0] >= t_ms:
|
||||
if snap[0] - t_ms > BOOK_TOL_MS:
|
||||
self.n_stale += 1
|
||||
return None
|
||||
return snap
|
||||
return None
|
||||
|
||||
|
||||
class Shadow:
|
||||
def __init__(self, workers: int, hours: float) -> None:
|
||||
self.workers = workers
|
||||
self.deadline = time.time() + hours * 3600
|
||||
self.books = BookBuffer()
|
||||
self.pool: ProcessPoolExecutor | None = None
|
||||
self.feeds_l: dict = {}
|
||||
self.feeds_h: dict = {}
|
||||
self.connector = None
|
||||
self.stop = asyncio.Event()
|
||||
# 持有 fire-and-forget 任务的强引用。只 create_task 不留引用的话,
|
||||
# 任务可能在完成前被 GC 掉,asyncio 官方文档明确警告过这一点
|
||||
self._tasks: set = set()
|
||||
self.n_broken = 0
|
||||
self._hb_last_bars = 0
|
||||
# 排队 / 纯计算的滚动窗口,用来判断加核有没有用
|
||||
self.q_hist: deque = deque(maxlen=90)
|
||||
self.i_hist: deque = deque(maxlen=90)
|
||||
# 每个收盘时刻「清空所有币」耗时。这才是决定信号何时可下单的量:
|
||||
# 币同一秒收盘,币数超 worker 数时后面的币串行等待,而这笔代价不
|
||||
# 出现在任何单根的 queue_ms 或 inner_ms 里
|
||||
self.clear_hist: deque = deque(maxlen=60)
|
||||
self._clear_cur: dict[int, float] = {}
|
||||
# 成交监听:已挂上的币,以及必须持有的 forwarder 强引用
|
||||
# (PubSub 只存弱引用,不持有的话监听会被 GC 静默摘掉)
|
||||
self._hooked: set[str] = set()
|
||||
self._trade_fwd: dict = {}
|
||||
self.n_signal = 0
|
||||
self.n_pass = 0
|
||||
self.n_bars = 0
|
||||
# lag 探针的滚动窗口,逐币独立:一个币的行情退化不该连累其他币
|
||||
self.lag_hist: dict[str, deque] = {
|
||||
s: deque(maxlen=LAG_WINDOW) for s in SYMS}
|
||||
self.lag_ok: dict[str, bool] = {s: True for s in SYMS}
|
||||
d = out_dir()
|
||||
# 追加模式:长跑期间若重启,已收集的样本不该被清掉
|
||||
self.f_sig, self.w_sig = _writer(d / "shadow_signals.csv", [
|
||||
"site", "sym", "kline_ts", "direction",
|
||||
"h1_agree", "ladder_ok", "gate_ok", "pass_all", "lag_ok",
|
||||
"atr_pct", "atr_bp",
|
||||
"t_close_ms", "t_data_ms", "t_signal_ms",
|
||||
"lag_data_ms", "lag_signal_ms",
|
||||
"delay_label", "delay_ms", "book_ts", "book_lag_ms",
|
||||
"notional", "base_amt",
|
||||
"baseline_px", "mid", "best_px", "fill_px", "filled", "depth_ok",
|
||||
"slip_bp", "drift_bp", "spread_bp", "impact_bp"])
|
||||
self.f_lat, self.w_lat = _writer(d / "shadow_latency.csv", [
|
||||
"site", "sym", "kline_ts", "t_close_ms", "t_data_ms", "t_signal_ms",
|
||||
"lag_data_ms", "lag_signal_ms",
|
||||
# compute_ms 含排队;queue_ms/inner_ms 把它拆开,用来判断加核有没有用
|
||||
"compute_ms", "queue_ms", "inner_ms",
|
||||
"n_bars", "n_hits",
|
||||
"n_pass", "atr_bp", "lag_med_ms", "lag_ok",
|
||||
"stream_bars"])
|
||||
# 无条件漂移:每根都记,用来和信号根上的条件漂移对照
|
||||
self.f_drf, self.w_drf = _writer(d / "shadow_drift.csv", [
|
||||
"site", "sym", "kline_ts", "delay_label", "delay_ms",
|
||||
"book_ts", "book_lag_ms", "baseline_px", "mid", "drift_bp_long"])
|
||||
# 完整深度。挂在无条件漂移那条路径上,所以每根 K 线的四个固定延迟点
|
||||
# 都有一份,信号根上再补一份 actual 点
|
||||
self.blog = BookLog(d / "shadow_books.jsonl.gz")
|
||||
self.tape = TapeLog(d / "shadow_tape.jsonl.gz")
|
||||
|
||||
def _spawn(self, coro, what: str) -> None:
|
||||
"""起一个后台任务,但异常要吼出来。
|
||||
|
||||
裸 create_task 的异常只在对象被 GC 时才由 asyncio 打一句
|
||||
「Task exception was never retrieved」,很容易整晚没人发现。
|
||||
这套东西最怕的就是不崩不报错的静默退化。
|
||||
"""
|
||||
async def guard():
|
||||
try:
|
||||
await coro
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as e:
|
||||
import traceback
|
||||
print(f" [异常] {what}: {type(e).__name__}: {e}", flush=True)
|
||||
traceback.print_exc()
|
||||
|
||||
t = asyncio.create_task(guard())
|
||||
self._tasks.add(t)
|
||||
t.add_done_callback(self._tasks.discard)
|
||||
|
||||
def _restart_pool(self) -> None:
|
||||
"""进程池坏了之后重建。
|
||||
|
||||
用 spawn 而非 fork:此刻进程里已经有活跃的 WS 连接,fork 会把连接
|
||||
状态一起复制进子进程。spawn 启动慢几秒,但只在故障时走这条路。
|
||||
"""
|
||||
import multiprocessing
|
||||
self.n_broken += 1
|
||||
try:
|
||||
self.pool.shutdown(wait=False, cancel_futures=True)
|
||||
except Exception:
|
||||
pass
|
||||
self.pool = ProcessPoolExecutor(
|
||||
max_workers=self.workers,
|
||||
mp_context=multiprocessing.get_context("spawn"))
|
||||
print(f" [进程池] 已重建(第 {self.n_broken} 次)", flush=True)
|
||||
|
||||
# ---------- 启动 ----------
|
||||
|
||||
async def start(self) -> None:
|
||||
from hummingbot.connector.derivative.bitget_perpetual.bitget_perpetual_derivative import (
|
||||
BitgetPerpetualDerivative,
|
||||
)
|
||||
from patched_candles import (PatchedBitgetPerpetualCandles,
|
||||
assert_patch_effective)
|
||||
|
||||
# 覆盖失效是静默的(悄悄退回慢 1.06 秒,不报错),所以在启动就验一次
|
||||
assert_patch_effective()
|
||||
|
||||
for s in SYMS:
|
||||
self.feeds_l[s] = PatchedBitgetPerpetualCandles(
|
||||
f"{s}-USDT", "1m", LTF_BARS)
|
||||
self.feeds_h[s] = PatchedBitgetPerpetualCandles(
|
||||
f"{s}-USDT", "5m", HTF_BARS)
|
||||
self.feeds_l[s].start()
|
||||
self.feeds_h[s].start()
|
||||
print(f"[影子] {SYMS} · 1m×{LTF_BARS} + 5m×{HTF_BARS} · "
|
||||
f"{self.workers} 个计算进程", flush=True)
|
||||
|
||||
# 只取公开数据:无密钥 + trading_required=False
|
||||
self.connector = BitgetPerpetualDerivative(
|
||||
bitget_perpetual_api_key="", bitget_perpetual_secret_key="",
|
||||
bitget_perpetual_passphrase="",
|
||||
trading_pairs=[f"{s}-USDT" for s in SYMS],
|
||||
trading_required=False)
|
||||
await self.connector.start_network()
|
||||
print(" 连接器已启动,等盘口与历史回填", flush=True)
|
||||
self._hook_trades()
|
||||
|
||||
t0 = time.time()
|
||||
while time.time() - t0 < 600:
|
||||
ready = all(f.ready for f in
|
||||
list(self.feeds_l.values()) + list(self.feeds_h.values()))
|
||||
books = all(self._snapshot(s) is not None for s in SYMS)
|
||||
if ready and books:
|
||||
break
|
||||
await asyncio.sleep(1)
|
||||
print(f" 就绪 {time.time() - t0:.1f}s · "
|
||||
f"1m {[len(self.feeds_l[s]._candles) for s in SYMS]} 根 · "
|
||||
f"5m {[len(self.feeds_h[s]._candles) for s in SYMS]} 根", flush=True)
|
||||
|
||||
def _hook_trades(self) -> None:
|
||||
"""给每个盘口挂成交监听。
|
||||
|
||||
盘口对象可能还没建好(订阅是异步的),所以挂不上的先记下来,由
|
||||
watch_bars 那圈重试;一直挂不上会在心跳里显示成交笔数为 0。
|
||||
"""
|
||||
from hummingbot.core.event.event_forwarder import EventForwarder
|
||||
from hummingbot.core.event.events import OrderBookEvent
|
||||
from hummingbot.core.data_type.common import TradeType
|
||||
|
||||
def make(sym: str):
|
||||
def cb(ev) -> None:
|
||||
self.tape.add(sym, ev.type == TradeType.BUY,
|
||||
float(ev.price), float(ev.amount))
|
||||
return EventForwarder(cb)
|
||||
|
||||
for s in SYMS:
|
||||
if s in self._hooked: # 重复挂会让同一笔成交被记两次
|
||||
continue
|
||||
try:
|
||||
ob = self.connector.get_order_book(f"{s}-USDT")
|
||||
except Exception:
|
||||
ob = None
|
||||
if ob is None:
|
||||
continue
|
||||
fwd = make(s)
|
||||
ob.add_listener(OrderBookEvent.TradeEvent, fwd)
|
||||
self._trade_fwd[s] = fwd
|
||||
self._hooked.add(s)
|
||||
miss = [s for s in SYMS if s not in self._hooked]
|
||||
print(f" 成交流已挂 {sorted(self._hooked)}"
|
||||
+ (f",待重试 {miss}" if miss else ""), flush=True)
|
||||
|
||||
def _snapshot(self, sym: str):
|
||||
try:
|
||||
ob = self.connector.get_order_book(f"{sym}-USDT")
|
||||
except Exception:
|
||||
return None
|
||||
if ob is None:
|
||||
return None
|
||||
# 存成 apply_numpy_snapshot 要的 [价, 量, update_id] 三列,
|
||||
# 回查时才能直接搭 OrderBook 调框架的吃单查询
|
||||
bids = np.array([(float(r.price), float(r.amount), i)
|
||||
for i, (r, _) in enumerate(
|
||||
zip(ob.bid_entries(), range(BOOK_DEPTH)))])
|
||||
asks = np.array([(float(r.price), float(r.amount), i)
|
||||
for i, (r, _) in enumerate(
|
||||
zip(ob.ask_entries(), range(BOOK_DEPTH)))])
|
||||
if not len(bids) or not len(asks):
|
||||
return None
|
||||
return bids, asks
|
||||
|
||||
# ---------- 三个循环 ----------
|
||||
|
||||
async def sample_books(self) -> None:
|
||||
"""按截止时刻补睡,且对齐到墙钟 100ms 网格。
|
||||
|
||||
补睡是因为「干完活再睡固定时长」的实际周期是 100ms 加采样耗时,
|
||||
名义 10Hz 到不了 10Hz。
|
||||
|
||||
对齐是因为回查目标都是 `kline_ts + n×500ms`,而 kline_ts 是整分钟,
|
||||
所以目标必然落在墙钟 100ms 的整数倍上。采样相位若随启动时刻漂移,
|
||||
每个回查点就会固定晚半个采样周期(实测 52ms)——四个固定延迟点
|
||||
同向偏置,虽不改曲线形状,但白白多算了 50ms 的漂移。
|
||||
"""
|
||||
period = 1.0 / BOOK_HZ
|
||||
nxt = math.ceil(time.time() / period) * period
|
||||
while not self.stop.is_set():
|
||||
t = int(time.time() * 1000)
|
||||
for s in SYMS:
|
||||
snap = self._snapshot(s)
|
||||
if snap:
|
||||
self.books.push(s, t, snap[0], snap[1])
|
||||
nxt += period
|
||||
await asyncio.sleep(max(0.0, nxt - time.time()))
|
||||
|
||||
async def watch_bars(self) -> None:
|
||||
last = {s: (int(self.feeds_l[s]._candles[-1][0])
|
||||
if len(self.feeds_l[s]._candles) else None) for s in SYMS}
|
||||
while not self.stop.is_set():
|
||||
for s in SYMS:
|
||||
c = self.feeds_l[s]._candles
|
||||
if not len(c):
|
||||
continue
|
||||
newest = int(c[-1][0])
|
||||
if last[s] is not None and newest > last[s]:
|
||||
t_data = int(time.time() * 1000)
|
||||
kts = newest * 1000 if newest < 1e12 else newest
|
||||
# 刚收盘那根的成交聚合先落盘,再算信号
|
||||
self.tape.flush_bar(s, kts)
|
||||
if len(self._hooked) < len(SYMS):
|
||||
self._hook_trades() # 换根时才重试,避免重复挂
|
||||
self._spawn(self.on_bar(s, kts, t_data), f"on_bar {s}")
|
||||
last[s] = newest
|
||||
await asyncio.sleep(0.01)
|
||||
|
||||
async def on_bar(self, sym: str, kline_ts: int, t_data: int) -> None:
|
||||
"""kline_ts 是新一根的开盘时刻,也就是上一根的收盘时刻 t_close。"""
|
||||
df_l = hb_to_research(self.feeds_l[sym].candles_df)
|
||||
df_h = hb_to_research(self.feeds_h[sym].candles_df)
|
||||
# 末行是刚开始的那根,未收盘,必须剔除,否则等于用未来数据
|
||||
df_l = df_l[df_l["timestamp"] < kline_ts]
|
||||
df_h = df_h[df_h["timestamp"] < kline_ts]
|
||||
# WS 重连的瞬间 feed 的 deque 可能是空的。放行的话 worker 会抛
|
||||
# 「DataFrame for 1m is empty」,白占一个计算槽(币数超核数时这笔
|
||||
# 代价会推迟后面所有币),而报错文本还会让人以为是缺历史数据
|
||||
if len(df_l) < MIN_BARS or len(df_h) < MIN_BARS:
|
||||
print(f" [{sym}] 窗口过短(1m {len(df_l)} / 5m {len(df_h)} 根),"
|
||||
f"跳过本根。feed 大概在重连", flush=True)
|
||||
return
|
||||
baseline = self._new_bar_open(sym, kline_ts)
|
||||
|
||||
lag_med, lag_ok = self._probe_lag(sym, t_data - kline_ts)
|
||||
|
||||
t0 = time.perf_counter()
|
||||
payload = (df_l[NUM_COLS].values.tolist(),
|
||||
df_h[NUM_COLS].values.tolist(), baseline, time.time(), sym)
|
||||
loop = asyncio.get_running_loop()
|
||||
from shadow_signal import compute_packed
|
||||
try:
|
||||
res = await loop.run_in_executor(self.pool, compute_packed, payload)
|
||||
except BrokenProcessPool as e:
|
||||
# 不重建的话,之后每一根都会走到这里,采集静默停摆到跑完为止
|
||||
print(f" [{sym}] 进程池损坏 {e},重建后跳过本根", flush=True)
|
||||
self._restart_pool()
|
||||
return
|
||||
compute_ms = int((time.perf_counter() - t0) * 1000)
|
||||
t_signal = int(time.time() * 1000)
|
||||
if res.get("queue_ms") is not None:
|
||||
self.q_hist.append(res["queue_ms"])
|
||||
if res.get("inner_ms") is not None:
|
||||
self.i_hist.append(res["inner_ms"])
|
||||
# 同一 kline_ts 上取各币最大值即该时刻的清空耗时;只保留最近几个
|
||||
# 时刻,否则这个 dict 会随运行时长无界增长
|
||||
cur = self._clear_cur
|
||||
cur[kline_ts] = max(cur.get(kline_ts, 0.0), float(compute_ms))
|
||||
if len(cur) > 3:
|
||||
done = min(cur)
|
||||
self.clear_hist.append(cur.pop(done))
|
||||
|
||||
hits = res.get("hits", [])
|
||||
atr_pct = res.get("atr_pct")
|
||||
atr_bp = round(atr_pct * 1e4, 3) if atr_pct else ""
|
||||
n_pass = sum(h["pass_all"] for h in hits)
|
||||
|
||||
self.n_bars += 1
|
||||
self.w_lat.writerow({
|
||||
"sym": sym, "kline_ts": kline_ts, "t_close_ms": kline_ts,
|
||||
"t_data_ms": t_data, "t_signal_ms": t_signal,
|
||||
"lag_data_ms": t_data - kline_ts,
|
||||
"lag_signal_ms": t_signal - kline_ts,
|
||||
"compute_ms": compute_ms,
|
||||
"queue_ms": res.get("queue_ms"), "inner_ms": res.get("inner_ms"),
|
||||
"n_bars": res.get("n_bars", 0),
|
||||
"n_hits": len(hits), "n_pass": n_pass, "atr_bp": atr_bp,
|
||||
"lag_med_ms": lag_med, "lag_ok": int(lag_ok),
|
||||
# 增量流当前窗口。恒等于 2001 说明缺口判定在每根都
|
||||
# 回退重建,增量静默失效——只从耗时上看不出是哪一环
|
||||
"stream_bars": res.get("stream_bars")})
|
||||
self.f_lat.flush()
|
||||
|
||||
if baseline is not None and np.isfinite(baseline):
|
||||
# 无条件漂移:每根都记,不管有没有信号
|
||||
self._spawn(self._drift_later(sym, kline_ts, baseline),
|
||||
f"drift {sym}")
|
||||
|
||||
if res.get("error"):
|
||||
print(f" [{sym}] 信号计算出错 {res['error']}", flush=True)
|
||||
return
|
||||
if not hits:
|
||||
return
|
||||
if baseline is None or not np.isfinite(baseline):
|
||||
print(f" [{sym}] 有信号但拿不到次根开盘价,跳过", flush=True)
|
||||
return
|
||||
|
||||
for h in hits:
|
||||
self.n_signal += 1
|
||||
self.n_pass += h["pass_all"]
|
||||
mark = "★" if h["pass_all"] else "·"
|
||||
print(f" {mark} [{sym}] {kline_ts} 方向 {h['direction']:+d} "
|
||||
f"同向{h['h1_agree']} 阶梯{h['ladder_ok']} 门控{h['gate_ok']} "
|
||||
f"(ATR {atr_bp or 'na'}bp) · 数据 {t_data - kline_ts}ms "
|
||||
f"信号 {t_signal - kline_ts}ms", flush=True)
|
||||
# 最远的回查点在 t_close+5s,此刻尚未发生;等它过去再一次性落盘
|
||||
self._spawn(
|
||||
self._record_later(sym, kline_ts, h, t_data, t_signal,
|
||||
baseline, atr_pct, lag_ok),
|
||||
f"record {sym}")
|
||||
# 手工执行的推送。只推过全部滤网的,且 lag 退化时不推——那与
|
||||
# 「停开新仓」是同一条规则,不能只在自动化里执行
|
||||
if h["pass_all"] and atr_pct:
|
||||
if not lag_ok:
|
||||
print(f" [TG] {sym} lag 退化,按停开新仓规则不推",
|
||||
flush=True)
|
||||
else:
|
||||
# 总线先写、推送后发。写盘是同步的且已 fsync,实盘据此
|
||||
# 下单;推送要走网络,不能让它的延迟挡在下单前面
|
||||
signal_bus.emit(sym, kline_ts, h["direction"],
|
||||
float(baseline), float(atr_pct),
|
||||
t_data - kline_ts)
|
||||
self._spawn(
|
||||
tg_notify.push_signal(
|
||||
sym, h["direction"], float(baseline),
|
||||
float(atr_pct), kline_ts, t_data - kline_ts),
|
||||
f"tg {sym}")
|
||||
|
||||
def _probe_lag(self, sym: str, lag_ms: int) -> tuple[float, bool]:
|
||||
"""记一根的到达延迟,返回 (滚动中位数, 该币是否健康)。
|
||||
|
||||
不健康时应停止开新仓;影子期不下单,故只落到 lag_ok 字段并告警。
|
||||
"""
|
||||
self.lag_hist[sym].append(lag_ms)
|
||||
ok = lag_healthy(self.lag_hist[sym])
|
||||
med = float(np.median(self.lag_hist[sym]))
|
||||
if ok != self.lag_ok[sym]:
|
||||
state = "恢复" if ok else f"退化,超 {LAG_ALARM_MS:.0f}ms 阈值,停开新仓"
|
||||
print(f" [lag] {sym} {state}:近 {len(self.lag_hist[sym])} 根"
|
||||
f"中位 {med:.0f}ms", flush=True)
|
||||
self.lag_ok[sym] = ok
|
||||
return round(med, 1), ok
|
||||
|
||||
async def _wait_for_delays(self, kline_ts: int) -> None:
|
||||
"""最远回查点是 t_close+5s,等它过去(多留 0.5s 给采样)。"""
|
||||
wait = (kline_ts + int(max(DELAYS_S) * 1000) + 500) / 1000.0 - time.time()
|
||||
if wait > 0:
|
||||
await asyncio.sleep(wait)
|
||||
|
||||
async def _record_later(self, sym: str, kline_ts: int, hit: dict,
|
||||
t_data: int, t_signal: int, baseline: float,
|
||||
atr_pct: float | None, lag_ok: bool) -> None:
|
||||
await self._wait_for_delays(kline_ts)
|
||||
self._record(sym, kline_ts, hit, t_data, t_signal, baseline,
|
||||
atr_pct, lag_ok)
|
||||
|
||||
async def _drift_later(self, sym: str, kline_ts: int,
|
||||
baseline: float) -> None:
|
||||
await self._wait_for_delays(kline_ts)
|
||||
for label, delay_ms in self._points(None):
|
||||
target = kline_ts + delay_ms
|
||||
snap = self.books.at(sym, target)
|
||||
if snap is None:
|
||||
continue
|
||||
book_ts, bids, asks = snap
|
||||
mid = (float(bids[0][0]) + float(asks[0][0])) / 2.0
|
||||
self.blog.write(sym, kline_ts, label, delay_ms, target,
|
||||
book_ts, bids, asks)
|
||||
self.w_drf.writerow({
|
||||
"sym": sym, "kline_ts": kline_ts, "delay_label": label,
|
||||
"delay_ms": delay_ms, "book_ts": book_ts,
|
||||
"book_lag_ms": book_ts - target, "baseline_px": baseline,
|
||||
"mid": mid,
|
||||
"drift_bp_long": round((mid - baseline) / baseline * 1e4, 4)})
|
||||
self.f_drf.flush()
|
||||
self.blog.flush() # 每根冲刷一次,进程被杀最多丢一根
|
||||
|
||||
@staticmethod
|
||||
def _points(t_signal_delay: int | None) -> list[tuple[str, int]]:
|
||||
pts = [(f"{d}s", int(d * 1000)) for d in DELAYS_S]
|
||||
if t_signal_delay is not None:
|
||||
pts.insert(0, ("actual", t_signal_delay))
|
||||
return pts
|
||||
|
||||
def _new_bar_open(self, sym: str, kline_ts: int) -> float | None:
|
||||
"""次根开盘价 = 回测假设的成交价。"""
|
||||
c = self.feeds_l[sym]._candles
|
||||
if not len(c):
|
||||
return None
|
||||
row = c[-1]
|
||||
ts = int(row[0])
|
||||
ts = ts * 1000 if ts < 1e12 else ts
|
||||
return float(row[1]) if ts == kline_ts else None
|
||||
|
||||
def _record(self, sym: str, kline_ts: int, hit: dict,
|
||||
t_data: int, t_signal: int, baseline: float,
|
||||
atr_pct: float | None, lag_ok: bool) -> None:
|
||||
d_sign = hit["direction"]
|
||||
|
||||
for label, delay_ms in self._points(t_signal - kline_ts):
|
||||
target = kline_ts + delay_ms
|
||||
snap = self.books.at(sym, target)
|
||||
if snap is None:
|
||||
continue
|
||||
book_ts, bids, asks = snap
|
||||
best_bid, best_ask = float(bids[0][0]), float(asks[0][0])
|
||||
mid = (best_bid + best_ask) / 2.0
|
||||
best_px = best_ask if d_sign > 0 else best_bid
|
||||
ob = book_from(bids, asks)
|
||||
if label == "actual":
|
||||
# 四个固定点已由无条件漂移那条路径落过,只补这一个
|
||||
self.blog.write(sym, kline_ts, label, delay_ms, target,
|
||||
book_ts, bids, asks)
|
||||
|
||||
for notional in NOTIONALS:
|
||||
# 名义额按基准价折成基础币再下单——真实委托是基础币计价的,
|
||||
# 框架的 get_vwap_for_volume 也收基础币量。名义额那一栏留着
|
||||
# 是为了跨币可比(1 BTC 和 1 SOL 没法横向比)
|
||||
base_amt = notional / baseline
|
||||
r = ob.get_vwap_for_volume(d_sign > 0, base_amt)
|
||||
fill = float(r.result_price)
|
||||
depth_ok = int(float(r.result_volume) >= base_amt * 0.999)
|
||||
if not np.isfinite(fill):
|
||||
# 25 档吃不下这个量,框架直接给 nan。记一行标明深度不足,
|
||||
# 免得「某个仓位档在薄盘时段整段消失」看不出来
|
||||
self.w_sig.writerow({
|
||||
"sym": sym, "kline_ts": kline_ts, "direction": d_sign,
|
||||
"h1_agree": hit["h1_agree"],
|
||||
"ladder_ok": hit["ladder_ok"],
|
||||
"gate_ok": hit["gate_ok"], "pass_all": hit["pass_all"],
|
||||
"lag_ok": int(lag_ok),
|
||||
"atr_pct": atr_pct if atr_pct else "",
|
||||
"atr_bp": round(atr_pct * 1e4, 3) if atr_pct else "",
|
||||
"t_close_ms": kline_ts, "t_data_ms": t_data,
|
||||
"t_signal_ms": t_signal,
|
||||
"lag_data_ms": t_data - kline_ts,
|
||||
"lag_signal_ms": t_signal - kline_ts,
|
||||
"delay_label": label, "delay_ms": delay_ms,
|
||||
"book_ts": book_ts, "book_lag_ms": book_ts - target,
|
||||
"notional": notional, "base_amt": round(base_amt, 8),
|
||||
"baseline_px": baseline, "mid": mid,
|
||||
"best_px": best_px, "fill_px": "",
|
||||
"filled": round(float(r.result_volume), 8),
|
||||
"depth_ok": 0, "slip_bp": "", "drift_bp": "",
|
||||
"spread_bp": "", "impact_bp": ""})
|
||||
continue
|
||||
slip = d_sign * (fill - baseline) / baseline * 1e4
|
||||
drift = d_sign * (mid - baseline) / baseline * 1e4
|
||||
spread = d_sign * (best_px - mid) / mid * 1e4
|
||||
impact = d_sign * (fill - best_px) / best_px * 1e4
|
||||
self.w_sig.writerow({
|
||||
"sym": sym, "kline_ts": kline_ts,
|
||||
"direction": d_sign, "h1_agree": hit["h1_agree"],
|
||||
"ladder_ok": hit["ladder_ok"], "gate_ok": hit["gate_ok"],
|
||||
"pass_all": hit["pass_all"], "lag_ok": int(lag_ok),
|
||||
"atr_pct": atr_pct if atr_pct else "",
|
||||
"atr_bp": round(atr_pct * 1e4, 3) if atr_pct else "",
|
||||
"t_close_ms": kline_ts, "t_data_ms": t_data,
|
||||
"t_signal_ms": t_signal,
|
||||
"lag_data_ms": t_data - kline_ts,
|
||||
"lag_signal_ms": t_signal - kline_ts,
|
||||
"delay_label": label, "delay_ms": delay_ms,
|
||||
"book_ts": book_ts, "book_lag_ms": book_ts - target,
|
||||
"notional": notional, "base_amt": round(base_amt, 8),
|
||||
"baseline_px": baseline,
|
||||
"mid": mid, "best_px": best_px, "fill_px": fill,
|
||||
"filled": round(float(r.result_volume), 8),
|
||||
"depth_ok": depth_ok,
|
||||
"slip_bp": round(slip, 4), "drift_bp": round(drift, 4),
|
||||
"spread_bp": round(spread, 4),
|
||||
"impact_bp": round(impact, 4)})
|
||||
self.f_sig.flush()
|
||||
|
||||
async def heartbeat(self) -> None:
|
||||
while not self.stop.is_set():
|
||||
await asyncio.sleep(300)
|
||||
depth = {s: len(self.books.buf[s]) for s in SYMS}
|
||||
lag = {s: (f"{np.median(h):.0f}ms" if h else "na")
|
||||
+ ("" if self.lag_ok[s] else "!")
|
||||
for s, h in self.lag_hist.items()}
|
||||
print(f" [心跳] 已处理 {self.n_bars} 根 · 命中 {self.n_signal} 个"
|
||||
f"(过全部滤网 {self.n_pass}) · lag {lag} · 盘口缓冲 {depth}"
|
||||
f" · 回查超容差 {self.books.n_stale} 次"
|
||||
f" · 在途任务 {len(self._tasks)}"
|
||||
f" · 盘口落盘 {self.blog.n} 份"
|
||||
f" · 成交 {self.tape.n_trades} 笔{'' if self.tape.n_trades else ' ⚠监听未生效'}",
|
||||
flush=True)
|
||||
if self.q_hist and self.i_hist:
|
||||
q, i = float(np.median(self.q_hist)), float(np.median(self.i_hist))
|
||||
# 建议要看绝对量级:lean + 新引擎后纯计算约 128ms,此时再提
|
||||
clear = float(np.median(self.clear_hist)) if self.clear_hist \
|
||||
else float("nan")
|
||||
print(f" [计算] 每币排队 {q:.0f}ms · 纯计算 {i:.0f}ms · "
|
||||
f"清空全部 {len(SYMS)} 币 {clear:.0f}ms"
|
||||
f"(worker {self.workers} / 核 {CORES})", flush=True)
|
||||
print(f" → {self._compute_verdict(q, i, clear)}", flush=True)
|
||||
# 五分钟一根都没进来,说明管道断了。不喊一声就只能靠人翻日志
|
||||
if self.n_bars == self._hb_last_bars:
|
||||
print(f" ⚠ [停滞] 距上次心跳未处理任何 K 线"
|
||||
f"(进程池重建 {self.n_broken} 次),管道可能已断",
|
||||
flush=True)
|
||||
self._hb_last_bars = self.n_bars
|
||||
|
||||
def _compute_verdict(self, q: float, i: float, clear: float) -> str:
|
||||
"""给出唯一可行的出路,而不是「哪一项数字更大」。
|
||||
|
||||
旧版比逐根的 q 与 i,结构上错了两处:
|
||||
|
||||
1. 判据错。真正要紧的是**一个收盘时刻清空所有币要多久**(clear),
|
||||
不是单币的 q 或 i。所有币同一秒收盘,币数超过 worker 数时后面的
|
||||
币必然串行等待,而这笔代价不出现在任何单根的 q 或 i 里。
|
||||
2. 出路错。「排队为主 → 加核」只在还有空闲核时成立。worker 已等于
|
||||
核数时,加 worker 不会增加吞吐——CPU 密集的活变不出来,只会把
|
||||
等待从 queue_ms 挪到 inner_ms。十币实测正是如此:inner 被争抢从
|
||||
144ms 抬到 192ms,反而超过 queue 135ms,于是判定落到「量级已低、
|
||||
无需优化」,而此时最后一个币已经落在 1376ms。
|
||||
|
||||
所以币数超过核数时,加 worker 不增吞吐。出路有两级:先上增量把真实计算
|
||||
压下来;增量之后剩的是争抢放大(实测 3.3 倍,§5.72),那一级只能加核或
|
||||
减币,继续改算法收益有限。
|
||||
"""
|
||||
if not np.isfinite(clear):
|
||||
return "样本不足,暂不判定"
|
||||
if clear < 400:
|
||||
return f"清空 {clear:.0f}ms,宽裕,无需优化"
|
||||
if self.workers < CORES and q > i:
|
||||
return (f"排队为主且还有 {CORES - self.workers} 个空闲核 → "
|
||||
f"--workers 加到 {CORES}")
|
||||
if len(SYMS) <= CORES:
|
||||
return f"清空 {clear:.0f}ms 偏高,但币数未超核数,先查别的争抢"
|
||||
# 币数超核数:加 worker 不增吞吐,只能压单币耗时。但要看增量开没开,
|
||||
# 否则会在增量已生效时继续推荐「走增量」——上线后实测踩到过
|
||||
if not INCR_ON:
|
||||
return (f"币数 {len(SYMS)} > 核数 {CORES},加 worker 无用(CPU 密集)"
|
||||
f"。压单币耗时 → 开 SHADOW_INCR=1 走增量(实测 3.56x)")
|
||||
# 增量已生效时(§5.72 口径对齐后):单币 100ms ≈ 信号链 30ms + 追加
|
||||
# 2.81 根 37ms + 争抢 33ms。争抢只有 1.49x,加核收益有限;而追加那 37ms
|
||||
# 里约 22ms 纯属浪费——symbol 随机落 worker,各缓存都漏掉对方处理的根
|
||||
return (f"币数 {len(SYMS)} > 核数 {CORES},增量已生效,加 worker 无用"
|
||||
f"(worker 已等于核数)。单币 {i:.0f}ms 里争抢只占约 1.5x,"
|
||||
f"最便宜的一刀是按币绑定 worker(每次只追 1 根,省约三分之一),"
|
||||
f"其次是 add_indicators 增量化")
|
||||
|
||||
async def run(self) -> None:
|
||||
await self.start()
|
||||
tasks = [asyncio.create_task(self.sample_books()),
|
||||
asyncio.create_task(self.watch_bars()),
|
||||
asyncio.create_task(self.heartbeat())]
|
||||
while time.time() < self.deadline:
|
||||
await asyncio.sleep(5)
|
||||
self.stop.set()
|
||||
for t in tasks:
|
||||
t.cancel()
|
||||
await asyncio.gather(*tasks, return_exceptions=True)
|
||||
for f in list(self.feeds_l.values()) + list(self.feeds_h.values()):
|
||||
f.stop()
|
||||
await self.connector.stop_network()
|
||||
for f in (self.f_sig, self.f_lat, self.f_drf):
|
||||
f.close()
|
||||
self.blog.close()
|
||||
self.tape.close()
|
||||
print(f"\n收工:{self.n_bars} 根 · {self.n_signal} 个信号"
|
||||
f"(过全部滤网 {self.n_pass})", flush=True)
|
||||
|
||||
|
||||
async def main_async(workers: int, hours: float, pool) -> None:
|
||||
sh = Shadow(workers, hours)
|
||||
sh.pool = pool
|
||||
await sh.run()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
global SYMS
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--hours", type=float, default=24.0)
|
||||
ap.add_argument("--workers", type=int, default=2)
|
||||
# 币数直接决定排队:所有币在同一秒收盘,worker 少于币数就必然排队,
|
||||
# 最后一个币的信号要等 ceil(n/worker) 轮计算。TRX 不在默认池里——
|
||||
# 实盘口径 208 天只有 5 笔,ATR 门控几乎全刷掉(HANDOFF §step48)
|
||||
ap.add_argument("--syms", default=",".join(SYMS),
|
||||
help="逗号分隔。十币池:BTC,ETH,SOL,BNB,XRP,DOGE,ADA,"
|
||||
"AVAX,LINK,LTC")
|
||||
a = ap.parse_args()
|
||||
SYMS = tuple(s.strip().upper() for s in a.syms.split(",") if s.strip())
|
||||
# 进程池必须在事件循环和任何 WS 连接之前建好:fork 一个已带活跃 socket
|
||||
# 的进程会把连接状态一起复制过去,后果不可预测
|
||||
with ProcessPoolExecutor(max_workers=a.workers) as pool:
|
||||
asyncio.run(main_async(a.workers, a.hours, pool))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,314 @@
|
||||
"""影子交易器的报表:延迟门槛 + 滑点对延迟曲线。
|
||||
|
||||
### 判据常数一律从研究侧 import,不在这里写死
|
||||
|
||||
`BUDGET_BP` 等常数留在 `research/lib/shadow_budget.py`。理由是这些数会变——
|
||||
2026-08-27 一天之内预算就动了四次(3.91 → 11.06 → 14.25 → 15.19bp),
|
||||
费率也改了一次。本文件曾经写死过 BTC −0.13 / ETH 4.02 / SOL 2.92,那三个数
|
||||
由六处差异叠加而来(只有同向没有阶梯、费率按 6bp 双边 taker、TP=3.0、
|
||||
余量没除 taker 名义额、无 ATR 门控,且 BTC/ETH/SOL 恰是 ATR 最低的三个币)。
|
||||
正确值是 8.58 / 20.64 / 16.83——**ETH 差了五倍**。
|
||||
|
||||
这件事要紧是因为下面的判读是自动停机开关:用 4.02 当 ETH 的预算,真实滑点
|
||||
只要到 2.4bp 就会报「需要压延迟或放弃」,会误杀一个可行的策略。
|
||||
|
||||
### 什么时候能判什么
|
||||
|
||||
| | 一天的样本量 | 够不够 |
|
||||
|---|---|---|
|
||||
| 延迟 | 1440 根/币 | 够,统计上很厚 |
|
||||
| 滑点 | 门控后 4~6 笔/天 | **不够**,判据要 30 笔以上,即一周起步 |
|
||||
|
||||
所以首日只能判延迟和管道通不通。滑点那一节在样本不足时会明说。
|
||||
|
||||
### 延迟门槛(提前止损用)
|
||||
|
||||
若**总延迟已令预期漂移超过预算**,说明方案在这台机器上就不成立,不必等
|
||||
两周样本再停。漂移按随机游走折算 σ_1m · √(t/60)。这是下限——入场条件是
|
||||
「收盘突破转强」,那一刻价格正朝我们方向跑,延迟造成的是系统性追价,
|
||||
不会正负抵消。所以实测滑点理应比折算值更差,两者对照本身就是个校验。
|
||||
|
||||
### 滑点对延迟曲线
|
||||
|
||||
把延迟当自变量:0.5s / 1s / 2s / 5s 各一个滑点值,外加「actual」= 本机实际
|
||||
算完的时刻。同信号内的受控对比,能直接读出「若延迟压到 X 秒,滑点是多少」,
|
||||
决策不被当前实现拖累。
|
||||
|
||||
.venv/bin/python research/live/shadow_report.py
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
OUT = HERE.parent / "out"
|
||||
sys.path.insert(0, str(HERE.parent))
|
||||
|
||||
from lib.shadow_budget import ( # noqa: E402
|
||||
ATR_GATE_BP, BUDGET_PORTFOLIO_2026, LAG_ALARM_MS, budget_of, lag_healthy,
|
||||
verdict,
|
||||
)
|
||||
|
||||
SYMS = ("BTC", "ETH", "SOL")
|
||||
ORDER = ["0.5s", "1.0s", "2.0s", "5.0s", "actual"]
|
||||
MIN_N = 30 # 滑点判据的最低笔数,低于此只报数不下结论
|
||||
|
||||
|
||||
def vol_bp() -> dict[str, float]:
|
||||
v = {}
|
||||
for s in SYMS:
|
||||
f = HERE / "cache" / f"bitget_{s}_1m_30d.feather"
|
||||
if not f.exists():
|
||||
f = HERE / "cache" / f"bitget_{s}_1m_210d.feather"
|
||||
if not f.exists():
|
||||
continue
|
||||
c = np.log(pd.read_feather(f)["close"].to_numpy(float))
|
||||
v[s] = float(np.nanstd(np.diff(c)) * 1e4)
|
||||
return v
|
||||
|
||||
|
||||
def lag_health(lat: pd.DataFrame) -> None:
|
||||
"""运行时 lag 探针的回看。补丁后实测 506~642ms,理论下限约 500ms。"""
|
||||
print("\n\n########## 二、lag 探针(>%.0fms 该停开仓)##########"
|
||||
% LAG_ALARM_MS)
|
||||
print(f"{'币':<5}{'根数':>6}{'中位ms':>9}{'P90ms':>8}{'最差30根中位':>14}"
|
||||
f"{'超阈根数':>10}{'判定':>8}")
|
||||
for s in SYMS:
|
||||
g = lat[lat["sym"] == s].sort_values("kline_ts")
|
||||
if g.empty:
|
||||
continue
|
||||
x = g["lag_data_ms"].to_numpy(float)
|
||||
roll = pd.Series(x).rolling(30).median()
|
||||
worst = float(np.nanmax(roll)) if roll.notna().any() else float("nan")
|
||||
ok = lag_healthy(x)
|
||||
print(f"{s:<5}{len(g):>6}{np.median(x):>9.0f}"
|
||||
f"{np.percentile(x, 90):>8.0f}{worst:>14.0f}"
|
||||
f"{int((x > LAG_ALARM_MS).sum()):>10}"
|
||||
f"{'健康' if ok else '退化':>8}")
|
||||
if "lag_ok" in lat.columns:
|
||||
bad = int((lat["lag_ok"] == 0).sum())
|
||||
if bad:
|
||||
print(f"\n 采集期间有 {bad} 根被判不健康,那些根上的信号"
|
||||
f"(lag_ok=0)在真实运行下不会开仓,统计时应排除")
|
||||
|
||||
|
||||
def latency_gate(lat: pd.DataFrame, vols: dict) -> None:
|
||||
print("########## 一、延迟门槛 ##########")
|
||||
span_h = (lat["t_close_ms"].max() - lat["t_close_ms"].min()) / 3.6e6
|
||||
print(f"样本跨度 {span_h:.1f} 小时 · 共 {len(lat)} 根\n")
|
||||
print(f"{'币':<5}{'根数':>6}{'数据ms':>9}{'计算ms':>9}{'总延迟ms':>10}"
|
||||
f"{'P90ms':>8}{'折算漂移bp':>12}{'预算bp':>9}{'占预算':>9}")
|
||||
rows = {}
|
||||
for s in SYMS:
|
||||
g = lat[lat["sym"] == s]
|
||||
if g.empty:
|
||||
continue
|
||||
d = float(np.median(g["lag_data_ms"]))
|
||||
c = float(np.median(g["compute_ms"]))
|
||||
t = float(np.median(g["lag_signal_ms"]))
|
||||
p90 = float(np.percentile(g["lag_signal_ms"], 90))
|
||||
vol = vols.get(s)
|
||||
drift = vol * np.sqrt(t / 60_000) if vol else float("nan")
|
||||
b = budget_of(s)
|
||||
share = drift / b if np.isfinite(b) and b > 0 else float("nan")
|
||||
rows[s] = (drift, b)
|
||||
txt = f"{share * 100:.0f}%" if np.isfinite(share) else "—"
|
||||
print(f"{s:<5}{len(g):>6}{d:>9.0f}{c:>9.0f}{t:>10.0f}{p90:>8.0f}"
|
||||
f"{drift:>12.2f}{b:>9.2f}{txt:>9}")
|
||||
|
||||
print("\n判读(预算来自 lib/shadow_budget,2026 年口径):")
|
||||
for s, (drift, b) in rows.items():
|
||||
if not np.isfinite(b):
|
||||
print(f" {s}: 当前环境预算不足,不作交易标的,仅作延迟参照")
|
||||
elif not np.isfinite(drift):
|
||||
# 不特判的话 nan > b 是 False,会一路落到「尚有空间」说反话
|
||||
print(f" {s}: 缺 1m 波动率缓存,折算不出漂移,无法判读")
|
||||
elif drift > b:
|
||||
print(f" {s}: 折算漂移 {drift:.2f}bp 已超预算 {b:.2f}bp —— 停下改方案")
|
||||
elif drift > b * 0.6:
|
||||
print(f" {s}: 折算漂移 {drift:.2f}bp 吃掉预算 {b:.2f}bp 的六成以上,"
|
||||
f"需要压延迟或放弃")
|
||||
else:
|
||||
print(f" {s}: 折算漂移 {drift:.2f}bp 对预算 {b:.2f}bp 尚有空间,继续收集")
|
||||
|
||||
|
||||
def book_quality(sig: pd.DataFrame, drf: pd.DataFrame) -> None:
|
||||
"""回查到的盘口比目标时刻晚多少。晚太多的已在采集侧丢弃,这里做复核。"""
|
||||
frames = [d for d in (sig, drf) if not d.empty and "book_lag_ms" in d]
|
||||
if not frames:
|
||||
return
|
||||
x = pd.concat([d["book_lag_ms"] for d in frames]).astype(float)
|
||||
print("\n\n########## 二·五、盘口回查质量 ##########")
|
||||
print(f" 回查 {len(x)} 次 · 中位 {x.median():.0f}ms · "
|
||||
f"P90 {np.percentile(x, 90):.0f}ms · 最大 {x.max():.0f}ms")
|
||||
print(f" 10Hz 采样下这个值应在 0~100ms。它是所有延迟点的同向偏置,"
|
||||
f"不改变曲线形状,但要确认没有异常长尾")
|
||||
|
||||
|
||||
def tick_floor(drf: pd.DataFrame) -> None:
|
||||
"""标出哪些币的漂移只是 tick 量化的地板,不是真实漂移。
|
||||
|
||||
中价的最小变动是半个 tick,所以 tick 相对价格粗的币(ADA 半 tick 就有
|
||||
2.34bp、LTC 1.00bp),漂移中位会**精确等于半 tick** 且在所有延迟点上
|
||||
完全相同。这很容易被读成「ADA 漂移 2.34bp」,实际是「测不到更细」。
|
||||
|
||||
方向上是安全的——真实漂移只会更小,所以这些数是上界。但必须标出来,
|
||||
否则会拿一个测量地板去和预算做比较,然后误判某个币不可做。
|
||||
"""
|
||||
if drf.empty or "mid" not in drf.columns:
|
||||
return
|
||||
print("\n ── tick 地板检查")
|
||||
hit = False
|
||||
for sym, g in drf.groupby("sym"):
|
||||
mid = g["mid"].median()
|
||||
# 同一币在各延迟点的漂移若几乎不变,就是被量化了
|
||||
by = g.groupby("delay_label")["drift_bp_long"].apply(
|
||||
lambda x: x.abs().median())
|
||||
if len(by) < 3 or not np.isfinite(mid) or mid <= 0:
|
||||
continue
|
||||
spread = by.max() - by.min()
|
||||
if spread < 0.02 and by.median() > 0.2:
|
||||
hit = True
|
||||
print(f" {sym:<6}漂移在各延迟点恒为 {by.median():.2f}bp"
|
||||
f" → 半 tick 地板,真实漂移低于此值")
|
||||
if not hit:
|
||||
print(" 没有币落在 tick 地板上,漂移数值可直接读")
|
||||
|
||||
|
||||
def drift_split(sig: pd.DataFrame, drf: pd.DataFrame) -> None:
|
||||
"""条件漂移 vs 无条件漂移。两者的差就是「系统性追价」的大小。"""
|
||||
print("\n\n########## 三、条件漂移 vs 无条件漂移 ##########")
|
||||
if drf.empty:
|
||||
print(" 尚无逐根漂移数据(shadow_drift.csv 由本轮起才开始记)")
|
||||
return
|
||||
print(" 无条件 = 每根 K 线,方向未知故取 |漂移|;"
|
||||
"条件 = 信号根按下单方向定号")
|
||||
print(f"\n{'延迟':<8}{'无条件n':>9}{'无条件|漂移|':>14}"
|
||||
f"{'条件n':>7}{'条件漂移':>10}{'追价差':>9}")
|
||||
cond = sig[(sig["notional"] == sig["notional"].min())] if not sig.empty \
|
||||
else sig
|
||||
for lb in ORDER:
|
||||
u = drf[drf["delay_label"] == lb]["drift_bp_long"].abs()
|
||||
c = cond[cond["delay_label"] == lb]["drift_bp"] if not cond.empty \
|
||||
else pd.Series(dtype=float)
|
||||
if u.empty and c.empty:
|
||||
continue
|
||||
um = u.mean() if not u.empty else float("nan")
|
||||
cm = c.mean() if not c.empty else float("nan")
|
||||
print(f"{lb:<8}{len(u):>9}{um:>14.2f}{len(c):>7}{cm:>10.2f}"
|
||||
f"{cm - um:>9.2f}")
|
||||
if len(cond) and len(cond[cond["delay_label"] == "1.0s"]) < MIN_N:
|
||||
print(f"\n 条件侧样本不足 {MIN_N},差值还读不出方向")
|
||||
tick_floor(drf)
|
||||
|
||||
|
||||
def slippage_curve(sig: pd.DataFrame) -> None:
|
||||
print("\n\n########## 四、滑点对延迟曲线 ##########")
|
||||
if sig.empty:
|
||||
print(" 尚无信号样本")
|
||||
return
|
||||
|
||||
def n_of(df):
|
||||
return df.groupby(["sym", "kline_ts", "direction"]).ngroups
|
||||
|
||||
has_flags = "pass_all" in sig.columns
|
||||
if not has_flags:
|
||||
print(" ⚠ 数据来自旧版采集(只有 h1_agree,无阶梯与 ATR 门控)。"
|
||||
"这批不是我们要交易的那批信号,只能作管道验证,不能对预算判读。\n")
|
||||
main_scope, main_name = sig[sig["h1_agree"] == 1], "仅 h1_agree=1(旧口径)"
|
||||
else:
|
||||
# depth_ok=0 是 25 档吃不满该仓位,均价按部分成交算会**低估**冲击
|
||||
ok = (sig["pass_all"] == 1) & (sig["lag_ok"] == 1)
|
||||
if "depth_ok" in sig.columns:
|
||||
thin = int((sig["depth_ok"] == 0).sum())
|
||||
ok &= sig["depth_ok"] == 1
|
||||
if thin:
|
||||
print(f" ({thin} 行深度吃不满,已排除;这些行会低估冲击)")
|
||||
print(f"信号总数 {n_of(sig)} · 同向 {n_of(sig[sig['h1_agree'] == 1])}"
|
||||
f" · 同向+阶梯 "
|
||||
f"{n_of(sig[(sig['h1_agree'] == 1) & (sig['ladder_ok'] == 1)])}"
|
||||
f" · 三项全过 {n_of(sig[sig['pass_all'] == 1])}"
|
||||
f" · 再要求 lag 健康 {n_of(sig[ok])}")
|
||||
print(f"(门控阈值 ATR ≥ {ATR_GATE_BP:.0f}bp,是费率的函数不是市场常数)\n")
|
||||
main_scope = sig[ok]
|
||||
main_name = "三项滤网全过 + lag 健康 + 深度吃满(主口径)"
|
||||
|
||||
scopes = [("全部信号(含不会下单的,仅作提前读数)", sig),
|
||||
(main_name, main_scope)]
|
||||
for scope, sub in scopes:
|
||||
if sub.empty:
|
||||
print(f"--- {scope} ---\n 尚无样本\n")
|
||||
continue
|
||||
print(f"--- {scope} ---")
|
||||
print(f"{'延迟':<8}{'仓位':>9}{'n':>5}{'滑点均值bp':>12}"
|
||||
f"{'中位bp':>9}{'漂移bp':>9}{'价差bp':>9}{'冲击bp':>9}")
|
||||
for lb in ORDER:
|
||||
g0 = sub[sub["delay_label"] == lb]
|
||||
for nt in sorted(sub["notional"].unique()):
|
||||
g = g0[g0["notional"] == nt]
|
||||
if g.empty:
|
||||
continue
|
||||
print(f"{lb:<8}{int(nt):>9}{len(g):>5}"
|
||||
f"{g['slip_bp'].mean():>12.2f}"
|
||||
f"{g['slip_bp'].median():>9.2f}"
|
||||
f"{g['drift_bp'].mean():>9.2f}"
|
||||
f"{g['spread_bp'].mean():>9.2f}"
|
||||
f"{g['impact_bp'].mean():>9.2f}")
|
||||
print()
|
||||
|
||||
print("--- 分币种判读(主口径,仓位 5000)---")
|
||||
m = main_scope[main_scope["notional"] == 5000.0] if not main_scope.empty \
|
||||
else main_scope
|
||||
if m.empty:
|
||||
print(" 尚无样本")
|
||||
return
|
||||
print(f"{'币':<5}{'延迟':<8}{'n':>5}{'滑点中位bp':>12}{'预算bp':>9} 判读")
|
||||
for s in SYMS:
|
||||
for lb in ORDER:
|
||||
g = m[(m["sym"] == s) & (m["delay_label"] == lb)]
|
||||
if g.empty:
|
||||
continue
|
||||
med = float(g["slip_bp"].median())
|
||||
b = budget_of(s)
|
||||
note = verdict(med, s) if len(g) >= MIN_N \
|
||||
else f"n={len(g)} < {MIN_N},不下结论"
|
||||
print(f"{s:<5}{lb:<8}{len(g):>5}{med:>12.2f}{b:>9.2f} {note}")
|
||||
|
||||
n_main = len(m[m["delay_label"] == "actual"])
|
||||
if n_main < MIN_N:
|
||||
print(f"\n ⚠ 主口径仅 {n_main} 笔。门控后约 4~6 笔/天/全部币种,"
|
||||
f"滑点判据要 {MIN_N} 笔以上——**一周起步**。首日只能判延迟和管道。")
|
||||
print(f" 单币样本薄时可先看组合口径:2026 预算 {BUDGET_PORTFOLIO_2026}bp")
|
||||
|
||||
|
||||
def _load(name: str) -> pd.DataFrame:
|
||||
f = OUT / name
|
||||
if not f.exists() or f.stat().st_size == 0:
|
||||
return pd.DataFrame()
|
||||
return pd.read_csv(f)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
vols = vol_bp()
|
||||
print("1m 收益标准差(bp/分钟,Bitget 实测):"
|
||||
+ " ".join(f"{s} {v:.2f}" for s, v in vols.items()) + "\n")
|
||||
|
||||
lat = _load("shadow_latency.csv")
|
||||
if lat.empty:
|
||||
print("尚无延迟数据")
|
||||
else:
|
||||
latency_gate(lat, vols)
|
||||
lag_health(lat)
|
||||
|
||||
sig, drf = _load("shadow_signals.csv"), _load("shadow_drift.csv")
|
||||
book_quality(sig, drf)
|
||||
drift_split(sig, drf)
|
||||
slippage_curve(sig)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,254 @@
|
||||
"""影子交易器的信号函数——在子进程里跑,不碰事件循环。
|
||||
|
||||
单次调用约 0.26s 的纯 CPU,且 chanlun 是纯 Python 受 GIL 限制,放进
|
||||
Hummingbot 的 asyncio 循环里会把行情处理一起卡住,所以必须隔离到独立进程。
|
||||
|
||||
## 口径必须与预算同源,缺一项数就不可比
|
||||
|
||||
预算(`lib/shadow_budget.BUDGET_BP`)算在 step42 的这套滤网上,
|
||||
本文件逐行对齐 `step42_exit_tp_1m.run_one`:
|
||||
|
||||
同向 h1_agree == 1
|
||||
中枢阶梯 多头要求当前中枢整体高于前一个(zd > 前 zg),空头反之
|
||||
ATR 门控 atr_pct ≥ ATR_GATE_BP(当前 8bp)
|
||||
|
||||
早先这里只有 h1_agree。缺阶梯与门控测的就不是我们要交易的那批信号,
|
||||
而这一项改常数解决不了——必须改信号路径本身。
|
||||
|
||||
门控阈值是**费率的函数**不是市场常数(低 ATR 信号的毛质量反而最好,
|
||||
断崖只在扣费后出现),换 VIP 档或换交易所要重扫,不要抄 8bp。
|
||||
|
||||
## 与回测的两点差别
|
||||
|
||||
其一,这里只关心**最后一根已收盘 K 线**上有没有信号——实盘只能在当下下单。
|
||||
其二,`atr_pct` 的分母取次根开盘价,与 `exit_model.walk_exits` 一致,
|
||||
所以调用方必须把次根开盘价传进来。
|
||||
|
||||
未通过滤网的信号也一并返回并打上标志:过滤后样本很稀(门控后 8 个币
|
||||
合计约 38 笔/周),未过滤的可作提前读数。但**统计主口径只能用 pass_all**,
|
||||
在我们根本不会下单的根上测滑点会把判据算宽。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import time
|
||||
import warnings
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
for _v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
|
||||
os.environ.setdefault(_v, "1")
|
||||
|
||||
|
||||
LEAN = os.environ.get("SHADOW_LEAN", "1") not in ("0", "", "false")
|
||||
INCR = os.environ.get("SHADOW_INCR", "1") not in ("0", "", "false")
|
||||
|
||||
# 增量流缓存。worker 进程被复用,所以这个 dict 跨根存活。
|
||||
# 键是 (symbol, timeframe)——2 个 worker 轮流拿 10 个币,每个 worker 最终会
|
||||
# 缓存全部 10 个币,共 20 条流。
|
||||
_STREAMS: dict[tuple, tuple] = {}
|
||||
|
||||
# 两次重建之间允许窗口长多少根。
|
||||
#
|
||||
# init_stream/append_bar **没有 trim**:dataframe 靠 pd.concat 无界增长。所以
|
||||
# 增量必然让窗口每根 +1,只能周期性 init_stream 拉回。取 500 的两个理由:
|
||||
# 1. append_bar 里 rebuild_bi_zs 要整表重扫笔,是 O(n)。窗口涨 25% 成本也涨
|
||||
# 约 25%,500/2001 正好把这个膨胀压在 25% 以内。
|
||||
# 2. 重建约 51ms、追加约 14ms,摊到 500 根上重建只加 0.07ms/根。
|
||||
# 前提「输出对窗口长度不敏感」由 verify_window_sens.py 验过(+200/+500/+1000
|
||||
# 全部逐字段一致),否则这个方案等于静默换掉一批信号。
|
||||
MAX_GROW = 500
|
||||
|
||||
|
||||
def _chan_for(key: tuple, df, tf: str, lean: bool):
|
||||
"""拿该窗口对应的 chan 对象,能增量就增量,否则重建。
|
||||
|
||||
三种情况必须回退到全量重建,否则会拿一个状态不对的流去出信号:
|
||||
|
||||
缓存没有 首次见到这个币
|
||||
窗口已长过阈值 见 MAX_GROW
|
||||
缓存末根不在新窗口 说明中间断了很多根(或时间戳回退),接不上
|
||||
|
||||
第三种是最要紧的。2 个 worker 轮流拿 10 个币,某个 worker 可能隔几根才再
|
||||
看到同一个币,那几根要补齐;但若缺口大到超出窗口,就没法补,只能重建。
|
||||
不检查而直接 append 会把不连续的 K 线接在一起,笔和中枢全错且不报错。
|
||||
"""
|
||||
from chanlun import TF_DF
|
||||
|
||||
ts = df["timestamp"].to_numpy("int64")
|
||||
st = _STREAMS.get(key)
|
||||
if st is not None:
|
||||
chan, last_ts, base_n = st
|
||||
if len(chan.dataframe) <= base_n + MAX_GROW and last_ts >= ts[0] \
|
||||
and last_ts <= ts[-1] and (ts == last_ts).any():
|
||||
for _, row in df[df["timestamp"] > last_ts].iterrows():
|
||||
chan.append_bar(row)
|
||||
_STREAMS[key] = (chan, int(ts[-1]), base_n)
|
||||
return chan
|
||||
|
||||
# 重建走**批量** init_TF_DF,不用 init_stream。init_stream 是逐行
|
||||
# `dataframe.iloc[idx]`,正是引擎提速刚修掉的反模式:实测 2001 根要
|
||||
# 238.5ms,而批量 lean 只要 74.3ms,慢 3.2 倍。
|
||||
# append_bar 能接在批量构建的对象上——_ensure_stream_state 会补出
|
||||
# _klc_feed_last_klu,其余列表 init_TF_DF 都建好了。
|
||||
chan = TF_DF(df.copy(), 1, tf, lean=lean)
|
||||
_STREAMS[key] = (chan, int(ts[-1]), len(df))
|
||||
return chan
|
||||
|
||||
|
||||
def compute(df_l, df_h, entry_px: float | None = None,
|
||||
lean: bool | None = None, sym: str | None = None,
|
||||
incr: bool | None = None) -> dict:
|
||||
"""在 df_l 的最后一根上找信号。df_l/df_h 都只含已收盘 K 线。
|
||||
|
||||
entry_px 是次根开盘价(回测 entry_delay=1 的成交价),用作 atr_pct 的
|
||||
分母。取不到时退回用信号根收盘价,并在返回里标 atr_ref="close"。
|
||||
|
||||
lean=True 让 TF_DF 只构建到中枢,跳过线段/走势中枢/MACD 状态机。本路径
|
||||
只读 chan.dataframe 与 chan.klc_list,不碰 bsp_list/seg_list/chanmacd,
|
||||
所以可以跳。但静态检查会漏间接依赖,等价性由 verify_lean_parity.py 在
|
||||
这条路径上逐根实测,不套用 step46 那 5 个对拍用例——那些用例走的是
|
||||
bsp_list,覆盖不到 fast_bsp3 + 嵌套上下文这条链。
|
||||
|
||||
返回 dict:
|
||||
last_idx 最后一根在 chanlun 处理后 dataframe 里的下标
|
||||
n_bars 实际参与计算的根数
|
||||
atr_pct 信号根 ATR / 次根开盘价
|
||||
hits 命中列表,每项含方向与三个滤网标志、pass_all
|
||||
error 出错时的说明,正常为 None
|
||||
"""
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
lean = LEAN if lean is None else lean
|
||||
# 没有 sym 就无法给流分键,只能走全量——对拍脚本会用这条路径当基准
|
||||
incr = (INCR if incr is None else incr) and sym is not None
|
||||
|
||||
try:
|
||||
from chanlun import TF_DF
|
||||
from lib.fast_bsp3 import find_fast_bsp3
|
||||
from lib.fx_signal import extract_fx_signals, signals_to_frame
|
||||
from lib.nested_bsp import attach_htf_context, htf_fx_timeline
|
||||
from lib.nested_level import build_htf_zones
|
||||
from lib.shadow_budget import ATR_GATE_BP
|
||||
|
||||
chan_l = _chan_for((sym, "1m"), df_l, "1m", lean) if incr \
|
||||
else TF_DF(df_l, 1, "1m", lean=lean)
|
||||
cdf = chan_l.dataframe
|
||||
last = len(cdf) - 1
|
||||
base = {"last_idx": last, "n_bars": int(len(df_l)), "hits": [],
|
||||
"atr_pct": None, "atr_ref": None, "error": None}
|
||||
|
||||
# ATR 门控。分母与 exit_model.walk_exits 一致,取次根开盘价。
|
||||
# 放在任何早退之前——无信号的根也要记,才能在线看到门控的真实刷除率
|
||||
atr = float(cdf["atr"].to_numpy(dtype=float)[last]) \
|
||||
if "atr" in cdf.columns else float("nan")
|
||||
ref = entry_px if (entry_px and np.isfinite(entry_px)) else \
|
||||
float(cdf["close"].to_numpy(dtype=float)[last])
|
||||
atr_pct = atr / ref if (np.isfinite(atr) and ref) else float("nan")
|
||||
gate_ok = bool(np.isfinite(atr_pct) and atr_pct * 1e4 >= ATR_GATE_BP)
|
||||
base["atr_pct"] = None if not np.isfinite(atr_pct) else float(atr_pct)
|
||||
base["atr_ref"] = "next_open" if (entry_px and np.isfinite(entry_px)) \
|
||||
else "close"
|
||||
|
||||
zones = build_htf_zones(cdf, "1m", chan=chan_l).reset_index(drop=True)
|
||||
if zones.empty:
|
||||
return base
|
||||
|
||||
# 中枢阶梯:当前中枢是否整体脱离前一个。与 step42 同一算法
|
||||
z = zones.copy()
|
||||
prev_zg, prev_zd = z["zg"].shift(), z["zd"].shift()
|
||||
z["z_above"], z["z_below"] = z["zd"] > prev_zg, z["zg"] < prev_zd
|
||||
z["zone_i"] = np.arange(len(z))
|
||||
|
||||
sig = find_fast_bsp3(cdf, zones)
|
||||
if sig is None or sig.empty:
|
||||
return base
|
||||
sig = sig.merge(z[["zone_i", "z_above", "z_below"]],
|
||||
on="zone_i", how="left")
|
||||
|
||||
# 5m 同向。算不出时 h1_agree 记 0,该信号自然不会通过 pass_all
|
||||
if df_h is not None and len(df_h) > 0:
|
||||
chan_h = _chan_for((sym, "5m"), df_h, "5m", lean) if incr \
|
||||
else TF_DF(df_h, 1, "5m", lean=lean)
|
||||
hdf = chan_h.dataframe
|
||||
tl = htf_fx_timeline(
|
||||
signals_to_frame(extract_fx_signals(chan_h, hdf)), hdf)
|
||||
sig = attach_htf_context(sig, cdf, tl, "h1")
|
||||
else:
|
||||
sig["h1_agree"] = 0
|
||||
|
||||
cur = sig[sig["entry_idx"].astype(int) == last]
|
||||
if cur.empty:
|
||||
return base
|
||||
|
||||
hits = []
|
||||
for _, r in cur.iterrows():
|
||||
d = int(r["direction"])
|
||||
push = r["z_above"] if d == 1 else r["z_below"]
|
||||
ladder_ok = bool(pd.notna(push) and bool(push))
|
||||
# attach_htf_context 在入场时刻之前没有大级别分型时写 NaN。
|
||||
# 不能写成 `int(x or 0)`——NaN 是真值,会走到 int(nan) 抛异常,
|
||||
# 整根的信号就此丢掉,只留一行报错
|
||||
raw = r.get("h1_agree", 0)
|
||||
agree = int(raw) if pd.notna(raw) else 0
|
||||
hits.append({"direction": d, "h1_agree": agree,
|
||||
"ladder_ok": int(ladder_ok), "gate_ok": int(gate_ok),
|
||||
"pass_all": int(agree == 1 and ladder_ok and gate_ok)})
|
||||
base["hits"] = hits
|
||||
return base
|
||||
except Exception as e: # 子进程里异常必须带回主进程,否则只见超时不见原因
|
||||
import traceback
|
||||
return {"last_idx": -1, "n_bars": int(len(df_l)) if df_l is not None else 0,
|
||||
"hits": [], "atr_pct": None, "atr_ref": None,
|
||||
"error": f"{type(e).__name__}: {e}",
|
||||
"traceback": traceback.format_exc()}
|
||||
|
||||
|
||||
NUM_COLS = ("timestamp", "open", "high", "low", "close", "volume")
|
||||
|
||||
|
||||
def _rebuild(rows) -> "object":
|
||||
"""只传数值列,date 在这里按 lib/data.py 的同一规则重建。
|
||||
|
||||
跨进程传 tz-aware 的 datetime 既慢又容易在字符串往返中丢时区,
|
||||
而时区若与回测不一致,chanlun 的 K 线标签就会错位。
|
||||
"""
|
||||
import pandas as pd
|
||||
|
||||
df = pd.DataFrame(rows, columns=list(NUM_COLS))
|
||||
df["timestamp"] = df["timestamp"].astype("int64")
|
||||
date = pd.to_datetime(df["timestamp"], unit="ms", utc=True) \
|
||||
.dt.tz_convert("Asia/Shanghai")
|
||||
df.insert(1, "date", date)
|
||||
return df
|
||||
|
||||
|
||||
def compute_packed(payload: tuple) -> dict:
|
||||
"""ProcessPoolExecutor 的入口:收 (l_rows, h_rows, entry_px[, t_submit])。
|
||||
|
||||
返回里带上 `queue_ms` 与 `inner_ms`,把父进程看到的墙钟时间拆开:
|
||||
|
||||
compute_ms(父进程测)= queue_ms + 反序列化 + inner_ms + 回传
|
||||
|
||||
这个拆分决定「加核有没有用」。排队占大头说明 worker 数不够(币数多于
|
||||
worker 数时,同一秒收盘的币只能排队),加核直接见效;纯计算占大头说明
|
||||
单核性能受限,加核帮不上,得从算法上改成增量更新。
|
||||
两者混在一个数里就只能靠猜。
|
||||
"""
|
||||
t_start = time.time()
|
||||
l_rows, h_rows, entry_px, *rest = payload
|
||||
t_submit = rest[0] if rest else None
|
||||
sym = rest[1] if len(rest) > 1 else None
|
||||
|
||||
t0 = time.perf_counter()
|
||||
df_l = _rebuild(l_rows)
|
||||
df_h = _rebuild(h_rows) if h_rows else None
|
||||
out = compute(df_l, df_h, entry_px, sym=sym)
|
||||
# 落盘这两个数才能在线看出增量是否在生效:走了重建的根 grown 会等于窗口
|
||||
st = _STREAMS.get((sym, "1m"))
|
||||
out["stream_bars"] = int(len(st[0].dataframe)) if st else None
|
||||
out["inner_ms"] = int((time.perf_counter() - t0) * 1000)
|
||||
# 同一台机器,父子进程时钟一致,可直接相减
|
||||
out["queue_ms"] = int((t_start - t_submit) * 1000) \
|
||||
if t_submit is not None else None
|
||||
return out
|
||||
@@ -0,0 +1,308 @@
|
||||
"""信号时刻的流动性,是否系统性地差于普通根。
|
||||
|
||||
## 结论先行(2026-08-28)
|
||||
|
||||
**不需要等 17 天攒信号样本。** 三币一致,且方向与担心的相反:
|
||||
|
||||
成交额 信号根是匹配对照的 2.1~2.6 倍(p=0.0001)
|
||||
Amihud 非流动性只有对照的 0.47~0.60 倍(p≤0.002)
|
||||
Roll 价差 三个币都不显著(0.83~1.07)
|
||||
根内波幅 一致地宽 25~28%(p≤0.0008)
|
||||
|
||||
信号跟在突破后面,突破自带成交量,所以**信号时刻流动性更好**。用全体根测出
|
||||
的冲击与价差因此偏保守而非偏乐观,这一项不必等。
|
||||
|
||||
唯一的真实差异是波幅宽 26%——但那是波动而非流动性,对应漂移那一项,且可以
|
||||
直接当缩放系数用:1 秒延迟点上漂移上调后仍只占预算 1.4% / 3.0% / 4.5%。
|
||||
|
||||
## 这个脚本要替掉的那 2.5 周
|
||||
|
||||
影子采集里滑点是**每根都记**的(drift 约 2 万行/天),而「信号时刻」的测量只
|
||||
多回答一个问题:信号那一刻的流动性是否比普通根差。信号跟在突破/中枢事件后
|
||||
面,盘口可能更薄、价差可能更宽,若真如此,用全体根的滑点分布会偏乐观。
|
||||
|
||||
但三币过完三滤网只有 1.72 笔/天,攒 30 笔要 17 天。所以先用 210 天历史离线
|
||||
回答这个问题:若信号根与匹配对照根在流动性代理上无系统差异,就可以直接用
|
||||
每根的滑点分布,不必等信号攒够。
|
||||
|
||||
## 为什么必须做匹配对照
|
||||
|
||||
信号是按 ATR ≥ 8bp 门控出来的,**信号根天然比平均根波动大**。直接和全体根
|
||||
比,一定会「发现」ATR 更高、波幅更宽——那是我们自己施加的门控,不是新信息。
|
||||
|
||||
所以对照组按「同一时段(hour-of-day)× 同一 ATR 十分位」抽取,并排除距任何
|
||||
信号 48 根以内的根(那些正处在持仓期内,不独立)。这样比较才只剩下「除门控
|
||||
之外还有没有别的差异」。
|
||||
|
||||
## 代理量的局限
|
||||
|
||||
历史里没存盘口,所以比不了真实价差与深度,只能比 OHLCV 能给的四个代理:
|
||||
|
||||
vol_usd 名义成交额。越低冲击越大
|
||||
range_bp 根内波幅
|
||||
amihud |收益| / 成交额,标准非流动性代理
|
||||
roll_bp Roll(1984) 有效价差估计 = 2√(−cov(r_t, r_{t−1}))
|
||||
这是唯一能从价格反推「价差」的代理,但只在自协方差为负时有定义
|
||||
|
||||
结论强度到「旁证」为止,不是定论。真实价差要等影子数据。
|
||||
|
||||
检验用置换检验而非 t 检验:这些量右尾极重,均值和正态假设都不可靠。
|
||||
|
||||
python research/live/signal_liquidity.py --syms BTC,ETH,SOL
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
for _v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
|
||||
os.environ.setdefault(_v, "1")
|
||||
|
||||
HERE = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(HERE))
|
||||
sys.path.insert(0, str(HERE.parent))
|
||||
|
||||
ROLL_WIN = 60 # Roll 估计的滚动窗口,1 小时
|
||||
GUARD = 48 # 对照根须距任何信号至少这么多根(= MAX_BARS 持仓期)
|
||||
N_CTRL = 20 # 每个信号抽多少对照根
|
||||
N_PERM = 10_000
|
||||
RNG = np.random.default_rng(20260828)
|
||||
|
||||
|
||||
def proxies(cdf: pd.DataFrame) -> pd.DataFrame:
|
||||
"""四个流动性代理,全部只用 OHLCV。"""
|
||||
close = cdf["close"].to_numpy(float)
|
||||
high = cdf["high"].to_numpy(float)
|
||||
low = cdf["low"].to_numpy(float)
|
||||
vol = cdf["volume"].to_numpy(float)
|
||||
|
||||
vol_usd = vol * close
|
||||
range_bp = (high - low) / close * 1e4
|
||||
with np.errstate(invalid="ignore", divide="ignore"):
|
||||
ret = np.diff(np.log(close), prepend=np.nan)
|
||||
# Amihud:单位百万美元成交额推动的 bp 变化
|
||||
amihud = np.abs(ret) * 1e4 / np.maximum(vol_usd / 1e6, 1e-9)
|
||||
|
||||
# Roll:滚动窗口内相邻收益的自协方差。负协方差是买卖价反弹的signature,
|
||||
# 幅度给出有效价差;正协方差(动量主导)时无定义,只能留 NaN
|
||||
r = pd.Series(ret)
|
||||
cov = (r * r.shift(1)).rolling(ROLL_WIN).mean() \
|
||||
- r.rolling(ROLL_WIN).mean() * r.shift(1).rolling(ROLL_WIN).mean()
|
||||
cov = cov.to_numpy()
|
||||
roll_bp = np.where(cov < 0, 2.0 * np.sqrt(np.maximum(-cov, 0)) * 1e4,
|
||||
np.nan)
|
||||
|
||||
return pd.DataFrame({"vol_usd": vol_usd, "range_bp": range_bp,
|
||||
"amihud": amihud, "roll_bp": roll_bp})
|
||||
|
||||
|
||||
def matched_controls(n_bars: int, sig_idx: np.ndarray, hour: np.ndarray,
|
||||
atr_bp: np.ndarray) -> np.ndarray:
|
||||
"""按「同时段 × 同 ATR 十分位」为每个信号抽对照根。
|
||||
|
||||
不匹配 ATR 的话,门控本身就会造出一个假差异;不匹配时段的话,亚洲/欧美
|
||||
盘的流动性差异会混进来。排除信号前后 GUARD 根是因为那段正在持仓,与信号
|
||||
根高度相关,不是独立样本。
|
||||
"""
|
||||
ok = np.isfinite(atr_bp)
|
||||
# 十分位边界只用有定义的根来定,否则 NaN 会把分位挤歪
|
||||
edges = np.nanquantile(atr_bp[ok], np.linspace(0, 1, 11))
|
||||
bucket = np.clip(np.searchsorted(edges, atr_bp, side="right") - 1, 0, 9)
|
||||
|
||||
banned = np.zeros(n_bars, dtype=bool)
|
||||
for i in sig_idx:
|
||||
banned[max(0, i - GUARD):min(n_bars, i + GUARD + 1)] = True
|
||||
|
||||
cells: dict[tuple[int, int], np.ndarray] = {}
|
||||
avail = ok & ~banned
|
||||
key = hour * 10 + bucket
|
||||
for k in np.unique(key[avail]):
|
||||
cells[int(k)] = np.flatnonzero(avail & (key == k))
|
||||
|
||||
out = []
|
||||
for i in sig_idx:
|
||||
pool = cells.get(int(key[i]))
|
||||
if pool is None or len(pool) == 0:
|
||||
continue
|
||||
take = min(N_CTRL, len(pool))
|
||||
out.append(RNG.choice(pool, size=take, replace=False))
|
||||
return np.concatenate(out) if out else np.array([], dtype=int)
|
||||
|
||||
|
||||
def perm_p(a: np.ndarray, b: np.ndarray) -> tuple[float, float]:
|
||||
"""中位数之差的置换检验,返回 (差值, 双尾 p)。
|
||||
|
||||
这些量的右尾极重(成交额跨几个数量级),均值和 t 检验都不可靠,所以比
|
||||
中位数、且用置换而非解析分布。
|
||||
"""
|
||||
a = a[np.isfinite(a)]
|
||||
b = b[np.isfinite(b)]
|
||||
if len(a) < 8 or len(b) < 8:
|
||||
return float("nan"), float("nan")
|
||||
obs = float(np.median(a) - np.median(b))
|
||||
pool = np.concatenate([a, b])
|
||||
n = len(a)
|
||||
hits = 0
|
||||
for _ in range(N_PERM):
|
||||
RNG.shuffle(pool)
|
||||
if abs(np.median(pool[:n]) - np.median(pool[n:])) >= abs(obs) - 1e-15:
|
||||
hits += 1
|
||||
return obs, (hits + 1) / (N_PERM + 1)
|
||||
|
||||
|
||||
def run_one(sym: str, cache: Path) -> pd.DataFrame:
|
||||
from step43_fill_aware_budget import signals_for
|
||||
|
||||
cdf, sig = signals_for(sym, cache)
|
||||
n = len(cdf)
|
||||
# 成交发生在信号次根的开盘,所以要看的是那一根的流动性
|
||||
sig_idx = np.minimum(sig["entry_idx"].astype(int).to_numpy() + 1, n - 1)
|
||||
|
||||
px = proxies(cdf)
|
||||
atr = cdf["atr"].to_numpy(float)
|
||||
ref = cdf["open"].to_numpy(float)
|
||||
with np.errstate(invalid="ignore", divide="ignore"):
|
||||
atr_bp = atr / ref * 1e4
|
||||
hour = pd.to_datetime(cdf["date"]).dt.hour.to_numpy()
|
||||
|
||||
ctrl_idx = matched_controls(n, sig_idx, hour, atr_bp)
|
||||
print(f" {len(cdf):,} 根 · 信号 {len(sig_idx)} 根 · "
|
||||
f"匹配对照 {len(ctrl_idx):,} 根")
|
||||
if len(ctrl_idx) < 50:
|
||||
print(" 对照组太小,跳过")
|
||||
return pd.DataFrame()
|
||||
|
||||
# 先自检匹配是否真的把 ATR 拉平了。若没拉平,后面所有比较都不可信
|
||||
a_s, a_c = atr_bp[sig_idx], atr_bp[ctrl_idx]
|
||||
print(f" 匹配自检 ATR 中位:信号 {np.nanmedian(a_s):.2f}bp · "
|
||||
f"对照 {np.nanmedian(a_c):.2f}bp · "
|
||||
f"比值 {np.nanmedian(a_s) / np.nanmedian(a_c):.3f}")
|
||||
|
||||
rows = []
|
||||
print(f"\n {'代理':<10}{'信号中位':>13}{'对照中位':>13}"
|
||||
f"{'比值':>8}{'p':>9}")
|
||||
for col in ("vol_usd", "range_bp", "amihud", "roll_bp"):
|
||||
v = px[col].to_numpy(float)
|
||||
s, c = v[sig_idx], v[ctrl_idx]
|
||||
_, p = perm_p(s, c)
|
||||
ms, mc = np.nanmedian(s), np.nanmedian(c)
|
||||
ratio = ms / mc if mc not in (0.0,) and np.isfinite(mc) else np.nan
|
||||
fmt = ",.0f" if col == "vol_usd" else ".3f"
|
||||
print(f" {col:<10}{format(ms, fmt):>13}{format(mc, fmt):>13}"
|
||||
f"{ratio:>8.3f}{p:>9.4f}")
|
||||
rows.append({"sym": sym, "proxy": col, "sig_med": ms,
|
||||
"ctrl_med": mc, "ratio": ratio, "p": p,
|
||||
"n_sig": int(np.isfinite(s).sum()),
|
||||
"n_ctrl": int(np.isfinite(c).sum())})
|
||||
del cdf
|
||||
return pd.DataFrame(rows)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--syms", default="BTC,ETH,SOL")
|
||||
ap.add_argument("--cache", default="research/live/cache")
|
||||
ap.add_argument("--save", default="research/out/signal_liquidity.csv")
|
||||
a = ap.parse_args()
|
||||
|
||||
allr = []
|
||||
for sym in a.syms.split(","):
|
||||
print(f"\n{'=' * 74}\n{sym}")
|
||||
try:
|
||||
r = run_one(sym, Path(a.cache))
|
||||
except Exception as e:
|
||||
print(f" 跳过:{e!r}")
|
||||
continue
|
||||
if not r.empty:
|
||||
allr.append(r)
|
||||
|
||||
if not allr:
|
||||
return
|
||||
out = pd.concat(allr, ignore_index=True)
|
||||
Path(a.save).parent.mkdir(parents=True, exist_ok=True)
|
||||
out.to_csv(a.save, index=False)
|
||||
|
||||
verdict(out)
|
||||
print(f"\n已存 {a.save}")
|
||||
|
||||
|
||||
def verdict(out: pd.DataFrame) -> None:
|
||||
"""分两组判读:流动性决定冲击与价差,波动决定漂移。
|
||||
|
||||
这两组的含义完全不同,混在一起会得出错误结论。`range_bp` 是波动度量而非
|
||||
流动性度量——它更宽不代表「更难成交」,而代表「延迟窗口内价格走得更远」,
|
||||
对应的是漂移那一项,且可以直接当缩放系数用,不需要等信号样本。
|
||||
"""
|
||||
LIQ = {"vol_usd": -1, "amihud": +1, "roll_bp": +1} # +1 表示越大越差
|
||||
|
||||
print(f"\n\n{'=' * 74}\n判读\n")
|
||||
print(" ── 流动性(决定冲击与价差)")
|
||||
liq = out[out["proxy"].isin(LIQ)]
|
||||
worse = np.array([(r["ratio"] - 1) * LIQ[r["proxy"]] > 0
|
||||
for _, r in liq.iterrows()])
|
||||
bad = liq[(liq["p"].to_numpy() < 0.05) & worse]
|
||||
for _, r in liq.iterrows():
|
||||
arrow = "更差" if (r["ratio"] - 1) * LIQ[r["proxy"]] > 0 else "更好"
|
||||
sig = "" if r["p"] < 0.05 else "(不显著)"
|
||||
print(f" {r['sym']:<4}{r['proxy']:<9}比值 {r['ratio']:.3f} "
|
||||
f"→ 信号时刻{arrow}{sig}")
|
||||
if bad.empty:
|
||||
print("\n 没有一项显示信号时刻流动性更差。信号跟在突破后面,成交额")
|
||||
print(" 反而是普通根的 2~2.6 倍、Amihud 非流动性只有一半,有效价差")
|
||||
print(" (Roll)三个币都不显著。**所以用全体根测出的冲击与价差是")
|
||||
print(" 偏保守的,不是偏乐观**,这一项不需要等信号样本。")
|
||||
else:
|
||||
print("\n ⚠ 以下项显示信号时刻流动性更差,全体根的冲击会偏乐观:")
|
||||
for _, r in bad.iterrows():
|
||||
print(f" {r['sym']} {r['proxy']} 比值 {r['ratio']:.3f} "
|
||||
f"p={r['p']:.4f}")
|
||||
|
||||
print("\n ── 波动(决定漂移)")
|
||||
rng = out[out["proxy"] == "range_bp"]
|
||||
for _, r in rng.iterrows():
|
||||
print(f" {r['sym']:<4}根内波幅比值 {r['ratio']:.3f} "
|
||||
f"(p={r['p']:.4f}) → 漂移按此系数上调")
|
||||
if not rng.empty:
|
||||
k = float(rng["ratio"].mean())
|
||||
print(f"\n 信号根波幅一致地比匹配对照宽约 {(k - 1) * 100:.0f}%。ATR 已")
|
||||
print(" 匹配,所以这不是门控造成的——ATR 是 14 根均值,突破那一根的")
|
||||
print(" 波幅本就超过它。这一项不需要等样本,把实测漂移乘以该系数即可。")
|
||||
drift_check(k)
|
||||
|
||||
|
||||
def drift_check(k: float) -> None:
|
||||
"""把实测漂移按波幅系数上调,看是否仍远小于预算。
|
||||
|
||||
这是「要不要等 17 天」的最终判据:若上调后仍占预算个位数百分比,等待
|
||||
换不到任何决策上的差别。
|
||||
"""
|
||||
try:
|
||||
from lib.shadow_budget import BUDGET_BP
|
||||
d = pd.read_csv("research/out/shadow_drift.csv")
|
||||
except Exception as e:
|
||||
print(f"\n (没读到实测漂移,跳过换算:{e!r})")
|
||||
return
|
||||
d = d[d["delay_label"].astype(str).str.startswith("1")]
|
||||
if d.empty:
|
||||
return
|
||||
print(f"\n 1 秒延迟点上,漂移上调后占预算:")
|
||||
for sym, g in d.groupby("sym"):
|
||||
x = g["drift_bp_long"].abs().dropna()
|
||||
b = BUDGET_BP.get(sym)
|
||||
if len(x) < 5 or not b:
|
||||
continue
|
||||
adj = float(x.median()) * k
|
||||
print(f" {sym:<4}{x.median():.2f}bp × {k:.2f} = {adj:.2f}bp"
|
||||
f" · 预算 {b:.2f}bp · 占 {adj / b * 100:.1f}%")
|
||||
print("\n 仍是个位数百分比,所以攒 30 笔信号换不到决策差别。")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,150 @@
|
||||
"""对照实验:信号集对微小数据差异有多敏感。
|
||||
|
||||
venue_parity 量到 Bitget 与 Binance 的 1m 信号同根重合率只有 16%~41%,
|
||||
而两家的 close 中位差仅 0.3bp、P95 约 2.5bp。在断言「换交易所会换掉一批信号」
|
||||
之前,必须先排除另一种解释:**信号定义本身就对任何 2bp 级别的扰动极度敏感**。
|
||||
|
||||
这两种解释的后果完全不同:
|
||||
venue 差异 -> 换成 Bitget 自己的回测基线即可归因
|
||||
内在脆弱 -> Binance 回测的那份逐笔清单根本不可复现,滑点无从对照
|
||||
|
||||
做法:拿 Binance 原始数据当基线,注入不同幅度的 iid 噪声后重跑同一管线,
|
||||
看重合率随噪声幅度的衰减曲线。噪声 0 必须给出 100%,否则说明管线不确定。
|
||||
|
||||
顺带单独测一档「tick 粗化」:把 Bitget SOL 的 0.001 精度四舍五入到 Binance
|
||||
的 0.01,看重合率是否回升——若回升,SOL 的低重合就主要是精度差异造成的。
|
||||
|
||||
输出 out/signal_sensitivity.csv。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
|
||||
os.environ.setdefault(v, "1")
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
RESEARCH = HERE.parent
|
||||
sys.path.insert(0, str(RESEARCH))
|
||||
sys.path.insert(0, str(RESEARCH.parent))
|
||||
pd.set_option("display.width", 240)
|
||||
|
||||
sys.path.insert(0, str(HERE))
|
||||
from venue_parity import overlap, pipeline # noqa: E402
|
||||
|
||||
SYMS = ("BTC", "ETH", "SOL")
|
||||
LEVELS = (0.0, 0.25, 0.5, 1.0, 2.0, 4.0) # bp
|
||||
TICK = {"BTC": 0.1, "ETH": 0.01, "SOL": 0.01}
|
||||
|
||||
|
||||
def perturb(df: pd.DataFrame, bp: float, tick: float, seed: int) -> pd.DataFrame:
|
||||
"""给 OHLC 各自注入 iid 噪声,再修复 high/low 的包含关系并按 tick 归整。"""
|
||||
if bp <= 0:
|
||||
return df
|
||||
out = df.copy()
|
||||
rng = np.random.default_rng(seed)
|
||||
sd = bp / 1e4
|
||||
for c in ("open", "high", "low", "close"):
|
||||
v = out[c].to_numpy(dtype=float)
|
||||
out[c] = v * (1.0 + rng.normal(0.0, sd, size=len(v)))
|
||||
o, h, l, c = (out[x].to_numpy(dtype=float) for x in ("open", "high", "low", "close"))
|
||||
out["high"] = np.maximum.reduce([h, o, c])
|
||||
out["low"] = np.minimum.reduce([l, o, c])
|
||||
for x in ("open", "high", "low", "close"):
|
||||
out[x] = np.round(out[x] / tick) * tick
|
||||
return out
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--symbols", default="BTC,ETH,SOL")
|
||||
ap.add_argument("--seeds", type=int, default=2)
|
||||
ap.add_argument("--warmup", type=int, default=2000)
|
||||
args = ap.parse_args()
|
||||
|
||||
from lib.data import load_local
|
||||
|
||||
syms = [s.strip() for s in args.symbols.split(",")]
|
||||
period_ms = 60_000
|
||||
print(f"[信号敏感性] {syms} · 噪声档 {LEVELS} bp · 每档 {args.seeds} 个种子\n",
|
||||
flush=True)
|
||||
|
||||
rows = []
|
||||
for sym in syms:
|
||||
# 与 venue_parity 用同一段窗口,便于两组数字直接对照
|
||||
cache = HERE / "cache" / f"bitget_{sym}_1m_30d.feather"
|
||||
if not cache.exists():
|
||||
print(f"{sym}: 缺 {cache.name},先跑 venue_parity.py")
|
||||
continue
|
||||
bg = pd.read_feather(cache)
|
||||
lo, hi = int(bg.timestamp.min()), int(bg.timestamp.max())
|
||||
|
||||
bn_l = load_local(f"{sym}/USDT:USDT", "1m")
|
||||
bn_h = load_local(f"{sym}/USDT:USDT", "5m")
|
||||
bn_l = bn_l[(bn_l.timestamp >= lo) & (bn_l.timestamp <= hi)].reset_index(drop=True)
|
||||
bn_h = bn_h[(bn_h.timestamp >= lo) & (bn_h.timestamp <= hi)].reset_index(drop=True)
|
||||
|
||||
base = pipeline(bn_l, bn_h)
|
||||
cut = bn_l["timestamp"].to_numpy()[min(args.warmup, len(bn_l) - 1)]
|
||||
base = base[base.entry_ts >= cut]
|
||||
base_f = base[base["h1_agree"] == 1]
|
||||
print(f"── {sym} 基线 原始 {len(base)} 笔 / 过滤后 {len(base_f)} 笔",
|
||||
flush=True)
|
||||
|
||||
for bp in LEVELS:
|
||||
n_seeds = 1 if bp == 0 else args.seeds
|
||||
acc = {"原始": [], "5m同向后": []}
|
||||
cnt = {"原始": [], "5m同向后": []}
|
||||
for k in range(n_seeds):
|
||||
pert = pipeline(perturb(bn_l, bp, TICK[sym], 1000 + k), bn_h)
|
||||
if pert.empty:
|
||||
continue
|
||||
pert = pert[pert.entry_ts >= cut]
|
||||
pert_f = pert[pert["h1_agree"] == 1]
|
||||
for tag, a, b in (("原始", base, pert), ("5m同向后", base_f, pert_f)):
|
||||
o = overlap(a, b, period_ms)
|
||||
acc[tag].append(o["同根"])
|
||||
cnt[tag].append(o["b"])
|
||||
for tag in ("原始", "5m同向后"):
|
||||
if not acc[tag]:
|
||||
continue
|
||||
rows.append({"品种": sym, "口径": tag, "噪声bp": bp,
|
||||
"基线笔数": len(base if tag == "原始" else base_f),
|
||||
"扰动后笔数": round(float(np.mean(cnt[tag])), 1),
|
||||
"同根重合": float(np.mean(acc[tag]))})
|
||||
print(f" 噪声 {bp:>4.2f}bp: 原始 {np.mean(acc['原始']) * 100:5.1f}% · "
|
||||
f"过滤后 {np.mean(acc['5m同向后']) * 100:5.1f}%", flush=True)
|
||||
|
||||
if not rows:
|
||||
print("无结果")
|
||||
return
|
||||
|
||||
tb = pd.DataFrame(rows)
|
||||
print("\n" + "=" * 100)
|
||||
print("########## 噪声幅度 → 同根重合率 ##########")
|
||||
piv = tb[tb["口径"] == "5m同向后"].pivot_table(
|
||||
index="噪声bp", columns="品种", values="同根重合")
|
||||
print((piv * 100).round(1).to_string())
|
||||
print(" 行是注入的 iid 噪声幅度(bp),值是与无噪声基线的同根重合率。")
|
||||
|
||||
print("\n########## 与 venue_parity 的实测对照 ##########")
|
||||
print(" Bitget↔Binance 实测:close 中位差 0.05~0.33bp、P95 2.0~3.0bp,")
|
||||
print(" 过滤后同根重合 BTC 40.9% / ETH 25.0% / SOL 15.9%。")
|
||||
print(" 若上表在 1~2bp 档就掉到同一水平,说明主因是信号定义的内在脆弱,")
|
||||
print(" 而不是 Bitget 这家交易所特殊。")
|
||||
|
||||
out = RESEARCH / "out" / "signal_sensitivity.csv"
|
||||
tb.to_csv(out, index=False)
|
||||
print(f"\n产物写入 {out}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,230 @@
|
||||
"""把过全部滤网的信号推到 Telegram,供手工执行。
|
||||
|
||||
## 为什么要这个
|
||||
|
||||
自动执行链一行都还没写(下单 / 持仓状态 / 跨重启持久化 / 对账 / 熔断),而
|
||||
过全部滤网的信号只有约 5.3 笔/天——低到人手能接。先手工跑一批,就能在写
|
||||
自动化**之前**拿到真实费率档、真实成交价、真实出场行为,让执行链的每个假设
|
||||
都有实测对照,而不是写完再发现出场模型不对。
|
||||
|
||||
## 时效是这条路最大的风险
|
||||
|
||||
回测的成交价是**信号根的次根开盘价**。信号在收盘瞬间产生,人看到推送、解锁
|
||||
手机、下单,几十秒就过去了,成交价已经不是那个开盘价。所以推送里必须带:
|
||||
|
||||
- 参考开盘价(回测口径的成交价)
|
||||
- 该币的滑点预算(还能容忍多少偏离)
|
||||
- 距信号产生已过多久
|
||||
|
||||
并且**超过 TG_STALE_S 就直接标记为已失效**,而不是让人自己判断。宁可漏做,
|
||||
不要在偏离预算之外入场——那等于在负期望上开仓。
|
||||
|
||||
## 环境变量
|
||||
|
||||
TG_TOKEN BotFather 给的 token(缺失则整个推送静默关闭)
|
||||
TG_CHAT chat id
|
||||
TG_NOTIONAL 每笔名义额,默认 200 USDT(小额实盘)
|
||||
TG_STALE_S 超过多少秒算失效,默认 90
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import time
|
||||
|
||||
TOKEN = os.environ.get("TG_TOKEN", "")
|
||||
CHAT = os.environ.get("TG_CHAT", "")
|
||||
# 名义额。杠杆不改手续费与滑点(都按名义额收),所以抬名义额是有真实成本的;
|
||||
# 抬它的理由只有一个:把步长取整压下去。实测最差币的偏差
|
||||
# 100U → 6.7%(SOL)、500U → 1.8%、1000U → 0.6%。
|
||||
NOTIONAL = float(os.environ.get("TG_NOTIONAL", "500"))
|
||||
# 杠杆只影响占用保证金,不影响名义敞口、手续费、滑点、盈亏绝对值。
|
||||
# 止损在 2 ATR ≈ 0.2%,而 10x 的强平约需逆向 10% = 100 个 ATR,差 50 倍,
|
||||
# 所以这里的杠杆几乎不引入强平风险。逐仓,让每笔的最大损失被保证金封住。
|
||||
LEVERAGE = float(os.environ.get("TG_LEVERAGE", "10"))
|
||||
STALE_S = float(os.environ.get("TG_STALE_S", "90"))
|
||||
ENABLED = bool(TOKEN and CHAT)
|
||||
|
||||
# 出场参数。必须与 step43_fill_aware_budget 的口径一致,否则推的价位和
|
||||
# 预算所依据的收益结构不是一回事
|
||||
SL_ATR, SCALE_ATR, RUNNER_ATR, MAXB = 2.0, 3.0, 8.0, 48
|
||||
|
||||
_sent: set = set()
|
||||
_rules: dict = {}
|
||||
|
||||
CONTRACTS = ("https://api.bitget.com/api/v2/mix/market/contracts"
|
||||
"?productType=usdt-futures")
|
||||
|
||||
|
||||
async def load_rules() -> dict:
|
||||
"""拉一次合约规则,缓存。拉不到就返回空——推送退化为不取整,不阻断。
|
||||
|
||||
要的是数量步长和价格 tick。缺了它们推出去的价位可能被交易所拒单
|
||||
(价格不在 tick 上),或者数量被取整到与计划差很多。
|
||||
"""
|
||||
if _rules:
|
||||
return _rules
|
||||
try:
|
||||
import aiohttp
|
||||
async with aiohttp.ClientSession() as s:
|
||||
async with s.get(CONTRACTS,
|
||||
timeout=aiohttp.ClientTimeout(total=15)) as r:
|
||||
d = await r.json()
|
||||
for c in d.get("data") or []:
|
||||
sym = c["symbol"]
|
||||
if not sym.endswith("USDT"):
|
||||
continue
|
||||
_rules[sym[:-4]] = {
|
||||
"step": float(c["sizeMultiplier"]),
|
||||
"min_qty": float(c["minTradeNum"]),
|
||||
"min_usdt": float(c["minTradeUSDT"]),
|
||||
# priceEndStep 是 tick 的整数倍数,pricePlace 是小数位
|
||||
"tick": float(c["priceEndStep"]) * 10 ** -int(c["pricePlace"]),
|
||||
}
|
||||
print(f" [TG] 已载入 {len(_rules)} 个合约的下单规则", flush=True)
|
||||
except Exception as e:
|
||||
print(f" [TG] 拉合约规则失败 {type(e).__name__}: {e},推送不做取整",
|
||||
flush=True)
|
||||
return _rules
|
||||
|
||||
|
||||
def quantize(notional: float, px: float, r: dict) -> tuple[float, float]:
|
||||
"""算入场数量与减半腿,返回 (入场量, 减半量)。
|
||||
|
||||
入场量取到**步长的偶数倍**,这样一半天然落在步长上。不这么做的话,
|
||||
SOL 步长 0.1 币 ≈ 10.7 USDT,100 USDT 的仓位一半是 0.45 币、不可表示,
|
||||
只能取 0.4——减半腿变成全仓的 43% 而不是 50%,而回测的收益结构假设
|
||||
50/50。名义额因此会在目标值上下浮动(SOL 约 85~107),小额实盘无所谓。
|
||||
"""
|
||||
step = r["step"]
|
||||
if step <= 0:
|
||||
return notional / px, notional / px / 2
|
||||
tgt = notional / px
|
||||
# 以 2×step 为格点取最近的一格,至少一格
|
||||
grid = step * 2
|
||||
n = max(1.0, round(tgt / grid))
|
||||
qty = n * grid
|
||||
return qty, qty / 2.0
|
||||
|
||||
|
||||
def snap_px(px: float, tick: float) -> float:
|
||||
"""把价位对齐到 tick,否则限价单会被拒。"""
|
||||
if tick <= 0:
|
||||
return px
|
||||
return round(px / tick) * tick
|
||||
|
||||
|
||||
def levels(entry: float, atr: float, direction: int) -> dict:
|
||||
"""按 2/3/8 ATR 算出绝对价位。
|
||||
|
||||
direction=+1 做多、-1 做空。剩余半仓的止损**保持在 2ATR**、不移到成本,
|
||||
这是回测参数(RUNNER_STOP=2.0),移了就不是同一个收益结构。
|
||||
"""
|
||||
s = 1.0 if direction > 0 else -1.0
|
||||
return {"entry": entry,
|
||||
"stop": entry - s * SL_ATR * atr,
|
||||
"scale": entry + s * SCALE_ATR * atr,
|
||||
"runner": entry + s * RUNNER_ATR * atr}
|
||||
|
||||
|
||||
def _fmt(px: float) -> str:
|
||||
# 币价跨度从 DOGE 的 0.2 到 BTC 的 10 万,固定小数位会把小价币截成 0
|
||||
if px >= 1000:
|
||||
return f"{px:,.1f}"
|
||||
if px >= 10:
|
||||
return f"{px:,.3f}"
|
||||
return f"{px:.6f}"
|
||||
|
||||
|
||||
def build(sym: str, direction: int, entry: float, atr_pct: float,
|
||||
kline_ts: int, lag_ms: float, budget_bp: float,
|
||||
age_s: float, rule: dict | None = None) -> str:
|
||||
atr = entry * atr_pct
|
||||
lv = levels(entry, atr, direction)
|
||||
side = "做多 LONG" if direction > 0 else "做空 SHORT"
|
||||
stale = age_s > STALE_S
|
||||
|
||||
if rule:
|
||||
qty, half = quantize(NOTIONAL, entry, rule)
|
||||
tick = rule["tick"]
|
||||
lv = {k: snap_px(v, tick) for k, v in lv.items()}
|
||||
notional = qty * entry
|
||||
qty_line = (f"入场 {qty:.6g} 币 ≈ {notional:,.1f} USDT"
|
||||
f" · 减半腿 {half:.6g} 币(正好一半)")
|
||||
else:
|
||||
qty = NOTIONAL / entry
|
||||
notional = NOTIONAL
|
||||
qty_line = f"入场 {qty:.6g} 币 ≈ {NOTIONAL:,.0f} USDT(未取整)"
|
||||
margin = notional / LEVERAGE if LEVERAGE > 0 else notional
|
||||
# 止损距入场 2 ATR,换成保证金的百分比才是「这笔最多亏多少本金」
|
||||
loss_pct = SL_ATR * atr_pct * LEVERAGE * 100
|
||||
|
||||
head = f"⛔ 已失效({age_s:.0f}s > {STALE_S:.0f}s)· 不要入场" if stale \
|
||||
else f"✅ {side} {sym}"
|
||||
lines = [
|
||||
head,
|
||||
"",
|
||||
f"参考成交价 {_fmt(lv['entry'])} ← 回测口径(次根开盘)",
|
||||
qty_line,
|
||||
f"{LEVERAGE:.0f}x 逐仓 → 占用保证金 {margin:,.1f} USDT"
|
||||
f" · 触止损亏 {notional * SL_ATR * atr_pct:,.2f} USDT"
|
||||
f"(保证金的 {loss_pct:.1f}%)",
|
||||
f"距参考价成立 {age_s:.1f}s(含数据延迟 {lag_ms:.0f}ms,不可压缩)",
|
||||
"",
|
||||
f"止损 {_fmt(lv['stop'])} (2 ATR,stop-market,全仓)",
|
||||
f"减半 {_fmt(lv['scale'])} (3 ATR,限价 maker)",
|
||||
f"目标 {_fmt(lv['runner'])} (8 ATR,限价 maker,剩余半仓)",
|
||||
f"超时 {MAXB} 分钟后市价平(剩余半仓止损仍在 2 ATR,不移成本)",
|
||||
"",
|
||||
f"ATR {atr_pct * 1e4:.1f}bp · 滑点预算 {budget_bp:.1f}bp",
|
||||
f"→ 实际成交偏离参考价超过 {budget_bp:.1f}bp 就不值得做",
|
||||
]
|
||||
if stale:
|
||||
lines.append("")
|
||||
lines.append("时效已过:成交价已不是回测那个价,宁可漏做。")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
async def send(text: str) -> None:
|
||||
"""推一条。传输层复用生产侧的 `live/tg.py`,这里不再维护第二份。
|
||||
|
||||
方向与 `signal_bus` 一致:**生产持有实现,研究侧反过来 import**。反过来
|
||||
写成生产 import 研究侧,就等于把研究侧的依赖树绑到实盘进程上。
|
||||
"""
|
||||
from pathlib import Path
|
||||
import sys
|
||||
p = str(Path(__file__).resolve().parents[2] / "live")
|
||||
if p not in sys.path:
|
||||
sys.path.insert(0, p)
|
||||
import tg
|
||||
await tg.send(text)
|
||||
|
||||
|
||||
async def push_signal(sym: str, direction: int, entry: float, atr_pct: float,
|
||||
kline_ts: int, lag_ms: float) -> None:
|
||||
"""去重后推一条信号。
|
||||
|
||||
去重键取 (币, K线时刻, 方向):同一根被重复处理(补根、池重建后重放)不该
|
||||
推两次,否则人会开两次仓。
|
||||
|
||||
时效的起点是 `kline_ts` 而不是信号产生时刻——参考成交价(次根开盘)就是
|
||||
在 kline_ts 那一刻存在的。从信号时刻起算会漏掉数据延迟加计算那 0.5~1.5s,
|
||||
而那段是无法压缩的固定成本,必须计入。
|
||||
"""
|
||||
if not ENABLED:
|
||||
return
|
||||
key = (sym, int(kline_ts), int(direction))
|
||||
if key in _sent:
|
||||
return
|
||||
_sent.add(key)
|
||||
if len(_sent) > 5000:
|
||||
_sent.clear()
|
||||
|
||||
from lib.shadow_budget import budget_of
|
||||
b = budget_of(sym)
|
||||
if not (b == b): # nan:该币当前环境不可做(如 TRX)
|
||||
print(f" [TG] {sym} 无预算(当前环境不可做),不推", flush=True)
|
||||
return
|
||||
age = time.time() - kline_ts / 1000.0
|
||||
rule = (await load_rules()).get(sym.upper())
|
||||
await send(build(sym, direction, entry, atr_pct, kline_ts, lag_ms, b, age,
|
||||
rule))
|
||||
@@ -0,0 +1,306 @@
|
||||
"""前置测量二:venue 对齐——Bitget 与 Binance 的 1m 是不是同一批信号。
|
||||
|
||||
研究数据全部来自 Binance,影子交易器却跑在 Bitget。若两家的 1m K 线有差异,
|
||||
信号集就会不同,而这个差异会被误记到滑点账上——那样收集一两周也不可归因。
|
||||
|
||||
所以先把两家同期的 1m 拉齐,跑同一套管线,比三件事:
|
||||
K 线层 时间戳缺口、close 价差(bp)、high/low 差异
|
||||
信号层 原始 fast_bsp3 的重合率
|
||||
过滤后 加 5m 同向过滤后的重合率(这才是实际要交易的那批)
|
||||
|
||||
重合率高 → 后面测到的滑点可以直接对照 Binance 回测的 3.9bp 预算。
|
||||
重合率低 → 必须先补 Bitget 自己的回测基线,否则实验不可归因。
|
||||
|
||||
输出 out/venue_parity.csv。Bitget 数据缓存在 live/cache/,重跑不必再拉。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
|
||||
os.environ.setdefault(v, "1")
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
RESEARCH = HERE.parent
|
||||
sys.path.insert(0, str(RESEARCH))
|
||||
sys.path.insert(0, str(RESEARCH.parent))
|
||||
pd.set_option("display.width", 240)
|
||||
|
||||
CACHE = HERE / "cache"
|
||||
LTF, HTF = "1m", "5m"
|
||||
HTF_RATIO = 5
|
||||
SYMS = ("BTC", "ETH", "SOL")
|
||||
NUMERIC = ("open", "high", "low", "close", "volume")
|
||||
|
||||
|
||||
def _exchange():
|
||||
import ccxt
|
||||
# 这台机器在新加坡,直连 Bitget 0.30s。绝不要照抄交接文档里 Mac 的代理配置,
|
||||
# 代理会把延迟放大到秒级,测出来的滑点就是代理的账。
|
||||
# rateLimit 默认 50ms,连拉上千页 history-candles 会被 429,放宽到 120ms
|
||||
return ccxt.bitget({"options": {"defaultType": "swap"},
|
||||
"enableRateLimit": True, "rateLimit": 120})
|
||||
|
||||
|
||||
def fetch_bitget(sym: str, tf: str, days: int, refresh: bool = False) -> pd.DataFrame:
|
||||
"""分页拉 Bitget 永续 K 线,落盘缓存,列结构对齐 lib/data.py。"""
|
||||
CACHE.mkdir(parents=True, exist_ok=True)
|
||||
path = CACHE / f"bitget_{sym}_{tf}_{days}d.feather"
|
||||
if path.exists() and not refresh:
|
||||
return pd.read_feather(path)
|
||||
|
||||
ex = _exchange()
|
||||
pair = f"{sym}/USDT:USDT"
|
||||
period_ms = ex.parse_timeframe(tf) * 1000
|
||||
t0 = time.perf_counter()
|
||||
calls = 0
|
||||
|
||||
# 远端 history-candles 每页硬上限 200 根,而 ccxt 会按 limit 推算 endTime,
|
||||
# 只把窗口末尾的 200 根还给你。若照 limit=1000 步进,每页就白丢 800 根——
|
||||
# 210 天曾因此只拿到应有量的 31%。故分页一律按 200 走。
|
||||
PAGE = 200
|
||||
|
||||
def one(since: int) -> list:
|
||||
"""单次取数并退避重试。history-candles 连拉上千次会触发 429。"""
|
||||
for attempt in range(6):
|
||||
try:
|
||||
return ex.fetch_ohlcv(pair, tf, since=since, limit=PAGE)
|
||||
except Exception as e:
|
||||
if attempt == 5:
|
||||
raise
|
||||
wait = 2 ** attempt
|
||||
print(f" {sym} {tf}: {type(e).__name__},{wait}s 后重试",
|
||||
flush=True)
|
||||
time.sleep(wait)
|
||||
return []
|
||||
|
||||
def page(start: int, stop: int) -> list:
|
||||
"""向前分页。Bitget 把 since 当开区间,故每次从上一批最后一根重取,
|
||||
边界少的那一根靠去重消化。"""
|
||||
nonlocal calls
|
||||
got, since = [], start
|
||||
while since < stop:
|
||||
batch = one(since)
|
||||
calls += 1
|
||||
if len(batch) < 2:
|
||||
break
|
||||
got.extend(batch)
|
||||
if batch[-1][0] <= since:
|
||||
break
|
||||
since = batch[-1][0]
|
||||
if calls % 200 == 0:
|
||||
print(f" {sym} {tf}: {len(got)} 根 / {calls} 次请求", flush=True)
|
||||
return got
|
||||
|
||||
now = ex.milliseconds()
|
||||
rows = page(now - days * 86_400_000, now)
|
||||
|
||||
# 补缺口:远端接口一次只给 200 根,个别区段仍可能漏,逐个补到补不动为止
|
||||
for _ in range(5):
|
||||
ts = np.unique(np.array([r[0] for r in rows], dtype="int64"))
|
||||
if len(ts) < 2:
|
||||
break
|
||||
holes = np.where(np.diff(ts) > period_ms)[0]
|
||||
if not len(holes):
|
||||
break
|
||||
before = len(ts)
|
||||
for i in holes:
|
||||
rows.extend(page(int(ts[i]), int(ts[i + 1])))
|
||||
if len(np.unique([r[0] for r in rows])) <= before:
|
||||
break
|
||||
|
||||
df = pd.DataFrame(rows, columns=["timestamp", *NUMERIC])
|
||||
df["timestamp"] = df["timestamp"].astype("int64")
|
||||
for c in NUMERIC:
|
||||
df[c] = pd.to_numeric(df[c], errors="coerce")
|
||||
df = (df.dropna(subset=list(NUMERIC))
|
||||
.drop_duplicates(subset=["timestamp"])
|
||||
.sort_values("timestamp")
|
||||
.reset_index(drop=True))
|
||||
df["date"] = (pd.to_datetime(df["timestamp"], unit="ms", utc=True)
|
||||
.dt.tz_convert("Asia/Shanghai"))
|
||||
df = df[["timestamp", "date", *NUMERIC]]
|
||||
gap = int(((np.diff(df["timestamp"].to_numpy()) // period_ms) - 1).clip(0).sum())
|
||||
print(f" {sym} {tf}: {len(df)} 根,{calls} 次请求,"
|
||||
f"{time.perf_counter() - t0:.1f}s,残余缺口 {gap} 根", flush=True)
|
||||
df.to_feather(path)
|
||||
return df
|
||||
|
||||
|
||||
def pipeline(df_l: pd.DataFrame, df_h: pd.DataFrame) -> pd.DataFrame:
|
||||
"""全量口径跑一遍:原始信号 + 5m 同向过滤,返回带时间戳的信号表。"""
|
||||
from chanlun import TF_DF
|
||||
from lib.fast_bsp3 import find_fast_bsp3
|
||||
from lib.fx_signal import extract_fx_signals, signals_to_frame
|
||||
from lib.nested_bsp import attach_htf_context, htf_fx_timeline
|
||||
from lib.nested_level import build_htf_zones
|
||||
|
||||
chan_l = TF_DF(df_l, 1, LTF)
|
||||
cdf = chan_l.dataframe
|
||||
zones = build_htf_zones(cdf, LTF, chan=chan_l)
|
||||
if zones.empty:
|
||||
return pd.DataFrame()
|
||||
sig = find_fast_bsp3(cdf, zones.reset_index(drop=True))
|
||||
if sig.empty:
|
||||
return pd.DataFrame()
|
||||
|
||||
chan_h = TF_DF(df_h, 1, HTF)
|
||||
hdf = chan_h.dataframe
|
||||
tl = htf_fx_timeline(signals_to_frame(extract_fx_signals(chan_h, hdf)), hdf)
|
||||
full = attach_htf_context(sig, cdf, tl, "h1")
|
||||
full["entry_ts"] = cdf["timestamp"].to_numpy()[full["entry_idx"].astype(int)]
|
||||
return full
|
||||
|
||||
|
||||
def overlap(a: pd.DataFrame, b: pd.DataFrame, period_ms: int,
|
||||
tol_bars: int = 1) -> dict:
|
||||
"""按时间戳比对两个信号集。方向也必须一致才算命中。"""
|
||||
if a.empty or b.empty:
|
||||
return {"a": len(a), "b": len(b), "同根": np.nan, f"±{tol_bars}根": np.nan}
|
||||
bt = b["entry_ts"].to_numpy()
|
||||
bd = b["direction"].to_numpy()
|
||||
exact = near = 0
|
||||
for ts, d in zip(a["entry_ts"].to_numpy(), a["direction"].to_numpy()):
|
||||
hit = np.where((bt == ts) & (bd == d))[0]
|
||||
if len(hit):
|
||||
exact += 1
|
||||
near += 1
|
||||
continue
|
||||
if np.any((np.abs(bt - ts) <= tol_bars * period_ms) & (bd == d)):
|
||||
near += 1
|
||||
return {"a": len(a), "b": len(b),
|
||||
"同根": exact / len(a), f"±{tol_bars}根": near / len(a)}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--days", type=int, default=30)
|
||||
ap.add_argument("--symbols", default="BTC,ETH,SOL")
|
||||
ap.add_argument("--warmup", type=int, default=2000,
|
||||
help="丢弃前若干根的信号,避开中枢左边界效应")
|
||||
ap.add_argument("--refresh", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
||||
from lib.data import load_local
|
||||
|
||||
syms = [s.strip() for s in args.symbols.split(",")]
|
||||
period_ms = 60_000
|
||||
print(f"[venue 对齐] {syms} · 近 {args.days} 天 1m · "
|
||||
f"预热丢弃 {args.warmup} 根\n", flush=True)
|
||||
|
||||
bar_rows, sig_rows = [], []
|
||||
for sym in syms:
|
||||
print(f"── {sym}", flush=True)
|
||||
bg_l = fetch_bitget(sym, LTF, args.days, args.refresh)
|
||||
bg_h = fetch_bitget(sym, HTF, args.days, args.refresh)
|
||||
|
||||
pair = f"{sym}/USDT:USDT"
|
||||
bn_l_all = load_local(pair, LTF)
|
||||
bn_h_all = load_local(pair, HTF)
|
||||
if bn_l_all is None or bn_h_all is None:
|
||||
print(f" 跳过:本地无 Binance 数据")
|
||||
continue
|
||||
|
||||
# 只比两家都有的那段时间
|
||||
lo = max(bg_l["timestamp"].min(), bn_l_all["timestamp"].min())
|
||||
hi = min(bg_l["timestamp"].max(), bn_l_all["timestamp"].max())
|
||||
bg_l = bg_l[(bg_l.timestamp >= lo) & (bg_l.timestamp <= hi)].reset_index(drop=True)
|
||||
bn_l = bn_l_all[(bn_l_all.timestamp >= lo) & (bn_l_all.timestamp <= hi)].reset_index(drop=True)
|
||||
bg_h = bg_h[bg_h.timestamp <= hi].reset_index(drop=True)
|
||||
bn_h = bn_h_all[(bn_h_all.timestamp >= bg_h["timestamp"].min())
|
||||
& (bn_h_all.timestamp <= hi)].reset_index(drop=True)
|
||||
|
||||
span_d = (hi - lo) / 86_400_000
|
||||
expect = int((hi - lo) / period_ms) + 1
|
||||
# K 线层比对
|
||||
m = bg_l.merge(bn_l, on="timestamp", suffixes=("_bg", "_bn"))
|
||||
dc = (m["close_bg"] - m["close_bn"]) / m["close_bn"] * 1e4
|
||||
dh = (m["high_bg"] - m["high_bn"]) / m["high_bn"] * 1e4
|
||||
dl = (m["low_bg"] - m["low_bn"]) / m["low_bn"] * 1e4
|
||||
bar_rows.append({
|
||||
"品种": sym, "重叠天数": round(span_d, 1),
|
||||
"Bitget根数": len(bg_l), "Binance根数": len(bn_l),
|
||||
"应有根数": expect,
|
||||
"Bitget缺口": expect - len(bg_l), "Binance缺口": expect - len(bn_l),
|
||||
"共有根数": len(m),
|
||||
"close中位差": f"{dc.median():+.2f}bp",
|
||||
"close绝对差P95": f"{dc.abs().quantile(.95):.2f}bp",
|
||||
"high绝对差P95": f"{dh.abs().quantile(.95):.2f}bp",
|
||||
"low绝对差P95": f"{dl.abs().quantile(.95):.2f}bp",
|
||||
})
|
||||
print(f" K线:重叠 {span_d:.1f} 天,共有 {len(m)} 根,"
|
||||
f"close 中位差 {dc.median():+.2f}bp,P95 {dc.abs().quantile(.95):.2f}bp",
|
||||
flush=True)
|
||||
|
||||
# 信号层比对
|
||||
t0 = time.perf_counter()
|
||||
s_bg = pipeline(bg_l, bg_h)
|
||||
s_bn = pipeline(bn_l, bn_h)
|
||||
print(f" 管线跑完 {time.perf_counter() - t0:.1f}s", flush=True)
|
||||
if s_bg.empty or s_bn.empty:
|
||||
print(" 信号为空,跳过信号层")
|
||||
continue
|
||||
|
||||
cut_bg = bg_l["timestamp"].to_numpy()[min(args.warmup, len(bg_l) - 1)]
|
||||
cut_bn = bn_l["timestamp"].to_numpy()[min(args.warmup, len(bn_l) - 1)]
|
||||
cut = max(cut_bg, cut_bn)
|
||||
s_bg = s_bg[s_bg.entry_ts >= cut]
|
||||
s_bn = s_bn[s_bn.entry_ts >= cut]
|
||||
f_bg = s_bg[s_bg["h1_agree"] == 1]
|
||||
f_bn = s_bn[s_bn["h1_agree"] == 1]
|
||||
|
||||
for tag, x, y in (("原始", s_bg, s_bn), ("5m同向后", f_bg, f_bn)):
|
||||
o1 = overlap(x, y, period_ms) # Bitget 的信号有多少在 Binance 也有
|
||||
o2 = overlap(y, x, period_ms) # 反向
|
||||
sig_rows.append({
|
||||
"品种": sym, "口径": tag,
|
||||
"Bitget信号": o1["a"], "Binance信号": o1["b"],
|
||||
"BG→BN同根": f"{o1['同根'] * 100:.1f}%",
|
||||
"BG→BN±1根": f"{o1['±1根'] * 100:.1f}%",
|
||||
"BN→BG同根": f"{o2['同根'] * 100:.1f}%",
|
||||
"BN→BG±1根": f"{o2['±1根'] * 100:.1f}%",
|
||||
})
|
||||
print(f" {tag}:Bitget {o1['a']} 笔 / Binance {o1['b']} 笔,"
|
||||
f"同根重合 {o1['同根'] * 100:.1f}%,±1根 {o1['±1根'] * 100:.1f}%",
|
||||
flush=True)
|
||||
|
||||
if not bar_rows:
|
||||
print("无结果")
|
||||
return
|
||||
|
||||
print("\n" + "=" * 120)
|
||||
print("########## 1. K 线层 ##########")
|
||||
tb_bar = pd.DataFrame(bar_rows)
|
||||
print(tb_bar.to_string(index=False))
|
||||
print(" 缺口是「应有根数 − 实际根数」,永续在极端行情或维护时会漏推。")
|
||||
|
||||
print("\n########## 2. 信号层 ##########")
|
||||
tb_sig = pd.DataFrame(sig_rows)
|
||||
print(tb_sig.to_string(index=False))
|
||||
|
||||
print("\n########## 结论 ##########")
|
||||
fin = tb_sig[tb_sig["口径"] == "5m同向后"]
|
||||
if not fin.empty:
|
||||
v = fin["BG→BN同根"].str.rstrip("%").astype(float)
|
||||
print(f" 过滤后口径的同根重合率:{v.min():.1f}% ~ {v.max():.1f}%,"
|
||||
f"均值 {v.mean():.1f}%")
|
||||
print(" 高 → 滑点可直接对照 Binance 回测的 3.9bp 预算;")
|
||||
print(" 低 → 必须先补 Bitget 自己的 1m 回测基线,否则实验不可归因。")
|
||||
|
||||
out_dir = RESEARCH / "out"
|
||||
tb_bar.to_csv(out_dir / "venue_parity_bars.csv", index=False)
|
||||
tb_sig.to_csv(out_dir / "venue_parity_signals.csv", index=False)
|
||||
print(f"\n产物写入 {out_dir}/venue_parity_bars.csv 与 venue_parity_signals.csv")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,222 @@
|
||||
"""逐根对拍「全量重算」与「增量追加」,并量提速。
|
||||
|
||||
## 为什么必须逐根对拍,不能引用 HANDOFF §5.5
|
||||
|
||||
§5.5 验的是 step46 那批用例(走 bsp_list 那条链),且是「追加 150~200 根 vs
|
||||
全量重建」的整体哈希。影子路径不同:
|
||||
|
||||
- 走 find_fast_bsp3 + build_htf_zones + htf_fx_timeline + attach_htf_context
|
||||
- 流式对象**跨根复用**,而 worker 轮流拿多个币,同一条流可能隔几根才被
|
||||
再次追加。状态污染只会让信号悄悄换一批,不报错、不崩
|
||||
|
||||
而且代码阅读已经暴露一处偏差:`init_stream/append_bar` 从不调 `cal_trend`
|
||||
(它只在 `get_klc_list` 里),所以增量路径下 `klc.trend` 恒为 UNKNOWN。
|
||||
HANDOFF 说「笔的计算依赖 klc.trend」——若为真,增量的笔就和全量不同。
|
||||
那句话所引的 bi.py:221 其实在 `cal_trend` 自己的循环里,不是 `cal_bi_list`
|
||||
的依赖。**这条只能由对拍来定论**,不能靠读代码。
|
||||
|
||||
## 判据
|
||||
|
||||
逐字段相同,排除 last_idx/n_bars(随窗口长度必然变,见 verify_window_sens)
|
||||
与计时字段。数量相同而标志不同一样算失败。
|
||||
|
||||
模拟真实调用模式:连续推进,且每根都按「全量」和「增量」各算一次,增量那侧
|
||||
复用同一条流。
|
||||
|
||||
python research/live/verify_incr_parity.py --syms BTC,ETH,SOL --n 300
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
for _v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
|
||||
os.environ.setdefault(_v, "1")
|
||||
|
||||
HERE = Path(__file__).resolve()
|
||||
sys.path.insert(0, str(HERE.parents[1]))
|
||||
sys.path.insert(0, str(HERE.parents[2]))
|
||||
sys.path.insert(0, str(HERE.parent))
|
||||
|
||||
BASE_L, BASE_H = 2001, 801
|
||||
|
||||
|
||||
def run_one(sym: str, cache: Path, n: int, start_at: int | None) -> dict:
|
||||
import shadow_signal as ss
|
||||
from verify_lean_parity import SKIP, WINDOW_KEYS, canon, load, signal_bars
|
||||
|
||||
skip = SKIP + WINDOW_KEYS + ("stream_bars",)
|
||||
l_all, h_all = load(sym, "1m", cache), load(sym, "5m", cache)
|
||||
l_ts = l_all["timestamp"].to_numpy("int64")
|
||||
h_ts = h_all["timestamp"].to_numpy("int64")
|
||||
|
||||
# 从最后一个信号根往前 n 根开始,保证这段里一定有信号分支被执行
|
||||
if start_at is None:
|
||||
try:
|
||||
sb = signal_bars(sym, cache)
|
||||
sb = sb[(sb > BASE_L + n) & (sb < len(l_all) - 1)]
|
||||
start_at = int(sb[-1]) - n + 5 if len(sb) else BASE_L + 10
|
||||
except Exception:
|
||||
start_at = BASE_L + 10
|
||||
ends = [e for e in range(start_at, start_at + n) if e < len(l_all) - 1]
|
||||
if not ends:
|
||||
raise RuntimeError("窗口不足")
|
||||
|
||||
ss._STREAMS.clear()
|
||||
same = diff = 0
|
||||
t_full = t_incr = 0.0
|
||||
n_hits = 0
|
||||
first = None
|
||||
rebuilds = 0
|
||||
prev_grown = 0
|
||||
|
||||
for e in ends:
|
||||
hi = int(np.searchsorted(h_ts, l_ts[e], side="right"))
|
||||
df_l = l_all.iloc[e - BASE_L + 1:e + 1]
|
||||
df_h = h_all.iloc[max(0, hi - BASE_H):hi]
|
||||
entry = float(l_all["open"].to_numpy(float)[e + 1])
|
||||
|
||||
t0 = time.perf_counter()
|
||||
rf = ss.compute(df_l.copy(), df_h.copy(), entry, incr=False)
|
||||
t_full += time.perf_counter() - t0
|
||||
|
||||
t0 = time.perf_counter()
|
||||
ri = ss.compute(df_l.copy(), df_h.copy(), entry, sym=sym, incr=True)
|
||||
t_incr += time.perf_counter() - t0
|
||||
|
||||
grown = len(ss._STREAMS[(sym, "1m")][0].dataframe)
|
||||
if grown <= prev_grown:
|
||||
rebuilds += 1
|
||||
prev_grown = grown
|
||||
|
||||
n_hits += len(rf.get("hits") or [])
|
||||
if canon(rf, skip) == canon(ri, skip):
|
||||
same += 1
|
||||
else:
|
||||
diff += 1
|
||||
if first is None:
|
||||
first = (e, canon(rf, skip), canon(ri, skip))
|
||||
|
||||
k = len(ends)
|
||||
print(f" {k} 根 · 一致 {same} · 不一致 {diff} · 命中 {n_hits} 个 · "
|
||||
f"重建 {rebuilds} 次 · 末窗 {prev_grown} 根")
|
||||
print(f" 单根 全量 {t_full / k * 1000:.1f}ms → "
|
||||
f"增量 {t_incr / k * 1000:.1f}ms "
|
||||
f"({t_full / max(t_incr, 1e-9):.2f}x)")
|
||||
if first:
|
||||
e, a, b = first
|
||||
print(f" ⚠ 首个分歧 idx={e}\n 全量: {a[:300]}\n 增量: {b[:300]}")
|
||||
ss._STREAMS.clear()
|
||||
del l_all, h_all
|
||||
return {"sym": sym, "n": k, "same": same, "diff": diff, "hits": n_hits,
|
||||
"full_ms": t_full / k * 1000, "incr_ms": t_incr / k * 1000}
|
||||
|
||||
|
||||
def interleave(sym: str, cache: Path, n: int, nw: int) -> dict:
|
||||
"""模拟多 worker 交错:nw 份独立缓存轮流接同一个币。
|
||||
|
||||
这是单进程对拍覆盖不到的路径。`ProcessPoolExecutor` 不保证同一个币落到
|
||||
同一个 worker,所以每个 worker 只能隔 nw 根才再见到这个币,一次要补 nw
|
||||
根。补根走的是 `for row in df[ts > last_ts]` 那个循环——逻辑上等价于连续
|
||||
追加 nw 次,但「等价」是推理,没实测过。
|
||||
|
||||
缓存是 worker 进程内的 dict、键含 symbol,所以不存在「worker A 的状态被
|
||||
worker B 读到」或「拿到别的币的状态」。亲和性影响的是内存(每个 worker
|
||||
最终缓存全部币)与补根次数,不影响正确性——本函数就是来证这一点的。
|
||||
"""
|
||||
import shadow_signal as ss
|
||||
from verify_lean_parity import SKIP, WINDOW_KEYS, canon, load, signal_bars
|
||||
|
||||
skip = SKIP + WINDOW_KEYS + ("stream_bars",)
|
||||
l_all, h_all = load(sym, "1m", cache), load(sym, "5m", cache)
|
||||
l_ts = l_all["timestamp"].to_numpy("int64")
|
||||
h_ts = h_all["timestamp"].to_numpy("int64")
|
||||
try:
|
||||
sb = signal_bars(sym, cache)
|
||||
sb = sb[(sb > BASE_L + n) & (sb < len(l_all) - 1)]
|
||||
start = int(sb[-1]) - n + 5 if len(sb) else BASE_L + 10
|
||||
except Exception:
|
||||
start = BASE_L + 10
|
||||
ends = [e for e in range(start, start + n) if e < len(l_all) - 1]
|
||||
|
||||
caches: list[dict] = [{} for _ in range(nw)]
|
||||
same = diff = n_hits = 0
|
||||
first = None
|
||||
for j, e in enumerate(ends):
|
||||
hi = int(np.searchsorted(h_ts, l_ts[e], side="right"))
|
||||
df_l = l_all.iloc[e - BASE_L + 1:e + 1]
|
||||
df_h = h_all.iloc[max(0, hi - BASE_H):hi]
|
||||
entry = float(l_all["open"].to_numpy(float)[e + 1])
|
||||
|
||||
rf = ss.compute(df_l.copy(), df_h.copy(), entry, incr=False)
|
||||
# 轮流换缓存 = 轮流换 worker
|
||||
ss._STREAMS = caches[j % nw]
|
||||
ri = ss.compute(df_l.copy(), df_h.copy(), entry, sym=sym, incr=True)
|
||||
|
||||
n_hits += len(rf.get("hits") or [])
|
||||
if canon(rf, skip) == canon(ri, skip):
|
||||
same += 1
|
||||
else:
|
||||
diff += 1
|
||||
if first is None:
|
||||
first = (e, canon(rf, skip), canon(ri, skip))
|
||||
|
||||
grown = [len(c[(sym, "1m")][0].dataframe) for c in caches
|
||||
if (sym, "1m") in c]
|
||||
print(f" {len(ends)} 根 · {nw} 份缓存轮流 · 一致 {same} · 不一致 {diff}"
|
||||
f" · 命中 {n_hits} 个 · 各缓存末窗 {grown}")
|
||||
if first:
|
||||
e, a, b = first
|
||||
print(f" ⚠ 首个分歧 idx={e}\n 全量: {a[:300]}\n 增量: {b[:300]}")
|
||||
ss._STREAMS = {}
|
||||
del l_all, h_all
|
||||
return {"sym": sym, "n": len(ends), "same": same, "diff": diff,
|
||||
"hits": n_hits, "full_ms": 0.0, "incr_ms": 0.0}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--interleave", type=int, default=0,
|
||||
help="模拟这么多个 worker 轮流接同一个币(一次补多根)")
|
||||
ap.add_argument("--syms", default="BTC,ETH,SOL")
|
||||
ap.add_argument("--cache", default="research/live/cache")
|
||||
ap.add_argument("--n", type=int, default=300)
|
||||
ap.add_argument("--start", type=int, default=None)
|
||||
a = ap.parse_args()
|
||||
|
||||
rows = []
|
||||
for sym in a.syms.split(","):
|
||||
print(f"\n{'=' * 70}\n{sym}")
|
||||
try:
|
||||
if a.interleave:
|
||||
rows.append(interleave(sym, Path(a.cache), a.n, a.interleave))
|
||||
else:
|
||||
rows.append(run_one(sym, Path(a.cache), a.n, a.start))
|
||||
except Exception as e:
|
||||
print(f" 跳过:{e!r}")
|
||||
if not rows:
|
||||
return
|
||||
d = pd.DataFrame(rows)
|
||||
print(f"\n\n{'=' * 70}\n汇总\n")
|
||||
print(f" 对拍 {int(d['n'].sum()):,} 根 · 不一致 {int(d['diff'].sum())} · "
|
||||
f"命中 {int(d['hits'].sum())} 个")
|
||||
if d["incr_ms"].sum() > 0:
|
||||
print(f" 单根 全量 {d['full_ms'].mean():.1f}ms → "
|
||||
f"增量 {d['incr_ms'].mean():.1f}ms "
|
||||
f"({d['full_ms'].sum() / max(d['incr_ms'].sum(), 1e-9):.2f}x)")
|
||||
if int(d["diff"].sum()) == 0:
|
||||
print("\n 逐字段一致,增量可以上线。")
|
||||
else:
|
||||
print("\n ⛔ 有分歧,不要上线。增量流的状态与全量重建不等价。")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,248 @@
|
||||
"""在影子信号路径上实测 lean 与 full 是否等价,并量提速。
|
||||
|
||||
## 为什么不能直接引用 step46 的对拍结论
|
||||
|
||||
step46 固化的是 klc/笔/中枢/`bsp_list`/被消费列的哈希,走的是 `bsp_list` 那条
|
||||
链。影子路径走的是另一条:`find_fast_bsp3` + `build_htf_zones` +
|
||||
`htf_fx_timeline` + `attach_htf_context`。两条链读的东西不完全一样,所以
|
||||
「lean ≡ full」在 step46 用例上成立,不等于在这条路径上成立。
|
||||
|
||||
静态检查显示这条链只读 `chan.dataframe` 与 `chan.klc_list`(lean 都不跳),
|
||||
但静态检查漏不掉间接依赖——`build_htf_zones` 收的是 chan 对象本体。所以逐根
|
||||
实测:同一个窗口分别用 full 和 lean 跑 `compute()`,比对返回的每一个字段。
|
||||
|
||||
判据是**逐字段完全相同**,不是「信号数量相同」。数量相同而方向或标志不同,
|
||||
会让影子测的是另一批信号,且不报错。
|
||||
|
||||
python research/live/verify_lean_parity.py --syms BTC,ETH,SOL --n 150
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
for _v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
|
||||
os.environ.setdefault(_v, "1")
|
||||
|
||||
HERE = Path(__file__).resolve()
|
||||
sys.path.insert(0, str(HERE.parents[1]))
|
||||
sys.path.insert(0, str(HERE.parents[2]))
|
||||
sys.path.insert(0, str(HERE.parent))
|
||||
|
||||
LTF_BARS, HTF_BARS = 2001, 801 # 与 shadow_hb 同窗口
|
||||
SKIP = ("inner_ms", "queue_ms") # 计时字段本就不同,不参与比对
|
||||
|
||||
|
||||
def load(sym: str, tf: str, cache: Path) -> pd.DataFrame:
|
||||
c = sorted(cache.glob(f"bitget_{sym}_{tf}_*.feather"),
|
||||
key=lambda p: p.stat().st_size, reverse=True)
|
||||
if not c:
|
||||
raise FileNotFoundError(f"没有 {sym} {tf} 缓存")
|
||||
return pd.read_feather(c[0])
|
||||
|
||||
|
||||
# 随窗口长度必然改变的记账字段。比「窗口长度会不会改信号」时要排除它们,
|
||||
# 否则一定 0/12 不一致,而那是记账字段在变,不是信号在变
|
||||
WINDOW_KEYS = ("last_idx", "n_bars")
|
||||
|
||||
|
||||
def canon(d: dict, skip: tuple = SKIP) -> str:
|
||||
"""把返回值规范化成可比较的字符串。
|
||||
|
||||
浮点直接比会被末位差异误判,但 lean 走的是同一段算术、不该有任何差异,
|
||||
所以这里**不设容差**:round 到 12 位只是为了消掉 repr 差异,真有数值
|
||||
分歧一定会被抓到。
|
||||
"""
|
||||
def norm(v):
|
||||
if isinstance(v, float):
|
||||
return None if not np.isfinite(v) else round(v, 12)
|
||||
if isinstance(v, dict):
|
||||
return {k: norm(x) for k, x in sorted(v.items())}
|
||||
if isinstance(v, (list, tuple)):
|
||||
return [norm(x) for x in v]
|
||||
if isinstance(v, (np.integer, np.floating, np.bool_)):
|
||||
return norm(v.item())
|
||||
return v
|
||||
return json.dumps({k: norm(v) for k, v in sorted(d.items())
|
||||
if k not in skip}, sort_keys=True, ensure_ascii=False)
|
||||
|
||||
|
||||
def signal_bars(sym: str, cache: Path) -> np.ndarray:
|
||||
"""全量历史里过三滤网的信号根下标。
|
||||
|
||||
随机取窗口几乎测不到信号分支——信号密度约 1/2000 根,12 个窗口命中 0 个。
|
||||
只测早退路径的「一致」是很弱的证据:lean 若真影响了中枢或笔,分歧恰恰
|
||||
出现在有信号的那些根上。所以把窗口对齐到这些根。
|
||||
|
||||
注意这些下标来自**全量历史**建的中枢,而 compute 只看 2000 根窗口,所以
|
||||
对齐后未必真的命中(这正是尚未收口的窗口左边界效应)。但命中率会从
|
||||
1/2000 提到可用水平。
|
||||
"""
|
||||
from step43_fill_aware_budget import signals_for
|
||||
_, sig = signals_for(sym, cache)
|
||||
return sig["entry_idx"].astype(int).to_numpy()
|
||||
|
||||
|
||||
def run_one(sym: str, cache: Path, n: int, step: int,
|
||||
at_signals: bool = True) -> dict:
|
||||
from shadow_signal import NUM_COLS, compute
|
||||
|
||||
l_all, h_all = load(sym, "1m", cache), load(sym, "5m", cache)
|
||||
for df in (l_all, h_all):
|
||||
for c in NUM_COLS:
|
||||
if c not in df.columns:
|
||||
raise RuntimeError(f"{sym} 缺列 {c}")
|
||||
|
||||
l_ts = l_all["timestamp"].to_numpy("int64")
|
||||
h_ts = h_all["timestamp"].to_numpy("int64")
|
||||
# 一半窗口对齐到已知信号根(测信号分支),一半均匀铺开(测早退路径)
|
||||
ends: list[int] = []
|
||||
if at_signals:
|
||||
try:
|
||||
sb = signal_bars(sym, cache)
|
||||
sb = sb[(sb > LTF_BARS) & (sb < len(l_all) - 1)]
|
||||
ends += list(sb[-(n // 2 or 1):])
|
||||
print(f" 对齐到信号根 {len(ends)} 个")
|
||||
except Exception as e:
|
||||
print(f" 取信号根失败,只用均匀窗口:{e!r}")
|
||||
ends += list(range(len(l_all) - 1, LTF_BARS, -step))[:max(n - len(ends), 1)]
|
||||
ends = sorted(set(ends))
|
||||
if not ends:
|
||||
raise RuntimeError("数据不足一个窗口")
|
||||
|
||||
n_same = n_diff = 0
|
||||
t_full = t_lean = 0.0
|
||||
first_diff = None
|
||||
n_hits_full = n_hits_lean = 0
|
||||
|
||||
for e in ends:
|
||||
df_l = l_all.iloc[e - LTF_BARS + 1:e + 1]
|
||||
# 5m 只取已收盘且不晚于 1m 窗口末尾的根,与实盘一致
|
||||
hi = int(np.searchsorted(h_ts, l_ts[e], side="right"))
|
||||
df_h = h_all.iloc[max(0, hi - HTF_BARS):hi]
|
||||
# 次根开盘价:窗口末尾的下一根,与实盘 entry_px 同义
|
||||
entry_px = float(l_all["open"].to_numpy(float)[e + 1]) \
|
||||
if e + 1 < len(l_all) else None
|
||||
|
||||
t0 = time.perf_counter()
|
||||
rf = compute(df_l.copy(), df_h.copy(), entry_px, lean=False)
|
||||
t_full += time.perf_counter() - t0
|
||||
|
||||
t0 = time.perf_counter()
|
||||
rl = compute(df_l.copy(), df_h.copy(), entry_px, lean=True)
|
||||
t_lean += time.perf_counter() - t0
|
||||
|
||||
n_hits_full += len(rf.get("hits") or [])
|
||||
n_hits_lean += len(rl.get("hits") or [])
|
||||
if canon(rf) == canon(rl):
|
||||
n_same += 1
|
||||
else:
|
||||
n_diff += 1
|
||||
if first_diff is None:
|
||||
first_diff = (e, canon(rf), canon(rl))
|
||||
|
||||
k = len(ends)
|
||||
print(f" 窗口 {k} 个 · 完全一致 {n_same} · 不一致 {n_diff}")
|
||||
print(f" 命中数 full {n_hits_full} / lean {n_hits_lean}")
|
||||
print(f" 单窗耗时 full {t_full / k * 1000:.0f}ms · "
|
||||
f"lean {t_lean / k * 1000:.0f}ms · "
|
||||
f"提速 {t_full / max(t_lean, 1e-9):.2f}x")
|
||||
if first_diff:
|
||||
e, a, b = first_diff
|
||||
print(f" ⚠ 首个分歧在窗口末尾 idx={e}")
|
||||
print(f" full: {a[:400]}")
|
||||
print(f" lean: {b[:400]}")
|
||||
return {"sym": sym, "n": k, "same": n_same, "diff": n_diff,
|
||||
"full_ms": t_full / k * 1000, "lean_ms": t_lean / k * 1000,
|
||||
"hits_full": n_hits_full, "hits_lean": n_hits_lean}
|
||||
|
||||
|
||||
def recall(syms: list[str], cache: Path, k: int = 30) -> None:
|
||||
"""全量历史找出的信号,在 2000 根窗口里还能不能复现。
|
||||
|
||||
这是窗口左边界效应的**一半**答案。窗口只有 2000 根,中枢是在窗口内重建
|
||||
的,理论上可能与全量历史建的中枢不同,从而漏掉信号。实测最近 k 笔全部
|
||||
复现,说明窗口不丢信号。
|
||||
|
||||
⚠ 另一半没答:窗口会不会**多造出**全量历史没有的信号。那个方向对实盘更
|
||||
危险(会多开仓),但要反向扫描——遍历窗口找命中、再回全量历史核对,成本
|
||||
高得多。lean 之后单窗 130ms,抽样 2 万个窗口约 43 分钟,已经可做。
|
||||
"""
|
||||
from shadow_signal import compute
|
||||
|
||||
print("全量历史找出的信号,在 2000 根窗口里的复现率\n")
|
||||
for sym in syms:
|
||||
try:
|
||||
l_all, h_all = load(sym, "1m", cache), load(sym, "5m", cache)
|
||||
l_ts = l_all["timestamp"].to_numpy("int64")
|
||||
h_ts = h_all["timestamp"].to_numpy("int64")
|
||||
sb = signal_bars(sym, cache)
|
||||
sb = sb[(sb > LTF_BARS) & (sb < len(l_all) - 1)][-k:]
|
||||
except Exception as e:
|
||||
print(f" {sym} 跳过:{e!r}")
|
||||
continue
|
||||
hit = 0
|
||||
for e in sb:
|
||||
df_l = l_all.iloc[e - LTF_BARS + 1:e + 1]
|
||||
hi = int(np.searchsorted(h_ts, l_ts[e], side="right"))
|
||||
df_h = h_all.iloc[max(0, hi - HTF_BARS):hi]
|
||||
r = compute(df_l.copy(), df_h.copy(),
|
||||
float(l_all["open"].to_numpy(float)[e + 1]))
|
||||
if any(h.get("pass_all") for h in (r.get("hits") or [])):
|
||||
hit += 1
|
||||
print(f" {sym} {hit}/{len(sb)} 复现(过全部滤网)")
|
||||
print("\n 只说明窗口不丢信号;会不会多造信号需反向扫描,见 docstring。")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--recall", action="store_true",
|
||||
help="只查窗口对全量历史信号的复现率")
|
||||
ap.add_argument("--syms", default="BTC,ETH,SOL")
|
||||
ap.add_argument("--cache", default="research/live/cache")
|
||||
ap.add_argument("--n", type=int, default=150, help="每币比对多少个窗口")
|
||||
ap.add_argument("--step", type=int, default=37,
|
||||
help="窗口间隔根数。取质数避免与任何周期共振")
|
||||
ap.add_argument("--no-signals", action="store_true",
|
||||
help="不对齐信号根(快,但测不到信号分支)")
|
||||
a = ap.parse_args()
|
||||
|
||||
if a.recall:
|
||||
recall(a.syms.split(","), Path(a.cache))
|
||||
return
|
||||
|
||||
rows = []
|
||||
for sym in a.syms.split(","):
|
||||
print(f"\n{'=' * 70}\n{sym}")
|
||||
try:
|
||||
rows.append(run_one(sym, Path(a.cache), a.n, a.step,
|
||||
at_signals=not a.no_signals))
|
||||
except Exception as e:
|
||||
print(f" 跳过:{e!r}")
|
||||
|
||||
if not rows:
|
||||
return
|
||||
d = pd.DataFrame(rows)
|
||||
print(f"\n\n{'=' * 70}\n汇总\n")
|
||||
print(f" 比对窗口 {int(d['n'].sum()):,} 个 · "
|
||||
f"不一致 {int(d['diff'].sum())} 个")
|
||||
print(f" 单窗耗时 full {d['full_ms'].mean():.0f}ms → "
|
||||
f"lean {d['lean_ms'].mean():.0f}ms "
|
||||
f"({d['full_ms'].sum() / max(d['lean_ms'].sum(), 1e-9):.2f}x)")
|
||||
if int(d["diff"].sum()) == 0:
|
||||
print("\n 逐字段完全一致,可以开 lean。")
|
||||
else:
|
||||
print("\n ⛔ 存在分歧,不要开 lean。lean 跳掉的东西这条路径确实在用。")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,179 @@
|
||||
"""验证补丁是否真把那 1.1 秒拿回来了。
|
||||
|
||||
容器内同一进程并行跑三条路径,共用时钟与网络:
|
||||
stock 上游 BitgetPerpetualCandles(取 data[0])
|
||||
patched PatchedBitgetPerpetualCandles(处理全部元素)
|
||||
raw 原始 aiohttp WS,按 data[-1] 判断换根(理论最快)
|
||||
|
||||
预期:patched ≈ raw,且比 stock 早约 1.1 秒。
|
||||
同时校验补丁没有破坏数据:两条 feed 的历史 K 线应逐根相等(补丁只影响
|
||||
最新一根的到达时刻与收盘价更新时机,不该改动已收盘的历史)。
|
||||
|
||||
docker run --rm -v $PWD:/repo:ro -v $PWD/research/out:/out \
|
||||
-w /home/hummingbot -e PYTHONPATH=/home/hummingbot:/repo/research/live \
|
||||
--entrypoint /opt/conda/envs/hummingbot/bin/python \
|
||||
hummingbot/hummingbot:latest /repo/research/live/verify_patch.py --minutes 20
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import csv
|
||||
import json
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
SYMS = ("BTC", "ETH", "SOL")
|
||||
WSS = "wss://ws.bitget.com/v2/ws/public"
|
||||
|
||||
|
||||
def out_dir() -> Path:
|
||||
p = Path("/out")
|
||||
return p if p.is_dir() else Path(__file__).resolve().parents[1] / "out"
|
||||
|
||||
|
||||
async def raw_ws(rec: dict, stop: asyncio.Event) -> None:
|
||||
import aiohttp
|
||||
|
||||
payload = {"op": "subscribe",
|
||||
"args": [{"instType": "USDT-FUTURES", "channel": "candle1m",
|
||||
"instId": f"{s}USDT"} for s in SYMS]}
|
||||
while not stop.is_set():
|
||||
try:
|
||||
async with aiohttp.ClientSession() as sess, \
|
||||
sess.ws_connect(WSS, heartbeat=20) as ws:
|
||||
await ws.send_str(json.dumps(payload))
|
||||
last: dict[str, int] = {}
|
||||
while not stop.is_set():
|
||||
msg = await ws.receive()
|
||||
if msg.type is not aiohttp.WSMsgType.TEXT:
|
||||
break
|
||||
if msg.data == "pong":
|
||||
continue
|
||||
d = json.loads(msg.data)
|
||||
if "data" not in d or "arg" not in d or not d["data"]:
|
||||
continue
|
||||
sym = d["arg"]["instId"].replace("USDT", "")
|
||||
kts = int(d["data"][-1][0]) # 关键:取末元素
|
||||
if last.get(sym) is not None and kts > last[sym]:
|
||||
rec.setdefault((sym, kts), {})["raw"] = \
|
||||
int(time.time() * 1000)
|
||||
last[sym] = max(kts, last.get(sym, 0))
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as e:
|
||||
print(f" [raw] {type(e).__name__}: {e}", flush=True)
|
||||
# 正常跳出与异常都要歇一下再重连,避免服务端持续拒绝时打成风暴
|
||||
if not stop.is_set():
|
||||
await asyncio.sleep(1)
|
||||
|
||||
|
||||
async def poll_feeds(feeds: dict, key: str, rec: dict,
|
||||
stop: asyncio.Event) -> None:
|
||||
last = {s: (int(f._candles[-1][0]) if len(f._candles) else None)
|
||||
for s, f in feeds.items()}
|
||||
while not stop.is_set():
|
||||
for s, f in feeds.items():
|
||||
if not len(f._candles):
|
||||
continue
|
||||
newest = int(f._candles[-1][0])
|
||||
if last[s] is not None and newest > last[s]:
|
||||
kts = newest * 1000 if newest < 1e12 else newest
|
||||
rec.setdefault((s, kts), {})[key] = int(time.time() * 1000)
|
||||
last[s] = newest
|
||||
await asyncio.sleep(0.01)
|
||||
|
||||
|
||||
async def main_async(minutes: int) -> None:
|
||||
from hummingbot.data_feed.candles_feed.bitget_perpetual_candles import (
|
||||
BitgetPerpetualCandles,
|
||||
)
|
||||
from patched_candles import PatchedBitgetPerpetualCandles
|
||||
|
||||
stock, patched = {}, {}
|
||||
for s in SYMS:
|
||||
stock[s] = BitgetPerpetualCandles(f"{s}-USDT", "1m", 20)
|
||||
patched[s] = PatchedBitgetPerpetualCandles(f"{s}-USDT", "1m", 20)
|
||||
for d in (stock, patched):
|
||||
for f in d.values():
|
||||
f.start()
|
||||
print(f"[补丁验证] {SYMS} · 跑 {minutes} 分钟", flush=True)
|
||||
|
||||
t0 = time.time()
|
||||
while time.time() - t0 < 120:
|
||||
if all(f.ready for d in (stock, patched) for f in d.values()):
|
||||
break
|
||||
await asyncio.sleep(0.5)
|
||||
print(f" 回填完成 {time.time() - t0:.1f}s", flush=True)
|
||||
|
||||
rec: dict = {}
|
||||
stop = asyncio.Event()
|
||||
tasks = [asyncio.create_task(raw_ws(rec, stop)),
|
||||
asyncio.create_task(poll_feeds(stock, "stock", rec, stop)),
|
||||
asyncio.create_task(poll_feeds(patched, "patched", rec, stop))]
|
||||
|
||||
deadline = time.time() + minutes * 60
|
||||
seen = set()
|
||||
while time.time() < deadline:
|
||||
await asyncio.sleep(2)
|
||||
for k, v in rec.items():
|
||||
if k in seen or not {"stock", "patched", "raw"} <= v.keys():
|
||||
continue
|
||||
seen.add(k)
|
||||
print(f" {k[0]}: patched 比 stock 早 "
|
||||
f"{v['stock'] - v['patched']}ms · 距 raw "
|
||||
f"{v['patched'] - v['raw']}ms", flush=True)
|
||||
stop.set()
|
||||
for t in tasks:
|
||||
t.cancel()
|
||||
await asyncio.gather(*tasks, return_exceptions=True)
|
||||
|
||||
# 数据一致性:两条 feed 的已收盘历史必须逐根相同
|
||||
print("\n########## 补丁是否改动了已收盘 K 线 ##########")
|
||||
for s in SYMS:
|
||||
a = [list(map(float, r)) for r in list(stock[s]._candles)[:-1]]
|
||||
b = [list(map(float, r)) for r in list(patched[s]._candles)[:-1]]
|
||||
n = min(len(a), len(b))
|
||||
ta = {int(r[0]): r for r in a[-n:]}
|
||||
tb = {int(r[0]): r for r in b[-n:]}
|
||||
common = sorted(set(ta) & set(tb))
|
||||
diff = [t for t in common if ta[t][1:6] != tb[t][1:6]]
|
||||
print(f" {s}: 共有 {len(common)} 根,OHLCV 不同 {len(diff)} 根"
|
||||
+ (f"(示例 ts={diff[:3]})" if diff else ""))
|
||||
for d in (stock, patched):
|
||||
for f in d.values():
|
||||
f.stop()
|
||||
|
||||
full = [(s, k, v) for (s, k), v in rec.items()
|
||||
if {"stock", "patched", "raw"} <= v.keys()]
|
||||
f = out_dir() / "verify_patch.csv"
|
||||
with f.open("w", newline="") as fh:
|
||||
w = csv.writer(fh)
|
||||
w.writerow(["sym", "kline_ts", "raw", "patched", "stock",
|
||||
"gain_ms", "patched_minus_raw_ms"])
|
||||
for s, k, v in sorted(full, key=lambda x: x[1]):
|
||||
w.writerow([s, k, v["raw"], v["patched"], v["stock"],
|
||||
v["stock"] - v["patched"], v["patched"] - v["raw"]])
|
||||
|
||||
print(f"\n########## 补丁收益(配对 {len(full)} 根)##########")
|
||||
print(f"{'币':<5}{'n':>5}{'早于stock中位':>14}{'距raw中位':>12}")
|
||||
for s in SYMS:
|
||||
g = [v for ss, _, v in full if ss == s]
|
||||
if not g:
|
||||
print(f" {s}: 无样本")
|
||||
continue
|
||||
gain = sorted(v["stock"] - v["patched"] for v in g)
|
||||
dr = sorted(v["patched"] - v["raw"] for v in g)
|
||||
print(f"{s:<5}{len(g):>5}{gain[len(gain) // 2]:>14}"
|
||||
f"{dr[len(dr) // 2]:>12}")
|
||||
print(f"\n产物写入 {f}")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--minutes", type=int, default=20)
|
||||
asyncio.run(main_async(ap.parse_args().minutes))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,205 @@
|
||||
"""验证 live 信号路径与 step42 的批量过滤等价。
|
||||
|
||||
live 侧每根只看最后一根、且只喂 2000 根窗口;研究侧一次性跑全量。两者
|
||||
用同一套滤网(同向 + 中枢阶梯 + ATR 门控),但**不保证逐笔一致**——
|
||||
缠论结构依赖历史,2000 根窗口是 step39 定的命中率饱和点,不是无损截断。
|
||||
|
||||
每个信号根上比三种口径:
|
||||
|
||||
批量 全量历史 + 完整 5m。这是预算的来源,是基准
|
||||
剔partial 窗口 2000 根 1m + 800 根**已收盘** 5m
|
||||
含partial 窗口,5m 末尾保留那根**尚未收盘**的。这是 live 现行做法
|
||||
|
||||
## 结论:partial 根要保留,不能剔
|
||||
|
||||
live 的 `df_h[df_h["timestamp"] < kline_ts]` 里 kline_ts 是 1m 的收盘时刻,
|
||||
而正在走的那根 5m 开盘更早,于是被保留下来——五根里有四根如此。乍看像是
|
||||
「把未收盘的根当完整根用」的口径错误,实测**反过来**:
|
||||
|
||||
BTC 25/25、ETH 24/25、SOL 23/25 与批量一致(合计 96%)
|
||||
剔掉则只有 21/25、21/25、18/25(合计 80%)
|
||||
|
||||
原因是批量口径里那根 5m 是存在的。缠论的包含处理与分型检测吃整条序列,
|
||||
凭空少一根会把结构整体挪位;保留一根「开盘价正确、高低点尚不完整」的
|
||||
近似根,比直接删掉更接近批量。
|
||||
|
||||
这也暴露了研究侧的一处残留:批量的 HTF 结构用到了那根 5m 的**最终**高低点,
|
||||
而实盘在该时刻不可能知道。`htf_fx_timeline` 的 `confirm_ts += period` 只挡住了
|
||||
分型**选取**上的未来函数,挡不住结构构建。live 用 partial 根逼近,落在 96%,
|
||||
差的那 4% 是这条残留的下界,不是可以修掉的 bug。
|
||||
|
||||
.venv/bin/python research/live/verify_signal_path.py --symbol BTC
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
HERE = Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(HERE))
|
||||
sys.path.insert(0, str(HERE.parent))
|
||||
sys.path.insert(0, str(HERE.parents[1]))
|
||||
|
||||
import numpy as np # noqa: E402
|
||||
import pandas as pd # noqa: E402
|
||||
|
||||
WINDOW_L, WINDOW_H = 2000, 800
|
||||
|
||||
|
||||
HTF_MS = 300_000
|
||||
|
||||
|
||||
def partial_htf_bar(df_l: pd.DataFrame, kline_ts: int) -> dict | None:
|
||||
"""用 1m 合成「此刻正在走的那根 5m」,复现 live 曾经喂进去的 partial 根。"""
|
||||
bucket = kline_ts // HTF_MS * HTF_MS
|
||||
if bucket >= kline_ts: # 正好落在 5m 边界,没有未收盘的根
|
||||
return None
|
||||
part = df_l[(df_l["timestamp"] >= bucket) & (df_l["timestamp"] < kline_ts)]
|
||||
if part.empty:
|
||||
return None
|
||||
date = pd.to_datetime(bucket, unit="ms", utc=True) \
|
||||
.tz_convert("Asia/Shanghai")
|
||||
return {"timestamp": bucket, "date": date,
|
||||
"open": float(part["open"].iloc[0]),
|
||||
"high": float(part["high"].max()), "low": float(part["low"].min()),
|
||||
"close": float(part["close"].iloc[-1]),
|
||||
"volume": float(part["volume"].sum())}
|
||||
|
||||
|
||||
def load(sym: str, tf: str, days: int) -> pd.DataFrame:
|
||||
f = HERE / "cache" / f"bitget_{sym}_{tf}_{days}d.feather"
|
||||
if not f.exists():
|
||||
raise SystemExit(f"缺数据 {f}")
|
||||
return pd.read_feather(f)
|
||||
|
||||
|
||||
def batch_flags(df_l: pd.DataFrame, df_h: pd.DataFrame) -> pd.DataFrame:
|
||||
"""step42_exit_tp_1m.run_one 的滤网,逐行照搬。"""
|
||||
from chanlun import TF_DF
|
||||
from lib.fast_bsp3 import find_fast_bsp3
|
||||
from lib.fx_signal import extract_fx_signals, signals_to_frame
|
||||
from lib.nested_bsp import attach_htf_context, htf_fx_timeline
|
||||
from lib.nested_level import build_htf_zones
|
||||
from lib.shadow_budget import ATR_GATE_BP
|
||||
|
||||
chan_l = TF_DF(df_l, 1, "1m")
|
||||
cdf = chan_l.dataframe
|
||||
zones = build_htf_zones(cdf, "1m", chan=chan_l).reset_index(drop=True)
|
||||
z = zones.copy()
|
||||
pg, pdn = z["zg"].shift(), z["zd"].shift()
|
||||
z["z_above"], z["z_below"] = z["zd"] > pg, z["zg"] < pdn
|
||||
z["zone_i"] = np.arange(len(z))
|
||||
|
||||
chan_h = TF_DF(df_h, 1, "5m")
|
||||
hdf = chan_h.dataframe
|
||||
tl = htf_fx_timeline(
|
||||
signals_to_frame(extract_fx_signals(chan_h, hdf)), hdf)
|
||||
|
||||
sig = find_fast_bsp3(cdf, zones)
|
||||
sig = sig.merge(z[["zone_i", "z_above", "z_below"]], on="zone_i", how="left")
|
||||
sig = attach_htf_context(sig, cdf, tl, "h1")
|
||||
|
||||
d = sig["direction"].astype(int)
|
||||
push = np.where(d == 1, sig["z_above"], sig["z_below"])
|
||||
idx = sig["entry_idx"].astype(int).to_numpy()
|
||||
entry = cdf["open"].to_numpy(float)[np.minimum(idx + 1, len(cdf) - 1)]
|
||||
atr_pct = cdf["atr"].to_numpy(float)[idx] / entry
|
||||
|
||||
out = pd.DataFrame({
|
||||
"entry_idx": idx,
|
||||
"ts": cdf["timestamp"].to_numpy()[idx],
|
||||
"direction": d.to_numpy(),
|
||||
"h1_agree": sig["h1_agree"].fillna(0).astype(int).to_numpy(),
|
||||
"ladder_ok": pd.Series(push).fillna(False).astype(int).to_numpy(),
|
||||
"atr_bp": atr_pct * 1e4,
|
||||
})
|
||||
out["gate_ok"] = (out["atr_bp"] >= ATR_GATE_BP).astype(int)
|
||||
out["pass_all"] = ((out["h1_agree"] == 1) & (out["ladder_ok"] == 1)
|
||||
& (out["gate_ok"] == 1)).astype(int)
|
||||
return out, cdf
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--symbol", default="BTC")
|
||||
ap.add_argument("--days", type=int, default=30)
|
||||
ap.add_argument("--checks", type=int, default=12)
|
||||
a = ap.parse_args()
|
||||
|
||||
df_l, df_h = load(a.symbol, "1m", a.days), load(a.symbol, "5m", a.days)
|
||||
print(f"[{a.symbol}] 1m {len(df_l)} 根 / 5m {len(df_h)} 根,跑批量滤网…",
|
||||
flush=True)
|
||||
batch, cdf = batch_flags(df_l, df_h)
|
||||
n = len(batch)
|
||||
print(f" 原始 B4/S4 {n} 个 · 同向 {int((batch.h1_agree == 1).sum())}"
|
||||
f" · 同向+阶梯 "
|
||||
f"{int(((batch.h1_agree == 1) & (batch.ladder_ok == 1)).sum())}"
|
||||
f" · 三项全过 {int(batch.pass_all.sum())}")
|
||||
print(f" ATR 中位 {batch.atr_bp.median():.2f}bp · "
|
||||
f"门控刷掉 {(1 - batch.gate_ok.mean()) * 100:.1f}%\n")
|
||||
|
||||
# 挑最近的若干个信号根做窗口复现
|
||||
from shadow_signal import compute
|
||||
cand = batch[batch["entry_idx"] >= WINDOW_L].tail(a.checks)
|
||||
if cand.empty:
|
||||
raise SystemExit("窗口内没有可核对的信号")
|
||||
|
||||
def run_window(r, with_partial: bool):
|
||||
i = int(r["entry_idx"])
|
||||
kline_ts = int(r["ts"]) + 60_000 # 信号根的收盘时刻
|
||||
wl = df_l[df_l["timestamp"] < kline_ts].tail(WINDOW_L)
|
||||
wh = df_h[df_h["timestamp"] + HTF_MS <= kline_ts].tail(WINDOW_H)
|
||||
if with_partial:
|
||||
p = partial_htf_bar(df_l, kline_ts)
|
||||
if p is not None:
|
||||
wh = pd.concat([wh, pd.DataFrame([p])], ignore_index=True)
|
||||
entry_px = float(cdf["open"].to_numpy(float)[min(i + 1, len(cdf) - 1)])
|
||||
res = compute(wl.reset_index(drop=True), wh.reset_index(drop=True),
|
||||
entry_px)
|
||||
if res.get("error"):
|
||||
raise SystemExit(f"compute 报错,测试本身有问题:{res['error']}\n"
|
||||
f"{res.get('traceback', '')}")
|
||||
return next((h for h in res["hits"]
|
||||
if h["direction"] == int(r["direction"])), None)
|
||||
|
||||
def fmt(h):
|
||||
if h is None:
|
||||
return f"{'未复现':>18}"
|
||||
return (f"{h['h1_agree']:>6}{h['ladder_ok']:>5}{h['gate_ok']:>5}")
|
||||
|
||||
print(f"{'K线时刻':<15}{'方向':>4}{' 批量':>18}{' 窗口':>18}"
|
||||
f"{' 含未收盘':>18}{' 截断':>7}{'partial':>9}")
|
||||
print(f"{'':<15}{'':>4}{'同向 阶梯 门控':>20}{'同向 阶梯 门控':>20}"
|
||||
f"{'同向 阶梯 门控':>20}")
|
||||
n_trunc = n_part = n_ok = n_bad = 0
|
||||
for _, r in cand.iterrows():
|
||||
h_ok = run_window(r, False)
|
||||
h_bad = run_window(r, True)
|
||||
t = pd.to_datetime(r["ts"], unit="ms", utc=True) \
|
||||
.tz_convert("Asia/Shanghai").strftime("%m-%d %H:%M")
|
||||
ref = (int(r.h1_agree), int(r.ladder_ok), int(r.gate_ok))
|
||||
got = None if h_ok is None else (h_ok["h1_agree"], h_ok["ladder_ok"],
|
||||
h_ok["gate_ok"])
|
||||
bad = None if h_bad is None else (h_bad["h1_agree"], h_bad["ladder_ok"],
|
||||
h_bad["gate_ok"])
|
||||
n_trunc += got != ref
|
||||
n_part += bad != got
|
||||
n_ok += got == ref
|
||||
n_bad += bad == ref
|
||||
print(f"{t:<15}{int(r.direction):>+4}"
|
||||
f"{ref[0]:>6}{ref[1]:>5}{ref[2]:>5}"
|
||||
f"{fmt(h_ok):>18}{fmt(h_bad):>18}"
|
||||
f"{'' if got == ref else '差':>7}"
|
||||
f"{'' if bad == got else '差':>9}")
|
||||
n = len(cand)
|
||||
print(f"\n 与批量(预算口径)一致:")
|
||||
print(f" 剔掉未收盘 5m 根 {n_ok}/{n}")
|
||||
print(f" 保留未收盘 5m 根 {n_bad}/{n} ← live 现行做法")
|
||||
print(f" 两种窗口口径互不相同 {n_part}/{n}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,115 @@
|
||||
"""compute() 的输出对窗口长度是否不变——增量路径的前提。
|
||||
|
||||
## 为什么这是增量路径的前提
|
||||
|
||||
`init_stream/append_bar` 没有 trim:`dataframe` 靠 `pd.concat` 无界增长。所以
|
||||
增量方案必然意味着**窗口会长大**(每根 +1),只能靠周期性 `init_stream` 重建
|
||||
拉回。于是在两次重建之间,实际窗口是 [W, W+slack] 而不是恒定 W。
|
||||
|
||||
这就把一个问题摆在前面:如果 `compute()` 的输出随窗口长度变化,增量路径等于
|
||||
静默把信号换了一批——不报错、不崩,只是测的不再是回测那批信号。
|
||||
|
||||
step39 的结论是「命中率在 2000 根饱和」。**饱和不等于不变**:再加根数不再提高
|
||||
命中率,与「结果逐字段相同」是两回事。所以要单独验。
|
||||
|
||||
判据是逐字段相同,不是命中数相同。数量相同而方向或滤网标志不同,会让影子测
|
||||
的是另一批信号。
|
||||
|
||||
python research/live/verify_window_sens.py --syms BTC,ETH,SOL
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
for _v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
|
||||
os.environ.setdefault(_v, "1")
|
||||
|
||||
HERE = Path(__file__).resolve()
|
||||
sys.path.insert(0, str(HERE.parents[1]))
|
||||
sys.path.insert(0, str(HERE.parents[2]))
|
||||
sys.path.insert(0, str(HERE.parent))
|
||||
|
||||
BASE_L, BASE_H = 2001, 801
|
||||
# 增量在两次重建之间会长这么多。取 500 是因为它对应约 8 小时,
|
||||
# 重建摊薄后单根成本仍接近纯追加
|
||||
GROW = (0, 200, 500, 1000)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--syms", default="BTC,ETH,SOL")
|
||||
ap.add_argument("--cache", default="research/live/cache")
|
||||
ap.add_argument("--n", type=int, default=40)
|
||||
a = ap.parse_args()
|
||||
|
||||
from shadow_signal import compute
|
||||
from verify_lean_parity import (SKIP, WINDOW_KEYS, canon, load,
|
||||
signal_bars)
|
||||
|
||||
# last_idx/n_bars 必然随窗口长度变。不排除的话结果一定是「全不一致」,
|
||||
# 而那说明的是记账字段在变,不是信号在变
|
||||
skip = SKIP + WINDOW_KEYS
|
||||
|
||||
cache = Path(a.cache)
|
||||
print("同一根上,只改窗口长度,比对 compute() 的全部返回字段")
|
||||
print(f"基准窗口 1m×{BASE_L} + 5m×{BASE_H};增量会让它长大,故试 "
|
||||
f"+{GROW[1:]}\n")
|
||||
|
||||
tot = {g: [0, 0] for g in GROW[1:]} # [相同, 不同]
|
||||
for sym in a.syms.split(","):
|
||||
l_all, h_all = load(sym, "1m", cache), load(sym, "5m", cache)
|
||||
l_ts = l_all["timestamp"].to_numpy("int64")
|
||||
h_ts = h_all["timestamp"].to_numpy("int64")
|
||||
try:
|
||||
sb = signal_bars(sym, cache)
|
||||
except Exception as e:
|
||||
print(f"{sym} 取信号根失败:{e!r}")
|
||||
continue
|
||||
need = BASE_L + max(GROW)
|
||||
sb = sb[(sb > need) & (sb < len(l_all) - 1)][-a.n:]
|
||||
if len(sb) == 0:
|
||||
print(f"{sym} 可用信号根不足")
|
||||
continue
|
||||
|
||||
res: dict[int, list[str]] = {g: [] for g in GROW}
|
||||
for e in sb:
|
||||
hi = int(np.searchsorted(h_ts, l_ts[e], side="right"))
|
||||
entry = float(l_all["open"].to_numpy(float)[e + 1])
|
||||
for g in GROW:
|
||||
df_l = l_all.iloc[e - (BASE_L + g) + 1:e + 1]
|
||||
df_h = h_all.iloc[max(0, hi - (BASE_H + g // 5)):hi]
|
||||
res[g].append(canon(compute(df_l.copy(), df_h.copy(), entry),
|
||||
skip=skip))
|
||||
|
||||
print(f"{sym} {len(sb)} 根信号窗口")
|
||||
for g in GROW[1:]:
|
||||
same = sum(1 for x, y in zip(res[0], res[g]) if x == y)
|
||||
tot[g][0] += same
|
||||
tot[g][1] += len(sb) - same
|
||||
print(f" +{g:>4} 根 → 一致 {same}/{len(sb)}"
|
||||
+ ("" if same == len(sb) else " ⚠ 有分歧"))
|
||||
del l_all, h_all
|
||||
|
||||
print(f"\n{'=' * 66}\n汇总\n")
|
||||
for g in GROW[1:]:
|
||||
s, d = tot[g]
|
||||
print(f" 窗口 +{g:>4} 根:一致 {s} · 不一致 {d}")
|
||||
worst = max(GROW[1:], key=lambda g: tot[g][1])
|
||||
if tot[worst][1] == 0:
|
||||
print("\n 窗口长度不影响输出,增量路径的前提成立。")
|
||||
print(" 可以按「长到 +N 根再 init_stream 重建」摊薄成本。")
|
||||
else:
|
||||
print("\n ⛔ 窗口长度会改变输出,增量路径会静默换掉一批信号。")
|
||||
print(" 此时要么每根都重建(等于没有增量),要么先把窗口效应本身收口。")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,22 @@
|
||||
品种,噪声bp,种子,笔数,胜率,毛均收益,净均收益,PF,t值,滑点余量bp,毛均bp
|
||||
BTC,0.0,0,294,49.3%,+0.0611%,+0.0111%,1.13,0.88,0.11,6.11
|
||||
BTC,0.5,0,322,53.4%,+0.0614%,+0.0114%,1.14,0.95,0.14,6.140000000000001
|
||||
BTC,0.5,1,304,51.3%,+0.0672%,+0.0172%,1.21,1.38,0.72,6.72
|
||||
BTC,1.0,0,262,53.1%,+0.0648%,+0.0148%,1.17,1.08,0.48,6.4799999999999995
|
||||
BTC,1.0,1,255,48.6%,+0.0595%,+0.0095%,1.1,0.65,-0.05,5.949999999999999
|
||||
BTC,2.0,0,196,53.6%,+0.0987%,+0.0487%,1.5,2.46,3.87,9.87
|
||||
BTC,2.0,1,191,50.3%,+0.0799%,+0.0299%,1.28,1.51,1.99,7.99
|
||||
ETH,0.0,0,288,55.6%,+0.1231%,+0.0731%,1.9,4.15,6.31,12.31
|
||||
ETH,0.5,0,293,52.9%,+0.0949%,+0.0449%,1.46,2.52,3.49,9.49
|
||||
ETH,0.5,1,291,60.1%,+0.1236%,+0.0736%,1.87,4.2,6.36,12.36
|
||||
ETH,1.0,0,280,51.1%,+0.1004%,+0.0504%,1.48,2.65,4.04,10.040000000000001
|
||||
ETH,1.0,1,270,54.8%,+0.1142%,+0.0642%,1.64,3.24,5.42,11.42
|
||||
ETH,2.0,0,240,55.8%,+0.1182%,+0.0682%,1.61,3.14,5.82,11.82
|
||||
ETH,2.0,1,222,58.1%,+0.1474%,+0.0974%,1.93,4.07,8.74,14.74
|
||||
SOL,0.0,0,245,52.2%,+0.1001%,+0.0501%,1.44,2.41,4.01,10.01
|
||||
SOL,0.5,0,232,53.4%,+0.1000%,+0.0500%,1.44,2.47,4.0,10.0
|
||||
SOL,0.5,1,234,57.7%,+0.1332%,+0.0832%,1.75,3.68,7.32,13.320000000000002
|
||||
SOL,1.0,0,246,57.7%,+0.1362%,+0.0862%,1.82,4.09,7.62,13.62
|
||||
SOL,1.0,1,230,54.8%,+0.1191%,+0.0691%,1.58,3.01,5.91,11.91
|
||||
SOL,2.0,0,195,58.5%,+0.1486%,+0.0986%,1.83,3.84,8.86,14.860000000000001
|
||||
SOL,2.0,1,193,55.4%,+0.1327%,+0.0827%,1.67,3.19,7.27,13.270000000000001
|
||||
|
@@ -0,0 +1,8 @@
|
||||
品种,噪声bp,种子,笔数,胜率,毛均收益,净均收益,PF,t值,滑点余量bp
|
||||
BTC,0.0,0,294,49.3%,+0.0611%,+0.0111%,1.13,0.88,0.11
|
||||
BTC,0.5,0,322,53.4%,+0.0614%,+0.0114%,1.14,0.95,0.14
|
||||
BTC,0.5,1,304,51.3%,+0.0672%,+0.0172%,1.21,1.38,0.72
|
||||
BTC,1.0,0,262,53.1%,+0.0648%,+0.0148%,1.17,1.08,0.48
|
||||
BTC,1.0,1,255,48.6%,+0.0595%,+0.0095%,1.1,0.65,-0.05
|
||||
BTC,2.0,0,196,53.6%,+0.0987%,+0.0487%,1.5,2.46,3.87
|
||||
BTC,2.0,1,191,50.3%,+0.0799%,+0.0299%,1.28,1.51,1.99
|
||||
|
@@ -0,0 +1,8 @@
|
||||
品种,噪声bp,种子,笔数,胜率,毛均收益,净均收益,PF,t值,滑点余量bp
|
||||
ETH,0.0,0,288,55.6%,+0.1231%,+0.0731%,1.9,4.15,6.31
|
||||
ETH,0.5,0,293,52.9%,+0.0949%,+0.0449%,1.46,2.52,3.49
|
||||
ETH,0.5,1,291,60.1%,+0.1236%,+0.0736%,1.87,4.2,6.36
|
||||
ETH,1.0,0,280,51.1%,+0.1004%,+0.0504%,1.48,2.65,4.04
|
||||
ETH,1.0,1,270,54.8%,+0.1142%,+0.0642%,1.64,3.24,5.42
|
||||
ETH,2.0,0,240,55.8%,+0.1182%,+0.0682%,1.61,3.14,5.82
|
||||
ETH,2.0,1,222,58.1%,+0.1474%,+0.0974%,1.93,4.07,8.74
|
||||
|
@@ -0,0 +1,8 @@
|
||||
品种,噪声bp,种子,笔数,胜率,毛均收益,净均收益,PF,t值,滑点余量bp
|
||||
SOL,0.0,0,245,52.2%,+0.1001%,+0.0501%,1.44,2.41,4.01
|
||||
SOL,0.5,0,232,53.4%,+0.1000%,+0.0500%,1.44,2.47,4.0
|
||||
SOL,0.5,1,234,57.7%,+0.1332%,+0.0832%,1.75,3.68,7.32
|
||||
SOL,1.0,0,246,57.7%,+0.1362%,+0.0862%,1.82,4.09,7.62
|
||||
SOL,1.0,1,230,54.8%,+0.1191%,+0.0691%,1.58,3.01,5.91
|
||||
SOL,2.0,0,195,58.5%,+0.1486%,+0.0986%,1.83,3.84,8.86
|
||||
SOL,2.0,1,193,55.4%,+0.1327%,+0.0827%,1.67,3.19,7.27
|
||||
|
@@ -0,0 +1,11 @@
|
||||
指标,值
|
||||
cpu核数,2.0
|
||||
1m窗口,2000.0
|
||||
5m窗口,800.0
|
||||
1m中位s,0.318
|
||||
1m_P95s,0.345
|
||||
5m中位s,0.136
|
||||
单币摊薄s,0.345
|
||||
串行3币s,1.38
|
||||
并行2_3币s,1.27
|
||||
并行3_3币s,1.28
|
||||
|
@@ -0,0 +1,7 @@
|
||||
品种,venue,笔数,胜率,毛均收益,净均收益,PF,t值,滑点余量bp,毛均bp
|
||||
BTC,Bitget,313,51.4%,+0.0587%,+0.0087%,1.11,0.71,-0.13,5.87
|
||||
BTC,Binance,296,49.3%,+0.0609%,+0.0109%,1.13,0.86,0.09,6.09
|
||||
ETH,Bitget,299,54.8%,+0.1002%,+0.0502%,1.56,3.07,4.02,10.02
|
||||
ETH,Binance,290,55.2%,+0.1206%,+0.0706%,1.85,4.01,6.06,12.06
|
||||
SOL,Bitget,329,52.3%,+0.0892%,+0.0392%,1.38,2.38,2.92,8.92
|
||||
SOL,Binance,244,52.0%,+0.0987%,+0.0487%,1.42,2.34,3.87,9.87
|
||||
|
@@ -0,0 +1,3 @@
|
||||
品种,venue,笔数,胜率,毛均收益,净均收益,PF,t值,滑点余量bp,毛均bp
|
||||
BTC,Bitget,313,51.4%,+0.0587%,+0.0087%,1.11,0.71,-0.13,5.87
|
||||
BTC,Binance,296,49.3%,+0.0609%,+0.0109%,1.13,0.86,0.09,6.09
|
||||
|
@@ -0,0 +1,3 @@
|
||||
品种,venue,笔数,胜率,毛均收益,净均收益,PF,t值,滑点余量bp,毛均bp
|
||||
ETH,Bitget,299,54.8%,+0.1002%,+0.0502%,1.56,3.07,4.02,10.02
|
||||
ETH,Binance,290,55.2%,+0.1206%,+0.0706%,1.85,4.01,6.06,12.06
|
||||
|
@@ -0,0 +1,3 @@
|
||||
品种,venue,笔数,胜率,毛均收益,净均收益,PF,t值,滑点余量bp,毛均bp
|
||||
SOL,Bitget,329,52.3%,+0.0892%,+0.0392%,1.38,2.38,2.92,8.92
|
||||
SOL,Binance,244,52.0%,+0.0987%,+0.0487%,1.42,2.34,3.87,9.87
|
||||
|
@@ -0,0 +1,55 @@
|
||||
kline_ts,sym,t_data_ms,lag_ms
|
||||
1787840400000,SOL,1787840401075,1075
|
||||
1787840400000,BTC,1787840401577,1577
|
||||
1787840400000,ETH,1787840401863,1863
|
||||
1787840460000,ETH,1787840460075,75
|
||||
1787840460000,BTC,1787840460549,549
|
||||
1787840460000,SOL,1787840460822,822
|
||||
1787840520000,SOL,1787840520463,463
|
||||
1787840520000,ETH,1787840520756,756
|
||||
1787840520000,BTC,1787840521026,1026
|
||||
1787840580000,ETH,1787840580384,384
|
||||
1787840580000,BTC,1787840580687,687
|
||||
1787840580000,SOL,1787840581292,1292
|
||||
1787840640000,BTC,1787840640204,204
|
||||
1787840640000,SOL,1787840640525,525
|
||||
1787840640000,ETH,1787840641155,1155
|
||||
1787840700000,BTC,1787840700192,192
|
||||
1787840700000,ETH,1787840701170,1170
|
||||
1787840700000,SOL,1787840701185,1185
|
||||
1787840760000,BTC,1787840760795,795
|
||||
1787840760000,ETH,1787840761231,1231
|
||||
1787840760000,SOL,1787840761346,1346
|
||||
1787840820000,BTC,1787840820308,308
|
||||
1787840820000,ETH,1787840820327,327
|
||||
1787840820000,SOL,1787840821241,1241
|
||||
1787840880000,BTC,1787840880441,441
|
||||
1787840880000,ETH,1787840880933,933
|
||||
1787840880000,SOL,1787840880936,936
|
||||
1787840940000,SOL,1787840940330,330
|
||||
1787840940000,ETH,1787840940551,551
|
||||
1787840940000,BTC,1787840940920,920
|
||||
1787841000000,SOL,1787841001488,1488
|
||||
1787841000000,BTC,1787841001683,1683
|
||||
1787841000000,ETH,1787841001862,1862
|
||||
1787841060000,BTC,1787841060406,406
|
||||
1787841060000,SOL,1787841060621,621
|
||||
1787841060000,ETH,1787841061151,1151
|
||||
1787841120000,ETH,1787841120329,329
|
||||
1787841120000,BTC,1787841120436,436
|
||||
1787841120000,SOL,1787841120928,928
|
||||
1787841180000,SOL,1787841180360,360
|
||||
1787841180000,BTC,1787841180591,591
|
||||
1787841180000,ETH,1787841180910,910
|
||||
1787841240000,ETH,1787841240385,385
|
||||
1787841240000,BTC,1787841240416,416
|
||||
1787841240000,SOL,1787841240627,627
|
||||
1787841300000,ETH,1787841300833,833
|
||||
1787841300000,SOL,1787841301169,1169
|
||||
1787841300000,BTC,1787841301299,1299
|
||||
1787841360000,ETH,1787841360166,166
|
||||
1787841360000,BTC,1787841360565,565
|
||||
1787841360000,SOL,1787841360991,991
|
||||
1787841420000,SOL,1787841420375,375
|
||||
1787841420000,ETH,1787841422267,2267
|
||||
1787841420000,BTC,1787841422314,2314
|
||||
|
@@ -0,0 +1,91 @@
|
||||
kline_ts,sym,t_data_ms,lag_ms
|
||||
1787836320000,SOL,1787836320241,241
|
||||
1787836320000,ETH,1787836320302,302
|
||||
1787836320000,BTC,1787836321824,1824
|
||||
1787836380000,SOL,1787836381417,1417
|
||||
1787836380000,ETH,1787836382286,2286
|
||||
1787836380000,BTC,1787836382710,2710
|
||||
1787836440000,SOL,1787836440534,534
|
||||
1787836440000,BTC,1787836441250,1250
|
||||
1787836440000,ETH,1787836441361,1361
|
||||
1787836500000,ETH,1787836500123,123
|
||||
1787836500000,SOL,1787836500134,134
|
||||
1787836500000,BTC,1787836502484,2484
|
||||
1787836560000,SOL,1787836560660,660
|
||||
1787836560000,BTC,1787836561486,1486
|
||||
1787836560000,ETH,1787836561748,1748
|
||||
1787836620000,SOL,1787836620499,499
|
||||
1787836620000,BTC,1787836621187,1187
|
||||
1787836620000,ETH,1787836621913,1913
|
||||
1787836680000,ETH,1787836680478,478
|
||||
1787836680000,BTC,1787836681568,1568
|
||||
1787836680000,SOL,1787836682143,2143
|
||||
1787836740000,BTC,1787836740072,72
|
||||
1787836740000,SOL,1787836740214,214
|
||||
1787836740000,ETH,1787836741453,1453
|
||||
1787836800000,SOL,1787836801678,1678
|
||||
1787836800000,BTC,1787836803383,3383
|
||||
1787836800000,ETH,1787836803485,3485
|
||||
1787836860000,BTC,1787836861120,1120
|
||||
1787836860000,ETH,1787836861523,1523
|
||||
1787836860000,SOL,1787836861634,1634
|
||||
1787836920000,BTC,1787836920212,212
|
||||
1787836920000,ETH,1787836920969,969
|
||||
1787836920000,SOL,1787836921292,1292
|
||||
1787836980000,SOL,1787836980715,715
|
||||
1787836980000,ETH,1787836981108,1108
|
||||
1787836980000,BTC,1787836982025,2025
|
||||
1787837040000,ETH,1787837041234,1234
|
||||
1787837040000,BTC,1787837041305,1305
|
||||
1787837040000,SOL,1787837042293,2293
|
||||
1787837100000,BTC,1787837100436,436
|
||||
1787837100000,SOL,1787837100436,436
|
||||
1787837100000,ETH,1787837101273,1273
|
||||
1787837160000,ETH,1787837161167,1167
|
||||
1787837160000,BTC,1787837161228,1228
|
||||
1787837160000,SOL,1787837161420,1420
|
||||
1787837220000,SOL,1787837221236,1236
|
||||
1787837220000,BTC,1787837221620,1620
|
||||
1787837220000,ETH,1787837222286,2286
|
||||
1787837280000,SOL,1787837280246,246
|
||||
1787837280000,BTC,1787837280750,750
|
||||
1787837280000,ETH,1787837281205,1205
|
||||
1787837340000,ETH,1787837340155,155
|
||||
1787837340000,BTC,1787837341234,1234
|
||||
1787837340000,SOL,1787837342232,2232
|
||||
1787837400000,SOL,1787837402232,2232
|
||||
1787837400000,ETH,1787837402374,2374
|
||||
1787837400000,BTC,1787837403010,3010
|
||||
1787837460000,SOL,1787837461262,1262
|
||||
1787837460000,BTC,1787837461403,1403
|
||||
1787837460000,ETH,1787837461988,1988
|
||||
1787837520000,ETH,1787837521167,1167
|
||||
1787837520000,BTC,1787837521298,1298
|
||||
1787837520000,SOL,1787837521298,1298
|
||||
1787837580000,SOL,1787837581311,1311
|
||||
1787837580000,BTC,1787837581634,1634
|
||||
1787837580000,ETH,1787837581725,1725
|
||||
1787837640000,SOL,1787837641179,1179
|
||||
1787837640000,ETH,1787837642047,2047
|
||||
1787837640000,BTC,1787837642057,2057
|
||||
1787837700000,ETH,1787837700834,834
|
||||
1787837700000,SOL,1787837701499,1499
|
||||
1787837700000,BTC,1787837701963,1963
|
||||
1787837760000,ETH,1787837761627,1627
|
||||
1787837760000,SOL,1787837761828,1828
|
||||
1787837760000,BTC,1787837761951,1951
|
||||
1787837820000,SOL,1787837821188,1188
|
||||
1787837820000,ETH,1787837821258,1258
|
||||
1787837820000,BTC,1787837821561,1561
|
||||
1787837880000,SOL,1787837881174,1174
|
||||
1787837880000,BTC,1787837881235,1235
|
||||
1787837880000,ETH,1787837881467,1467
|
||||
1787837940000,SOL,1787837941579,1579
|
||||
1787837940000,BTC,1787837941741,1741
|
||||
1787837940000,ETH,1787837941883,1883
|
||||
1787838000000,SOL,1787838002447,2447
|
||||
1787838000000,ETH,1787838002649,2649
|
||||
1787838000000,BTC,1787838003586,3586
|
||||
1787838060000,BTC,1787838060835,835
|
||||
1787838060000,ETH,1787838061561,1561
|
||||
1787838060000,SOL,1787838061774,1774
|
||||
|
@@ -0,0 +1,45 @@
|
||||
{
|
||||
"tag": "container",
|
||||
"python": "3.13.14",
|
||||
"pandas": "3.0.5",
|
||||
"numpy": "2.4.6",
|
||||
"per_symbol": {
|
||||
"BTC": {
|
||||
"n_bars": 2000,
|
||||
"n_signals": 3,
|
||||
"signal_idx": [
|
||||
685,
|
||||
1053,
|
||||
1483
|
||||
],
|
||||
"zone_checksum": 39331212411797.2,
|
||||
"n_zones": 11,
|
||||
"compute_s": 0.2582
|
||||
},
|
||||
"ETH": {
|
||||
"n_bars": 2000,
|
||||
"n_signals": 4,
|
||||
"signal_idx": [
|
||||
469,
|
||||
1063,
|
||||
1630,
|
||||
1731
|
||||
],
|
||||
"zone_checksum": 35755508019041.05,
|
||||
"n_zones": 10,
|
||||
"compute_s": 0.2646
|
||||
},
|
||||
"SOL": {
|
||||
"n_bars": 2000,
|
||||
"n_signals": 3,
|
||||
"signal_idx": [
|
||||
461,
|
||||
682,
|
||||
1087
|
||||
],
|
||||
"zone_checksum": 35755365663932.44,
|
||||
"n_zones": 10,
|
||||
"compute_s": 0.2626
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
{
|
||||
"tag": "venv",
|
||||
"python": "3.14.4",
|
||||
"pandas": "3.0.5",
|
||||
"numpy": "2.5.2",
|
||||
"per_symbol": {
|
||||
"BTC": {
|
||||
"n_bars": 2000,
|
||||
"n_signals": 3,
|
||||
"signal_idx": [
|
||||
685,
|
||||
1053,
|
||||
1483
|
||||
],
|
||||
"zone_checksum": 39331212411797.2,
|
||||
"n_zones": 11,
|
||||
"compute_s": 0.2948
|
||||
},
|
||||
"ETH": {
|
||||
"n_bars": 2000,
|
||||
"n_signals": 4,
|
||||
"signal_idx": [
|
||||
469,
|
||||
1063,
|
||||
1630,
|
||||
1731
|
||||
],
|
||||
"zone_checksum": 35755508019041.05,
|
||||
"n_zones": 10,
|
||||
"compute_s": 0.2861
|
||||
},
|
||||
"SOL": {
|
||||
"n_bars": 2000,
|
||||
"n_signals": 3,
|
||||
"signal_idx": [
|
||||
461,
|
||||
682,
|
||||
1087
|
||||
],
|
||||
"zone_checksum": 35755365663932.44,
|
||||
"n_zones": 10,
|
||||
"compute_s": 0.2902
|
||||
}
|
||||
}
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user