影子测量改用框架吃单原语,并补齐容量与 maker 成交率两项测算
吃单查询换成 Hummingbot 的 OrderBook.get_vwap_for_volume:手写的 walk_book 返回的是按计价币吃单的加权均价,但框架的 get_price_for_quote_volume 返回 边际价、get_vwap_for_volume 收基础币量,两者语义不同。改为按基础币下单 (真实委托与 PositionExecutor.amount 均是基础币计价),深度不足由 query_volume/result_volume 判定,框架此时返回 nan 而非一个看似正常的 部分成交均价。 落盘完整盘口(双边 50 档)。此前只记三个固定名义额的成交价,这批数据的 寿命就等于那几个档位的寿命;存完整深度后任意资金量级的冲击都能离线重算。 仓位档同时从 1k/5k/20k 提到十万量级,此前低估真实仓位约两个数量级。 订阅成交流,按根按价位聚合。买卖分开存——多头在目标位挂卖出靠主动买盘 成交,混在一起会把成交率高估约一倍。BTC 每根总成交额中位与 210 天历史 的 volume×close 差 0.3%,可确认采集完整。 新增两项测算: - 冲击不是绑定约束。32 万仓位单边冲击 0.19~2.39bp,对 8.58~20.64bp 的 预算只占 1.6~14.2%,冲击反推的资金上限 100~500 万。 - maker 成交率才是。止盈位被首次触及时,限价在该根价格区间中的位置 中位 k=0.28(63.9 万次触及,三币一致);合并每根成交额后,32 万仓位 的全额成交率仅 30.1%/15.6%/1.5%。要 80% 全额成交,仓位须 ≤ 4.7 万 /1.4 万/0.26 万——比冲击反推的上限低 40~370 倍。 回测把这些止盈按「全额成交在目标价」计,故预算所依据的收益流本身需重估。 Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -42,3 +42,6 @@ venv/
|
|||||||
|
|
||||||
# Local tooling
|
# Local tooling
|
||||||
.gstack/
|
.gstack/
|
||||||
|
research/out/*.jsonl.gz
|
||||||
|
research/out/penetration.csv
|
||||||
|
research/out/shadow_*.csv
|
||||||
|
|||||||
@@ -5,9 +5,11 @@
|
|||||||
的标准差。入场方向上还有系统性追价(信号触发往往伴随同向动量),所以随机
|
的标准差。入场方向上还有系统性追价(信号触发往往伴随同向动量),所以随机
|
||||||
漂移只是下限,真实成本更高——这也是为什么最终仍要用真实盘口测滑点。
|
漂移只是下限,真实成本更高——这也是为什么最终仍要用真实盘口测滑点。
|
||||||
|
|
||||||
余量(bitget_baseline.py 得出,Bitget 原生基线减去手续费后剩下的空间):
|
预算从 `lib/shadow_budget` import,不在这里写死。曾经写死的
|
||||||
BTC -0.13bp ETH +4.02bp SOL +2.92bp
|
`{BTC: -0.13, ETH: 4.02, SOL: 2.92}` 是错的——那是 `bitget_baseline.py`
|
||||||
BTC 本就为负,留着只作延迟测量的参照物,不作交易标的。
|
按「毛均 − 6bp 双边 taker」算的,六处口径叠加(费率档位记高、余量没除
|
||||||
|
taker 名义额、用了 5m~30m 的出场参数、只有同向没有阶梯与 ATR 门控)。
|
||||||
|
正确值是 8.58 / 20.64 / 16.83,ETH 差了五倍。
|
||||||
|
|
||||||
.venv/bin/python research/live/latency_compare.py
|
.venv/bin/python research/live/latency_compare.py
|
||||||
"""
|
"""
|
||||||
@@ -23,8 +25,9 @@ HERE = Path(__file__).resolve().parent
|
|||||||
RESEARCH = HERE.parent
|
RESEARCH = HERE.parent
|
||||||
sys.path.insert(0, str(RESEARCH))
|
sys.path.insert(0, str(RESEARCH))
|
||||||
|
|
||||||
|
from lib.shadow_budget import budget_of # noqa: E402
|
||||||
|
|
||||||
OUT = RESEARCH / "out"
|
OUT = RESEARCH / "out"
|
||||||
BUDGET_BP = {"BTC": -0.13, "ETH": 4.02, "SOL": 2.92}
|
|
||||||
SYMS = ("BTC", "ETH", "SOL")
|
SYMS = ("BTC", "ETH", "SOL")
|
||||||
|
|
||||||
|
|
||||||
@@ -67,8 +70,8 @@ def describe(df: pd.DataFrame, name: str, vols: dict) -> None:
|
|||||||
med, p90 = float(np.median(v)), float(np.percentile(v, 90))
|
med, p90 = float(np.median(v)), float(np.percentile(v, 90))
|
||||||
vol = vols.get(s)
|
vol = vols.get(s)
|
||||||
d = drift_bp(med, vol) if vol else float("nan")
|
d = drift_bp(med, vol) if vol else float("nan")
|
||||||
b = BUDGET_BP[s]
|
b = budget_of(s)
|
||||||
share = f"{d / b * 100:.0f}%" if b > 0 else "—(负)"
|
share = f"{d / b * 100:.0f}%" if np.isfinite(b) and b > 0 else "—"
|
||||||
print(f"{s:<5}{len(v):>5}{med:>9.0f}{p90:>9.0f}{v.max():>9.0f}"
|
print(f"{s:<5}{len(v):>5}{med:>9.0f}{p90:>9.0f}{v.max():>9.0f}"
|
||||||
f"{d:>12.2f}{b:>9.2f}{share:>9}")
|
f"{d:>12.2f}{b:>9.2f}{share:>9}")
|
||||||
|
|
||||||
|
|||||||
@@ -14,9 +14,21 @@
|
|||||||
再把最后一根交给基类走正常的 append 流程。
|
再把最后一根交给基类走正常的 append 流程。
|
||||||
|
|
||||||
已向上游反馈前,本地用子类覆盖,不改动镜像。
|
已向上游反馈前,本地用子类覆盖,不改动镜像。
|
||||||
|
|
||||||
|
## 为什么必须有启动断言
|
||||||
|
|
||||||
|
子类覆盖的失效方式是**静默**的:上游若把 `_parse_websocket_message` 改名、
|
||||||
|
或改走别的钩子,我们的覆盖就成了死代码,行情悄悄退回慢 1.06 秒,不崩、
|
||||||
|
不报错、不留日志,只会让收益慢慢变差,几周后才从统计里看出来。
|
||||||
|
|
||||||
|
`assert_patch_effective()` 不做名字检查——名字对不上未必失效,名字对得上
|
||||||
|
也未必生效。它喂一条合成的两元素消息,直接验证行为:基类返回首元素(bug
|
||||||
|
仍在、覆盖仍有必要),子类返回末元素(覆盖确实生效)。再加一条源码检查
|
||||||
|
确认基类的收包循环还在调这个钩子。任一不满足就在启动时抛错。
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import inspect
|
||||||
from typing import Any, Dict, Optional
|
from typing import Any, Dict, Optional
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
@@ -24,6 +36,7 @@ import numpy as np
|
|||||||
from hummingbot.data_feed.candles_feed.bitget_perpetual_candles import (
|
from hummingbot.data_feed.candles_feed.bitget_perpetual_candles import (
|
||||||
BitgetPerpetualCandles,
|
BitgetPerpetualCandles,
|
||||||
)
|
)
|
||||||
|
from hummingbot.data_feed.candles_feed.candles_base import CandlesBase
|
||||||
|
|
||||||
|
|
||||||
def _row_to_dict(row: list, ensure_s) -> Dict[str, Any]:
|
def _row_to_dict(row: list, ensure_s) -> Dict[str, Any]:
|
||||||
@@ -63,3 +76,51 @@ class PatchedBitgetPerpetualCandles(BitgetPerpetualCandles):
|
|||||||
d["volume"], d["quote_asset_volume"], d["n_trades"],
|
d["volume"], d["quote_asset_volume"], d["n_trades"],
|
||||||
d["taker_buy_base_volume"], d["taker_buy_quote_volume"]]
|
d["taker_buy_base_volume"], d["taker_buy_quote_volume"]]
|
||||||
).astype(float)
|
).astype(float)
|
||||||
|
|
||||||
|
|
||||||
|
# 换根时 Bitget 推的就是这个形状:[上一根, 新一根]
|
||||||
|
_PROBE = {
|
||||||
|
"action": "update",
|
||||||
|
"arg": {"instType": "USDT-FUTURES", "channel": "candle1m",
|
||||||
|
"instId": "BTCUSDT"},
|
||||||
|
"data": [
|
||||||
|
["1700000040000", "1", "1", "1", "1", "1", "1", "1"],
|
||||||
|
["1700000100000", "2", "2", "2", "2", "2", "2", "2"],
|
||||||
|
],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def assert_patch_effective() -> None:
|
||||||
|
"""启动即验证覆盖真的生效,否则抛错。让静默失效变成启动失败。"""
|
||||||
|
src = inspect.getsource(CandlesBase._process_websocket_messages_task)
|
||||||
|
if "_parse_websocket_message" not in src:
|
||||||
|
raise RuntimeError(
|
||||||
|
"上游收包循环已不再调用 _parse_websocket_message,"
|
||||||
|
"patched_candles 的覆盖失效。需重新定位钩子后再启动。")
|
||||||
|
|
||||||
|
stock = BitgetPerpetualCandles("BTC-USDT", "1m", 20)
|
||||||
|
ours = PatchedBitgetPerpetualCandles("BTC-USDT", "1m", 20)
|
||||||
|
got_stock = stock._parse_websocket_message(_PROBE)
|
||||||
|
got_ours = ours._parse_websocket_message(_PROBE)
|
||||||
|
|
||||||
|
head_ts = stock.ensure_timestamp_in_seconds(int(_PROBE["data"][0][0]))
|
||||||
|
tail_ts = stock.ensure_timestamp_in_seconds(int(_PROBE["data"][-1][0]))
|
||||||
|
|
||||||
|
if not got_ours or int(got_ours["timestamp"]) != int(tail_ts):
|
||||||
|
raise RuntimeError(
|
||||||
|
f"覆盖未生效:子类返回 {got_ours and got_ours.get('timestamp')},"
|
||||||
|
f"应为末元素 {tail_ts}。")
|
||||||
|
if got_stock and int(got_stock["timestamp"]) == int(tail_ts):
|
||||||
|
# 上游自己修好了。此时覆盖无害但已多余,明确说出来,免得以后
|
||||||
|
# 有人以为那 1.06 秒还是靠我们拿回来的
|
||||||
|
print(" [补丁] 上游已自行修正换根解析,本地覆盖现为冗余,可移除",
|
||||||
|
flush=True)
|
||||||
|
elif not got_stock or int(got_stock["timestamp"]) != int(head_ts):
|
||||||
|
raise RuntimeError(
|
||||||
|
f"基类行为与预期不符:返回 "
|
||||||
|
f"{got_stock and got_stock.get('timestamp')},"
|
||||||
|
f"既非首元素 {head_ts} 也非末元素 {tail_ts}。"
|
||||||
|
f"上游改了解析逻辑,补丁的前提需重新确认。")
|
||||||
|
else:
|
||||||
|
print(f" [补丁] 覆盖生效:基类取首元素 {int(head_ts)}、"
|
||||||
|
f"本地取末元素 {int(tail_ts)}", flush=True)
|
||||||
|
|||||||
@@ -0,0 +1,152 @@
|
|||||||
|
"""止盈位被首次触及那一根,价格穿透了多深。
|
||||||
|
|
||||||
|
为什么要这个数:影子成交流显示,限价单若正好落在某根的最高价,该价位之上
|
||||||
|
的主动买成交额只有几十到几千美元——对十万量级的仓位等于不成交。但那是
|
||||||
|
最坏情形。真实成交率取决于**止盈位被穿透了多深**:若价格一路冲过目标,
|
||||||
|
成交没问题;若只是上影线点一下就回落,就成交不了。
|
||||||
|
|
||||||
|
这个分布不需要再采数据,历史 K 线里就有:给定入场价与 ATR,目标位是
|
||||||
|
`entry + T×ATR`,找到首次 `high ≥ target` 的那根,穿透深度就是
|
||||||
|
`high − target`。把它折成「占该根价格区间的比例」,就能直接对上成交流
|
||||||
|
那条「≥ 限价的成交额 vs 限价在区间中的位置」曲线。
|
||||||
|
|
||||||
|
口径与 lib/exit_model.walk_exits 对齐:入场取信号次根开盘价,ATR 取信号
|
||||||
|
根的 Wilder ATR-14,上限 48 根。
|
||||||
|
|
||||||
|
**取样方式的局限**:这里用全体 K 线做候选入场点,而非真实的三滤网信号。
|
||||||
|
真实信号是按结构条件挑出来的,入场时刻可能与波动率状态相关。以 ATR 归一
|
||||||
|
后的穿透深度对波动率状态应当不敏感,但要精确到信号级别,得重跑一次
|
||||||
|
step42 的缠论链路(单币约 370s、峰值 24.5GB)。
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
SCALE_AT = 3.0 # 分批减仓位
|
||||||
|
RUNNER = 8.0 # 剩余半仓目标
|
||||||
|
MAX_BARS = 48
|
||||||
|
|
||||||
|
|
||||||
|
def wilder_atr(high, low, close, period: int = 14) -> np.ndarray:
|
||||||
|
"""与 chanlun.indicators.ta.ATR 逐位一致的 Wilder ATR。"""
|
||||||
|
n = high.size
|
||||||
|
out = np.full(n, np.nan)
|
||||||
|
if n <= period:
|
||||||
|
return out
|
||||||
|
prev_close = close[:-1]
|
||||||
|
tr = np.maximum.reduce([high[1:] - low[1:],
|
||||||
|
np.abs(high[1:] - prev_close),
|
||||||
|
np.abs(low[1:] - prev_close)])
|
||||||
|
sm = np.empty(tr.size)
|
||||||
|
sm[period - 1] = tr[:period].mean()
|
||||||
|
a = 1.0 / period
|
||||||
|
for k in range(period, tr.size):
|
||||||
|
sm[k] = sm[k - 1] + a * (tr[k] - sm[k - 1])
|
||||||
|
out[period:] = sm[period - 1:]
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def penetration(df: pd.DataFrame, target_atr: float,
|
||||||
|
stride: int = 1) -> pd.DataFrame:
|
||||||
|
"""对每个候选入场点,求首次触及 `target_atr` 时的穿透深度。
|
||||||
|
|
||||||
|
只统计**触及了**的那些(未触及的属止损或超时出场,不涉及 maker 腿)。
|
||||||
|
"""
|
||||||
|
high = df["high"].to_numpy(float)
|
||||||
|
low = df["low"].to_numpy(float)
|
||||||
|
close = df["close"].to_numpy(float)
|
||||||
|
open_ = df["open"].to_numpy(float)
|
||||||
|
atr = wilder_atr(high, low, close)
|
||||||
|
n = len(df)
|
||||||
|
rows = []
|
||||||
|
for i in range(20, n - MAX_BARS - 2, stride):
|
||||||
|
a = atr[i]
|
||||||
|
if not np.isfinite(a) or a <= 0:
|
||||||
|
continue
|
||||||
|
e = i + 1
|
||||||
|
entry = open_[e]
|
||||||
|
# 多头:目标在上方。空头对称,穿透深度分布按对称性等价,故只算一边
|
||||||
|
target = entry + target_atr * a
|
||||||
|
end = e + MAX_BARS
|
||||||
|
seg_hi = high[e:end + 1]
|
||||||
|
hit = np.flatnonzero(seg_hi >= target)
|
||||||
|
if not hit.size:
|
||||||
|
continue
|
||||||
|
j = e + int(hit[0])
|
||||||
|
rng = high[j] - low[j]
|
||||||
|
if rng <= 0:
|
||||||
|
continue
|
||||||
|
pen = high[j] - target
|
||||||
|
rows.append({
|
||||||
|
"bar": j,
|
||||||
|
# 限价在该根价格区间中的位置:0 = 正好在最高价(最坏),
|
||||||
|
# 1 = 在最低价(该根全部成交都在限价之上)
|
||||||
|
"k": min(1.0, pen / rng),
|
||||||
|
"pen_bp": pen / target * 1e4,
|
||||||
|
"pen_atr": pen / a,
|
||||||
|
"range_bp": rng / target * 1e4,
|
||||||
|
})
|
||||||
|
return pd.DataFrame(rows)
|
||||||
|
|
||||||
|
|
||||||
|
def report(sym: str, df: pd.DataFrame) -> dict:
|
||||||
|
print(f"\n{'=' * 68}\n{sym} 共 {len(df):,} 根 1m")
|
||||||
|
out = {}
|
||||||
|
for tgt, name in ((SCALE_AT, f"减仓位 {SCALE_AT:g}ATR"),
|
||||||
|
(RUNNER, f"目标位 {RUNNER:g}ATR")):
|
||||||
|
p = penetration(df, tgt)
|
||||||
|
if p.empty:
|
||||||
|
print(f" {name}: 无触及样本")
|
||||||
|
continue
|
||||||
|
k = p["k"].to_numpy()
|
||||||
|
print(f"\n {name} · 触及 {len(p):,} 次")
|
||||||
|
print(f" 穿透深度 中位 {p['pen_bp'].median():.2f}bp "
|
||||||
|
f"({p['pen_atr'].median():.2f} ATR) · "
|
||||||
|
f"P25 {p['pen_bp'].quantile(.25):.2f}bp · "
|
||||||
|
f"P75 {p['pen_bp'].quantile(.75):.2f}bp")
|
||||||
|
print(f" 限价在区间中的位置 k(0=正好在最高价,越大越靠下越易成交)")
|
||||||
|
print(f" 中位 {np.median(k):.3f} · P10 {np.percentile(k, 10):.3f}"
|
||||||
|
f" · P25 {np.percentile(k, 25):.3f}"
|
||||||
|
f" · P75 {np.percentile(k, 75):.3f}")
|
||||||
|
for thr in (0.05, 0.10, 0.25, 0.50):
|
||||||
|
print(f" k ≤ {thr:.2f}(限价挤在该根顶部 {thr * 100:.0f}% 内):"
|
||||||
|
f"{float((k <= thr).mean()) * 100:5.1f}% 的触及")
|
||||||
|
out[tgt] = p
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--syms", default="BTC,ETH,SOL")
|
||||||
|
ap.add_argument("--cache", default="research/live/cache")
|
||||||
|
ap.add_argument("--save", default="research/out/penetration.csv")
|
||||||
|
a = ap.parse_args()
|
||||||
|
|
||||||
|
root = Path(a.cache)
|
||||||
|
allp = []
|
||||||
|
for sym in a.syms.split(","):
|
||||||
|
cands = sorted(root.glob(f"bitget_{sym}_1m_*.feather"),
|
||||||
|
key=lambda p: p.stat().st_size, reverse=True)
|
||||||
|
if not cands:
|
||||||
|
print(f"{sym}: 找不到 1m 缓存,跳过")
|
||||||
|
continue
|
||||||
|
df = pd.read_feather(cands[0])
|
||||||
|
got = report(sym, df)
|
||||||
|
for tgt, p in got.items():
|
||||||
|
p = p.copy()
|
||||||
|
p["sym"], p["target_atr"] = sym, tgt
|
||||||
|
allp.append(p)
|
||||||
|
if allp:
|
||||||
|
out = pd.concat(allp, ignore_index=True)
|
||||||
|
Path(a.save).parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
out.to_csv(a.save, index=False)
|
||||||
|
print(f"\n已存 {a.save}({len(out):,} 行),"
|
||||||
|
f"供 shadow_depth.py 合并成交量曲线")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
"""真实 Bitget 盘口在 BOOK_DEPTH 档内能不能吃下各个名义额档位。
|
||||||
|
|
||||||
|
shadow_hb 把吃单换成了框架的 get_vwap_for_volume,深度不足时它返回 nan、
|
||||||
|
整行标 depth_ok=0。所以「档数够不够」直接决定某个仓位档会不会整段丢失,
|
||||||
|
不是个可以事后补救的参数——先量出来再定 BOOK_DEPTH。
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
sys.path.insert(0, "/repo/research/live")
|
||||||
|
|
||||||
|
from shadow_hb import BOOK_DEPTH, NOTIONALS, SYMS, book_from
|
||||||
|
|
||||||
|
|
||||||
|
async def run() -> None:
|
||||||
|
from hummingbot.connector.derivative.bitget_perpetual.bitget_perpetual_derivative import (
|
||||||
|
BitgetPerpetualDerivative,
|
||||||
|
)
|
||||||
|
conn = BitgetPerpetualDerivative(
|
||||||
|
bitget_perpetual_api_key="", bitget_perpetual_secret_key="",
|
||||||
|
bitget_perpetual_passphrase="",
|
||||||
|
trading_pairs=[f"{s}-USDT" for s in SYMS], trading_required=False)
|
||||||
|
await conn.start_network()
|
||||||
|
print(f"连接器已启动,等盘口(档数上限 {BOOK_DEPTH})")
|
||||||
|
for _ in range(60):
|
||||||
|
await asyncio.sleep(1)
|
||||||
|
try:
|
||||||
|
if all(conn.get_order_book(f"{s}-USDT") is not None for s in SYMS):
|
||||||
|
break
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
|
||||||
|
for s in SYMS:
|
||||||
|
ob = conn.get_order_book(f"{s}-USDT")
|
||||||
|
bids = np.array([(float(r.price), float(r.amount), i)
|
||||||
|
for i, (r, _) in enumerate(
|
||||||
|
zip(ob.bid_entries(), range(BOOK_DEPTH)))])
|
||||||
|
asks = np.array([(float(r.price), float(r.amount), i)
|
||||||
|
for i, (r, _) in enumerate(
|
||||||
|
zip(ob.ask_entries(), range(BOOK_DEPTH)))])
|
||||||
|
mid = (bids[0][0] + asks[0][0]) / 2.0
|
||||||
|
snap = book_from(bids, asks)
|
||||||
|
ask_notional = float((asks[:, 0] * asks[:, 1]).sum())
|
||||||
|
print(f"\n{s} 中价 {mid:.2f} · 取到 {len(asks)} 档 · "
|
||||||
|
f"卖盘 {len(asks)} 档合计 {ask_notional:,.0f} USDT")
|
||||||
|
print(f" 最深一档距中价 "
|
||||||
|
f"{(asks[-1][0] / mid - 1) * 1e4:.1f}bp")
|
||||||
|
for notional in NOTIONALS:
|
||||||
|
base = notional / mid
|
||||||
|
r = snap.get_vwap_for_volume(True, base)
|
||||||
|
px = float(r.result_price)
|
||||||
|
ok = float(r.result_volume) >= base * 0.999
|
||||||
|
if ok:
|
||||||
|
print(f" 名义 {notional:>7,.0f} → {base:.6f} 币 · "
|
||||||
|
f"冲击 {(px / mid - 1) * 1e4:6.2f}bp · 吃得下")
|
||||||
|
else:
|
||||||
|
print(f" 名义 {notional:>7,.0f} → {base:.6f} 币 · "
|
||||||
|
f"深度不足,仅 {r.result_volume:.6f} 币 · "
|
||||||
|
f"这一档会整段丢失")
|
||||||
|
await conn.stop_network()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
asyncio.run(run())
|
||||||
@@ -0,0 +1,287 @@
|
|||||||
|
"""从落盘的完整盘口与成交流,算资金容量与 maker 腿成交率。
|
||||||
|
|
||||||
|
这两个数都不该等实盘暴露:
|
||||||
|
|
||||||
|
**容量**。预算 20bp 意味着存在一个资金上限,超过它策略就不工作。既然完整
|
||||||
|
深度已落盘,任意仓位的冲击都能重算——一次采集回答所有资金量级,换个规模
|
||||||
|
不必重测一周。绑定约束是**薄盘时段**而非中位盘口,所以按分位数报。
|
||||||
|
|
||||||
|
**maker 成交率**。回测假设 3ATR / 8ATR 的限价单全额成交。深度回答不了这个
|
||||||
|
问题:深度说的是「现在挂着多少」,成交率问的是「之后打过来多少」。只有
|
||||||
|
成交流能回答,而且买卖必须分开——多头在 3ATR 挂卖出,靠主动买盘成交。
|
||||||
|
|
||||||
|
读 gzip 时必须容忍末尾成员不完整:采集进程还在写,最后一个 gzip 成员没有
|
||||||
|
结尾标记,直接遍历会在文件尾抛 EOFError 而丢掉**全部**已读记录。
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import gzip
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
|
||||||
|
def out_dir() -> Path:
|
||||||
|
p = Path("/out")
|
||||||
|
return p if p.is_dir() else Path(__file__).resolve().parents[1] / "out"
|
||||||
|
|
||||||
|
|
||||||
|
def read_jsonl_gz(path: Path):
|
||||||
|
"""逐行读 gzip JSONL,末尾截断则静默停止。
|
||||||
|
|
||||||
|
采集仍在进行时,最后一个 gzip 成员缺结尾标记;不接这个异常的话,
|
||||||
|
整个分析会因为文件尾而失败,前面几万条完好记录一起丢掉。
|
||||||
|
"""
|
||||||
|
if not path.exists():
|
||||||
|
return
|
||||||
|
n_ok = 0
|
||||||
|
try:
|
||||||
|
with gzip.open(path, "rt", encoding="utf-8") as fh:
|
||||||
|
for line in fh:
|
||||||
|
try:
|
||||||
|
rec = json.loads(line)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
break # 半行,说明写到这里被打断
|
||||||
|
n_ok += 1
|
||||||
|
yield rec
|
||||||
|
except (EOFError, OSError, gzip.BadGzipFile):
|
||||||
|
# 采集进程正在写,尾部不完整属正常
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def impact_bp(levels: list, notional: float, mid: float) -> float | None:
|
||||||
|
"""吃掉 notional 计价币后的加权均价相对中间价,bp。深度不足返回 None。"""
|
||||||
|
need = notional
|
||||||
|
cost = 0.0
|
||||||
|
qty = 0.0
|
||||||
|
for px, amt in levels:
|
||||||
|
avail = px * amt
|
||||||
|
take = min(avail, need)
|
||||||
|
q = take / px
|
||||||
|
cost += q * px
|
||||||
|
qty += q
|
||||||
|
need -= take
|
||||||
|
if need <= 1e-9:
|
||||||
|
break
|
||||||
|
if need > 1e-9 or qty <= 0:
|
||||||
|
return None
|
||||||
|
return (cost / qty / mid - 1.0) * 1e4
|
||||||
|
|
||||||
|
|
||||||
|
def capacity(books_path: Path, budgets: dict[str, float],
|
||||||
|
pctl: float = 10.0) -> None:
|
||||||
|
"""报各币的深度曲线与「冲击吃掉预算多少」的资金上限。"""
|
||||||
|
grid = [1e4, 5e4, 1e5, 2e5, 3.2e5, 5.3e5, 1e6, 2e6, 5e6]
|
||||||
|
per: dict[str, dict[float, list[float]]] = {}
|
||||||
|
n = 0
|
||||||
|
for r in read_jsonl_gz(books_path):
|
||||||
|
asks, bids = r["asks"], r["bids"]
|
||||||
|
if not asks or not bids:
|
||||||
|
continue
|
||||||
|
mid = (asks[0][0] + bids[0][0]) / 2.0
|
||||||
|
d = per.setdefault(r["sym"], {g: [] for g in grid})
|
||||||
|
for g in grid:
|
||||||
|
v = impact_bp(asks, g, mid)
|
||||||
|
d[g].append(np.nan if v is None else v)
|
||||||
|
n += 1
|
||||||
|
|
||||||
|
if not n:
|
||||||
|
print("没有盘口快照,先跑采集")
|
||||||
|
return
|
||||||
|
|
||||||
|
print(f"\n########## 资金容量 ##########")
|
||||||
|
print(f" 基于 {n:,} 份完整盘口快照(单边买入方向)\n")
|
||||||
|
for sym, d in per.items():
|
||||||
|
b = budgets.get(sym)
|
||||||
|
print(f" {sym} 预算 {b:.2f}bp" if b else f" {sym}")
|
||||||
|
print(f" {'名义额':>12} {'冲击中位':>10} {'冲击P90':>10} "
|
||||||
|
f"{'吃满深度率':>10} {'占预算':>8}")
|
||||||
|
for g in grid:
|
||||||
|
a = np.array(d[g], dtype=float)
|
||||||
|
fill = float(np.isfinite(a).mean())
|
||||||
|
if fill == 0:
|
||||||
|
print(f" {g:>12,.0f} {'—— 50 档吃不下 ——':>30}")
|
||||||
|
continue
|
||||||
|
med = float(np.nanmedian(a))
|
||||||
|
p90 = float(np.nanpercentile(a, 90))
|
||||||
|
share = f"{med / b * 100:6.1f}%" if b else " na"
|
||||||
|
print(f" {g:>12,.0f} {med:>10.2f} {p90:>10.2f} "
|
||||||
|
f"{fill * 100:>9.1f}% {share:>8}")
|
||||||
|
if b:
|
||||||
|
# 上限:冲击的 P90(薄盘时段)吃掉预算三成为止。三成是留给
|
||||||
|
# 漂移与价差的余地——它们才是主项,冲击不该独占预算
|
||||||
|
cap = None
|
||||||
|
for g in grid:
|
||||||
|
a = np.array(d[g], dtype=float)
|
||||||
|
if not np.isfinite(a).any():
|
||||||
|
break
|
||||||
|
if float(np.nanpercentile(a, 90)) > b * 0.30:
|
||||||
|
break
|
||||||
|
cap = g
|
||||||
|
if cap is None:
|
||||||
|
print(f" → 连最小档 {grid[0]:,.0f} 的薄盘冲击都超预算三成")
|
||||||
|
else:
|
||||||
|
print(f" → 资金上限约 {cap:,.0f} USDT"
|
||||||
|
f"(薄盘 P90 冲击 ≤ 预算 30%)")
|
||||||
|
print()
|
||||||
|
|
||||||
|
|
||||||
|
def maker_fill(tape_path: Path, mults=(3.0, 8.0),
|
||||||
|
notionals=(1e5, 3.2e5, 5.3e5)) -> None:
|
||||||
|
"""限价单挂在离场目标位,本根内有多少主动量打到那里。
|
||||||
|
|
||||||
|
这里只回答「量够不够」。真实成交还要看排队位置——我们的单排在该价位
|
||||||
|
已有挂单之后,所以这是**上界**:量不够则必然不能全成交,量够也未必成交。
|
||||||
|
"""
|
||||||
|
rows = list(read_jsonl_gz(tape_path))
|
||||||
|
if not rows:
|
||||||
|
print("没有成交流数据,先跑采集")
|
||||||
|
return
|
||||||
|
print(f"\n########## maker 腿成交量上界 ##########")
|
||||||
|
print(f" 基于 {len(rows):,} 根的逐价位成交聚合")
|
||||||
|
print(f" 多头在目标位挂卖出,成交靠主动**买**盘,故只计买方向\n")
|
||||||
|
|
||||||
|
# 限价单只能被**价格 ≥ 限价**的主动买成交打到。而止盈位被触及的那一根,
|
||||||
|
# 限价往往就落在该根价格区间的顶部——最高价刚好碰到目标位是最典型的
|
||||||
|
# 情形。所以按「限价距最高价多近」分层:depth=0 表示限价正好在最高价
|
||||||
|
# (只有打在最高价那一档的量算数),depth=0.25 表示限价在区间顶部 25% 处
|
||||||
|
depths = (0.0, 0.10, 0.25, 1.0)
|
||||||
|
per: dict[str, dict[float, list[float]]] = {}
|
||||||
|
for r in rows:
|
||||||
|
buys = {float(p): v for p, v in r["buys"].items()}
|
||||||
|
d = per.setdefault(r["sym"], {k: [] for k in depths})
|
||||||
|
if not buys:
|
||||||
|
for k in depths:
|
||||||
|
d[k].append(0.0)
|
||||||
|
continue
|
||||||
|
hi, lo = max(buys), min(buys)
|
||||||
|
rng = hi - lo
|
||||||
|
for k in depths:
|
||||||
|
floor_px = hi - k * rng
|
||||||
|
d[k].append(sum(p * v for p, v in buys.items() if p >= floor_px))
|
||||||
|
|
||||||
|
for sym, d in per.items():
|
||||||
|
print(f" {sym} ≥ 限价的主动买成交额(USDT),按限价所处位置分层")
|
||||||
|
print(f" {'限价位置':>16} {'中位':>12} {'P25':>12} "
|
||||||
|
+ " ".join(f"{n:>9,.0f}全仓" for n in notionals))
|
||||||
|
for k in depths:
|
||||||
|
a = np.array(d[k], dtype=float)
|
||||||
|
where = ("正好在最高价" if k == 0 else
|
||||||
|
"整根全部成交" if k == 1.0 else
|
||||||
|
f"区间顶部 {k * 100:.0f}%")
|
||||||
|
cells = " ".join(f"{float((a >= n).mean()) * 100:8.1f}%"
|
||||||
|
for n in notionals)
|
||||||
|
print(f" {where:>16} {np.median(a):>12,.0f} "
|
||||||
|
f"{np.percentile(a, 25):>12,.0f} {cells}")
|
||||||
|
print()
|
||||||
|
print(" 「正好在最高价」那一行才是止盈被刚好触及时的真实处境;")
|
||||||
|
print(" 「整根全部成交」是最宽松的上界。两行差多少,就是回测那个")
|
||||||
|
print(" 「限价单全额成交」假设虚了多少。而且这仍未计排队——我们的单")
|
||||||
|
print(" 排在该价位既有挂单之后,所以真实成交率比表里更低")
|
||||||
|
|
||||||
|
|
||||||
|
def tape_shape(tape_path: Path, kgrid: np.ndarray) -> dict[str, np.ndarray]:
|
||||||
|
"""成交流给「形状」:一根的主动买成交额里,有多少比例落在区间顶部 k 之内。
|
||||||
|
|
||||||
|
形状与规模分开是为了绕开成交流样本小的限制——形状是微观结构性质,
|
||||||
|
几十根就相当稳定;规模(每根成交多少钱)则由 210 天历史成交量提供。
|
||||||
|
"""
|
||||||
|
acc: dict[str, list[np.ndarray]] = {}
|
||||||
|
for r in read_jsonl_gz(tape_path):
|
||||||
|
buys = {float(p): v for p, v in r["buys"].items()}
|
||||||
|
if len(buys) < 2:
|
||||||
|
continue
|
||||||
|
hi, lo = max(buys), min(buys)
|
||||||
|
rng = hi - lo
|
||||||
|
if rng <= 0:
|
||||||
|
continue
|
||||||
|
tot = sum(p * v for p, v in buys.items())
|
||||||
|
if tot <= 0:
|
||||||
|
continue
|
||||||
|
frac = np.array([sum(p * v for p, v in buys.items()
|
||||||
|
if p >= hi - k * rng) / tot for k in kgrid])
|
||||||
|
acc.setdefault(r["sym"], []).append(frac)
|
||||||
|
return {s: np.mean(np.vstack(v), axis=0) for s, v in acc.items() if v}
|
||||||
|
|
||||||
|
|
||||||
|
def composite_fill(tape_path: Path, pen_path: Path, cache: Path,
|
||||||
|
notionals=(1e5, 3.2e5, 5.3e5)) -> None:
|
||||||
|
"""把穿透深度分布与成交量曲线合并,出真实 maker 成交率。"""
|
||||||
|
if not pen_path.exists():
|
||||||
|
print("\n没有 penetration.csv,先跑 penetration.py")
|
||||||
|
return
|
||||||
|
kgrid = np.linspace(0.0, 1.0, 51)
|
||||||
|
shape = tape_shape(tape_path, kgrid)
|
||||||
|
if not shape:
|
||||||
|
print("\n成交流样本不足,无法定形状")
|
||||||
|
return
|
||||||
|
pen = pd.read_csv(pen_path)
|
||||||
|
|
||||||
|
print(f"\n\n########## maker 腿真实成交率 ##########")
|
||||||
|
print(f" 穿透深度分布(历史 63 万次触及)× 每根成交额(210 天)")
|
||||||
|
print(f" × 区间内成交分布形状(影子成交流)\n")
|
||||||
|
|
||||||
|
for sym in sorted(shape):
|
||||||
|
cands = sorted(cache.glob(f"bitget_{sym}_1m_*.feather"),
|
||||||
|
key=lambda p: p.stat().st_size, reverse=True)
|
||||||
|
if not cands:
|
||||||
|
continue
|
||||||
|
bars = pd.read_feather(cands[0])
|
||||||
|
# 每根的主动买成交额。取总成交额的一半——买卖大致均衡,且这与
|
||||||
|
# 成交流实测的买卖比一致
|
||||||
|
bar_notional = (bars["volume"].to_numpy(float)
|
||||||
|
* bars["close"].to_numpy(float)) * 0.5
|
||||||
|
bar_notional = bar_notional[np.isfinite(bar_notional)
|
||||||
|
& (bar_notional > 0)]
|
||||||
|
f = shape[sym]
|
||||||
|
for tgt in sorted(pen["target_atr"].unique()):
|
||||||
|
k = pen[(pen["sym"] == sym)
|
||||||
|
& (pen["target_atr"] == tgt)]["k"].to_numpy(float)
|
||||||
|
if not k.size:
|
||||||
|
continue
|
||||||
|
# 独立配对:穿透位置与该根成交额各自抽样。真实触及根多为放量根,
|
||||||
|
# 故此处偏**保守**(低估可成交量)
|
||||||
|
rng = np.random.default_rng(0)
|
||||||
|
m = 200_000
|
||||||
|
ks = rng.choice(k, m)
|
||||||
|
ns = rng.choice(bar_notional, m)
|
||||||
|
avail = np.interp(ks, kgrid, f) * ns
|
||||||
|
print(f" {sym} · 目标 {tgt:g}ATR · 每根主动买额中位 "
|
||||||
|
f"{np.median(bar_notional):,.0f} USDT")
|
||||||
|
for nt in notionals:
|
||||||
|
full = float((avail >= nt).mean())
|
||||||
|
half = float((avail >= nt / 2).mean())
|
||||||
|
print(f" 仓位 {nt:>9,.0f}:全额成交 {full * 100:5.1f}%"
|
||||||
|
f" · 至少半额 {half * 100:5.1f}%"
|
||||||
|
f" · 可成交额中位 {np.median(avail):>10,.0f}")
|
||||||
|
# 成交率反推的资金上限。这才是绑定约束——它比冲击反推的上限
|
||||||
|
# 低一到两个数量级,而后者才是通常被当作「容量」的那个数
|
||||||
|
for want in (0.80, 0.90):
|
||||||
|
cap = float(np.quantile(avail, 1.0 - want))
|
||||||
|
print(f" → 要 {want * 100:.0f}% 的止盈全额成交,"
|
||||||
|
f"仓位须 ≤ {cap:,.0f} USDT")
|
||||||
|
print()
|
||||||
|
print(" 未计排队(我们的单排在该价位既有挂单之后),故仍是上界。")
|
||||||
|
print(" 回测把这些止盈按「全额成交在目标价」计,差多少就是收益虚多少")
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--books", default=None)
|
||||||
|
ap.add_argument("--tape", default=None)
|
||||||
|
a = ap.parse_args()
|
||||||
|
d = out_dir()
|
||||||
|
from lib.shadow_budget import BUDGET_BP
|
||||||
|
tape = Path(a.tape) if a.tape else d / "shadow_tape.jsonl.gz"
|
||||||
|
capacity(Path(a.books) if a.books else d / "shadow_books.jsonl.gz",
|
||||||
|
BUDGET_BP)
|
||||||
|
maker_fill(tape)
|
||||||
|
composite_fill(tape, d / "penetration.csv",
|
||||||
|
Path(__file__).resolve().parent / "cache")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
+458
-85
@@ -10,8 +10,25 @@
|
|||||||
为什么不能用 paper trade 的成交:那是 Hummingbot 自己的撮合模型模拟的,
|
为什么不能用 paper trade 的成交:那是 Hummingbot 自己的撮合模型模拟的,
|
||||||
测出来是模型行为不是市场行为。
|
测出来是模型行为不是市场行为。
|
||||||
|
|
||||||
口径对齐 aggregate_robustness.py:回测假设成交在**信号次根的开盘价**
|
口径对齐 step42_exit_tp_1m.py:回测假设成交在**信号次根的开盘价**
|
||||||
(entry_delay=1),所以基准价就是换根后新一根的 open。滑点为正表示比回测差。
|
(entry_delay=1),所以基准价就是换根后新一根的 open。滑点为正表示比回测差。
|
||||||
|
滤网(同向 + 中枢阶梯 + ATR 门控)在 shadow_signal.py 里,必须与预算同源。
|
||||||
|
|
||||||
|
### 统计口径三条硬要求
|
||||||
|
|
||||||
|
1. **主口径只用 pass_all 的信号根**。未过滤的照记但只作提前读数——
|
||||||
|
在我们根本不会下单的根上测滑点会把判据算宽
|
||||||
|
2. **条件漂移与无条件漂移分开报**。所以每根 K 线都记一份漂移
|
||||||
|
(shadow_drift.csv),不只信号根。两者的差就是「系统性追价」的大小
|
||||||
|
3. **出场腿按 maker/taker 分开**。止盈挂限价不吃滑点,把那 60% 混进
|
||||||
|
平均值会低估真实成本。出场腿属持仓管理,尚未实现
|
||||||
|
|
||||||
|
### lag 探针:超阈值要停开仓,不能只打日志
|
||||||
|
|
||||||
|
补丁只防得住「上游代码变了」,防不住 Bitget 再改一次消息格式。每根记
|
||||||
|
本地接收 − K 线收盘,近 30 根取中位数,超 800ms 即判该币不健康。
|
||||||
|
要停开仓是因为这种退化是**经济性且静默**的:不崩不报错,只让收益慢慢
|
||||||
|
变差,几周后才从统计里看得出来。影子期不下单,故落到 lag_ok 字段上。
|
||||||
|
|
||||||
### 盘口滚动缓冲把延迟变成自变量
|
### 盘口滚动缓冲把延迟变成自变量
|
||||||
|
|
||||||
@@ -38,22 +55,32 @@ from __future__ import annotations
|
|||||||
import argparse
|
import argparse
|
||||||
import asyncio
|
import asyncio
|
||||||
import csv
|
import csv
|
||||||
|
import gzip
|
||||||
|
import json
|
||||||
|
import math
|
||||||
import time
|
import time
|
||||||
from collections import deque
|
from collections import deque
|
||||||
from concurrent.futures import ProcessPoolExecutor
|
from concurrent.futures import ProcessPoolExecutor
|
||||||
|
from concurrent.futures.process import BrokenProcessPool
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
import pandas as pd
|
import pandas as pd
|
||||||
|
|
||||||
|
from lib.shadow_budget import LAG_ALARM_MS, LAG_WINDOW, lag_healthy
|
||||||
|
|
||||||
SYMS = ("BTC", "ETH", "SOL")
|
SYMS = ("BTC", "ETH", "SOL")
|
||||||
# 多存一根:deque 尾部是尚未收盘的当前根,剔除后正好剩 step39 定下的窗口
|
# 多存一根:deque 尾部是尚未收盘的当前根,剔除后正好剩 step39 定下的窗口
|
||||||
LTF_BARS, HTF_BARS = 2001, 801 # 有效窗口 2000 / 800,命中率在此饱和
|
LTF_BARS, HTF_BARS = 2001, 801 # 有效窗口 2000 / 800,命中率在此饱和
|
||||||
BOOK_HZ = 10 # 盘口采样 10Hz
|
BOOK_HZ = 10 # 盘口采样 10Hz
|
||||||
BOOK_KEEP_S = 30 # 缓冲保留 30 秒,够回查到 +5s
|
BOOK_KEEP_S = 30 # 缓冲保留 30 秒,够回查到 +5s
|
||||||
BOOK_DEPTH = 25
|
BOOK_TOL_MS = 250 # 回查容差:10Hz 正常 ≤100ms,留些余量
|
||||||
|
BOOK_DEPTH = 50 # 双边各 50 档,实测能撑 78 万~261 万美元
|
||||||
DELAYS_S = (0.5, 1.0, 2.0, 5.0) # 回查点
|
DELAYS_S = (0.5, 1.0, 2.0, 5.0) # 回查点
|
||||||
NOTIONALS = (1_000.0, 5_000.0, 20_000.0)
|
# 便利视图用的仓位档。**真正的答案在 shadow_books.jsonl.gz 里**——完整盘口
|
||||||
|
# 落了盘,任意资金量级的冲击都能离线算,换个规模不必重测。这里的档位只是
|
||||||
|
# 为了让 CSV 直接可读,覆盖到按 2ATR 止损反推的十万量级真实仓位
|
||||||
|
NOTIONALS = (50_000.0, 100_000.0, 320_000.0, 530_000.0)
|
||||||
NUM_COLS = ["timestamp", "open", "high", "low", "close", "volume"]
|
NUM_COLS = ["timestamp", "open", "high", "low", "close", "volume"]
|
||||||
|
|
||||||
|
|
||||||
@@ -62,6 +89,119 @@ def out_dir() -> Path:
|
|||||||
return p if p.is_dir() else Path(__file__).resolve().parents[1] / "out"
|
return p if p.is_dir() else Path(__file__).resolve().parents[1] / "out"
|
||||||
|
|
||||||
|
|
||||||
|
def _writer(path: Path, cols: list[str]):
|
||||||
|
"""追加模式打开;表头对不上就先把旧文件归档。
|
||||||
|
|
||||||
|
不校验的话,列一改,DictWriter 会按新顺序把行写到旧表头下面——
|
||||||
|
读出来整片错位,而且没有任何报错。长跑靠追加续命,这个校验是必需的。
|
||||||
|
"""
|
||||||
|
if path.exists() and path.stat().st_size > 0:
|
||||||
|
with path.open(newline="") as fh:
|
||||||
|
old = next(csv.reader(fh), [])
|
||||||
|
if old != cols:
|
||||||
|
arch = path.parent / "archive"
|
||||||
|
arch.mkdir(exist_ok=True)
|
||||||
|
dst = arch / f"{path.stem}_{time.strftime('%Y%m%d_%H%M%S')}.csv"
|
||||||
|
path.rename(dst)
|
||||||
|
print(f" [CSV] {path.name} 表头已变,旧数据归档为 {dst.name}",
|
||||||
|
flush=True)
|
||||||
|
fresh = not path.exists() or path.stat().st_size == 0
|
||||||
|
f = path.open("a", newline="")
|
||||||
|
w = csv.DictWriter(f, fieldnames=cols)
|
||||||
|
if fresh:
|
||||||
|
w.writeheader()
|
||||||
|
return f, w
|
||||||
|
|
||||||
|
|
||||||
|
class BookLog:
|
||||||
|
"""把完整盘口快照落成 gzip JSONL。
|
||||||
|
|
||||||
|
只记「某几个仓位档的成交价」的话,这批数据的寿命就等于那几个档位的寿命:
|
||||||
|
换一次资金规模就得重跑一周。存完整深度后,任意仓位的冲击都能离线重算,
|
||||||
|
一次采集回答所有资金量级的问题——包括容量上限那个必须现在就算、
|
||||||
|
不该等实盘暴露的数。
|
||||||
|
|
||||||
|
用 gzip 追加(多个 gzip 成员首尾相接仍可正常解压),进程被杀也只丢最后
|
||||||
|
一个缓冲块,不会毁掉整个文件。
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, path: Path) -> None:
|
||||||
|
self.path = path
|
||||||
|
self.fh = gzip.open(path, "at", encoding="utf-8")
|
||||||
|
self.n = 0
|
||||||
|
|
||||||
|
def write(self, sym: str, kline_ts: int, label: str, delay_ms: int,
|
||||||
|
target: int, book_ts: int, bids: np.ndarray,
|
||||||
|
asks: np.ndarray) -> None:
|
||||||
|
# 只留价与量两列,update_id 对离线分析没用。round 到 10 位避免
|
||||||
|
# float repr 把文件撑大一倍
|
||||||
|
rec = {"sym": sym, "kline_ts": kline_ts, "label": label,
|
||||||
|
"delay_ms": delay_ms, "target": target, "book_ts": book_ts,
|
||||||
|
"bids": [[round(float(p), 10), round(float(a), 10)]
|
||||||
|
for p, a, *_ in bids],
|
||||||
|
"asks": [[round(float(p), 10), round(float(a), 10)]
|
||||||
|
for p, a, *_ in asks]}
|
||||||
|
self.fh.write(json.dumps(rec, separators=(",", ":")) + "\n")
|
||||||
|
self.n += 1
|
||||||
|
|
||||||
|
def flush(self) -> None:
|
||||||
|
self.fh.flush()
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
try:
|
||||||
|
self.fh.close()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class TapeLog:
|
||||||
|
"""按 K 线、按价位聚合成交量,用来判 maker 腿能不能全额成交。
|
||||||
|
|
||||||
|
回测假设 3ATR 和 8ATR 的限价单全额成交。十万量级挂在那里,全成交还是
|
||||||
|
部分成交是完全不同的事——部分成交会把分批出场的收益结构改掉,而这个
|
||||||
|
问题盘口深度回答不了:深度说的是「现在有多少人挂着」,成交率问的是
|
||||||
|
「之后有多少人打过来」。只有成交流能回答。
|
||||||
|
|
||||||
|
**买卖必须分开存。** 多头在 3ATR 挂卖出止盈,成交靠的是主动**买盘**
|
||||||
|
打上来;把双边成交量合在一起会把成交率高估约一倍。
|
||||||
|
|
||||||
|
聚合到「根 × 价位」而不是逐笔:判据是「本根内有多少量在 ≥ 限价处成交」,
|
||||||
|
逐笔的时序对这个判据没有增量信息,而聚合能把体量压下两个数量级。
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, path: Path) -> None:
|
||||||
|
self.fh = gzip.open(path, "at", encoding="utf-8")
|
||||||
|
# sym -> side('b'/'s') -> price -> 累计基础币量
|
||||||
|
self.acc: dict[str, dict[str, dict[float, float]]] = {}
|
||||||
|
self.n_trades = 0
|
||||||
|
|
||||||
|
def add(self, sym: str, is_buy: bool, price: float, amount: float) -> None:
|
||||||
|
d = self.acc.setdefault(sym, {"b": {}, "s": {}})
|
||||||
|
side = d["b"] if is_buy else d["s"]
|
||||||
|
side[price] = side.get(price, 0.0) + amount
|
||||||
|
self.n_trades += 1
|
||||||
|
|
||||||
|
def flush_bar(self, sym: str, bar_ts: int) -> None:
|
||||||
|
"""一根走完就把这根的聚合结果落盘并清空。"""
|
||||||
|
d = self.acc.get(sym)
|
||||||
|
if not d or (not d["b"] and not d["s"]):
|
||||||
|
return
|
||||||
|
rec = {"sym": sym, "bar_ts": bar_ts,
|
||||||
|
"buys": {f"{p:.10g}": round(v, 10)
|
||||||
|
for p, v in sorted(d["b"].items())},
|
||||||
|
"sells": {f"{p:.10g}": round(v, 10)
|
||||||
|
for p, v in sorted(d["s"].items())}}
|
||||||
|
self.fh.write(json.dumps(rec, separators=(",", ":")) + "\n")
|
||||||
|
self.fh.flush()
|
||||||
|
self.acc[sym] = {"b": {}, "s": {}}
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
try:
|
||||||
|
self.fh.close()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
def hb_to_research(cdf: pd.DataFrame) -> pd.DataFrame:
|
def hb_to_research(cdf: pd.DataFrame) -> pd.DataFrame:
|
||||||
"""Hummingbot 的 candles_df 转成 research/lib/data.py 的列结构。
|
"""Hummingbot 的 candles_df 转成 research/lib/data.py 的列结构。
|
||||||
|
|
||||||
@@ -77,38 +217,32 @@ def hb_to_research(cdf: pd.DataFrame) -> pd.DataFrame:
|
|||||||
.sort_values("timestamp").reset_index(drop=True)
|
.sort_values("timestamp").reset_index(drop=True)
|
||||||
|
|
||||||
|
|
||||||
def walk_book(levels: list[tuple[float, float]], notional: float
|
def book_from(bids: np.ndarray, asks: np.ndarray):
|
||||||
) -> tuple[float, float]:
|
"""用缓冲里的快照临时搭一个 OrderBook,以便调用框架自带的吃单查询。
|
||||||
"""吃单到 notional(计价币)为止,返回 (加权成交价, 实际吃到的额度)。
|
|
||||||
|
|
||||||
深度不足时返回吃到的部分,由调用方按 filled < notional 判断是否可信。
|
自己手写吃单曾经踩过两个坑,框架版都没有:`get_vwap_for_volume` 返回的
|
||||||
|
是真加权均价(市价单的实际成交价),而 `get_price_for_quote_volume` 返回
|
||||||
|
的是**边际价**,用后者会高估冲击;深度不足时框架返回 nan 而不是一个
|
||||||
|
「看起来很正常」的部分成交均价,靠 query_volume/result_volume 判断。
|
||||||
"""
|
"""
|
||||||
if not levels:
|
from hummingbot.core.data_type.order_book import OrderBook
|
||||||
return float("nan"), 0.0
|
ob = OrderBook()
|
||||||
got = 0.0
|
ob.apply_numpy_snapshot(bids, asks)
|
||||||
cost = 0.0
|
return ob
|
||||||
qty = 0.0
|
|
||||||
for px, sz in levels:
|
|
||||||
avail = px * sz
|
|
||||||
take = min(avail, notional - got)
|
|
||||||
if take <= 0:
|
|
||||||
break
|
|
||||||
q = take / px
|
|
||||||
cost += q * px
|
|
||||||
qty += q
|
|
||||||
got += take
|
|
||||||
if got >= notional - 1e-9:
|
|
||||||
break
|
|
||||||
if qty <= 0:
|
|
||||||
return float("nan"), 0.0
|
|
||||||
return cost / qty, got
|
|
||||||
|
|
||||||
|
|
||||||
class BookBuffer:
|
class BookBuffer:
|
||||||
"""每币一份滚动盘口。按时间戳回查,取第一个不早于目标时刻的快照。"""
|
"""每币一份滚动盘口。按时间戳回查,取第一个不早于目标时刻的快照。
|
||||||
|
|
||||||
|
回查必须有容差上界。10Hz 下正常落在目标后 100ms 内(所有延迟点同向
|
||||||
|
偏约 +50ms,不影响曲线形状),但采样一旦卡顿,标着「0.5s」的那行可能
|
||||||
|
用的是 +3s 的盘口——数据看不出异常,判读却已经错了。超容差宁可丢弃,
|
||||||
|
并且把快照实际时刻写进 CSV,让这件事事后可查。
|
||||||
|
"""
|
||||||
|
|
||||||
def __init__(self) -> None:
|
def __init__(self) -> None:
|
||||||
self.buf: dict[str, deque] = {s: deque() for s in SYMS}
|
self.buf: dict[str, deque] = {s: deque() for s in SYMS}
|
||||||
|
self.n_stale = 0 # 因超容差被丢弃的回查次数
|
||||||
|
|
||||||
def push(self, sym: str, t_ms: int, bids: list, asks: list) -> None:
|
def push(self, sym: str, t_ms: int, bids: list, asks: list) -> None:
|
||||||
d = self.buf[sym]
|
d = self.buf[sym]
|
||||||
@@ -118,12 +252,13 @@ class BookBuffer:
|
|||||||
d.popleft()
|
d.popleft()
|
||||||
|
|
||||||
def at(self, sym: str, t_ms: int) -> tuple | None:
|
def at(self, sym: str, t_ms: int) -> tuple | None:
|
||||||
best = None
|
|
||||||
for snap in self.buf[sym]:
|
for snap in self.buf[sym]:
|
||||||
if snap[0] >= t_ms:
|
if snap[0] >= t_ms:
|
||||||
best = snap
|
if snap[0] - t_ms > BOOK_TOL_MS:
|
||||||
break
|
self.n_stale += 1
|
||||||
return best
|
return None
|
||||||
|
return snap
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
class Shadow:
|
class Shadow:
|
||||||
@@ -136,30 +271,84 @@ class Shadow:
|
|||||||
self.feeds_h: dict = {}
|
self.feeds_h: dict = {}
|
||||||
self.connector = None
|
self.connector = None
|
||||||
self.stop = asyncio.Event()
|
self.stop = asyncio.Event()
|
||||||
|
# 持有 fire-and-forget 任务的强引用。只 create_task 不留引用的话,
|
||||||
|
# 任务可能在完成前被 GC 掉,asyncio 官方文档明确警告过这一点
|
||||||
|
self._tasks: set = set()
|
||||||
|
self.n_broken = 0
|
||||||
|
self._hb_last_bars = 0
|
||||||
|
# 成交监听:已挂上的币,以及必须持有的 forwarder 强引用
|
||||||
|
# (PubSub 只存弱引用,不持有的话监听会被 GC 静默摘掉)
|
||||||
|
self._hooked: set[str] = set()
|
||||||
|
self._trade_fwd: dict = {}
|
||||||
self.n_signal = 0
|
self.n_signal = 0
|
||||||
|
self.n_pass = 0
|
||||||
self.n_bars = 0
|
self.n_bars = 0
|
||||||
|
# lag 探针的滚动窗口,逐币独立:一个币的行情退化不该连累其他币
|
||||||
|
self.lag_hist: dict[str, deque] = {
|
||||||
|
s: deque(maxlen=LAG_WINDOW) for s in SYMS}
|
||||||
|
self.lag_ok: dict[str, bool] = {s: True for s in SYMS}
|
||||||
d = out_dir()
|
d = out_dir()
|
||||||
# 追加模式:长跑期间若重启,已收集的样本不该被清掉
|
# 追加模式:长跑期间若重启,已收集的样本不该被清掉
|
||||||
p_sig = d / "shadow_signals.csv"
|
self.f_sig, self.w_sig = _writer(d / "shadow_signals.csv", [
|
||||||
new_sig = not p_sig.exists() or p_sig.stat().st_size == 0
|
"sym", "kline_ts", "direction",
|
||||||
self.f_sig = p_sig.open("a", newline="")
|
"h1_agree", "ladder_ok", "gate_ok", "pass_all", "lag_ok",
|
||||||
self.w_sig = csv.DictWriter(self.f_sig, fieldnames=[
|
"atr_pct", "atr_bp",
|
||||||
"sym", "kline_ts", "direction", "h1_agree",
|
|
||||||
"t_close_ms", "t_data_ms", "t_signal_ms",
|
"t_close_ms", "t_data_ms", "t_signal_ms",
|
||||||
"lag_data_ms", "lag_signal_ms",
|
"lag_data_ms", "lag_signal_ms",
|
||||||
"delay_label", "delay_ms", "notional",
|
"delay_label", "delay_ms", "book_ts", "book_lag_ms",
|
||||||
"baseline_px", "mid", "best_px", "fill_px", "filled",
|
"notional", "base_amt",
|
||||||
|
"baseline_px", "mid", "best_px", "fill_px", "filled", "depth_ok",
|
||||||
"slip_bp", "drift_bp", "spread_bp", "impact_bp"])
|
"slip_bp", "drift_bp", "spread_bp", "impact_bp"])
|
||||||
if new_sig:
|
self.f_lat, self.w_lat = _writer(d / "shadow_latency.csv", [
|
||||||
self.w_sig.writeheader()
|
|
||||||
p_lat = d / "shadow_latency.csv"
|
|
||||||
new_lat = not p_lat.exists() or p_lat.stat().st_size == 0
|
|
||||||
self.f_lat = p_lat.open("a", newline="")
|
|
||||||
self.w_lat = csv.DictWriter(self.f_lat, fieldnames=[
|
|
||||||
"sym", "kline_ts", "t_close_ms", "t_data_ms", "t_signal_ms",
|
"sym", "kline_ts", "t_close_ms", "t_data_ms", "t_signal_ms",
|
||||||
"lag_data_ms", "lag_signal_ms", "compute_ms", "n_bars", "n_hits"])
|
"lag_data_ms", "lag_signal_ms", "compute_ms", "n_bars", "n_hits",
|
||||||
if new_lat:
|
"n_pass", "atr_bp", "lag_med_ms", "lag_ok"])
|
||||||
self.w_lat.writeheader()
|
# 无条件漂移:每根都记,用来和信号根上的条件漂移对照
|
||||||
|
self.f_drf, self.w_drf = _writer(d / "shadow_drift.csv", [
|
||||||
|
"sym", "kline_ts", "delay_label", "delay_ms",
|
||||||
|
"book_ts", "book_lag_ms", "baseline_px", "mid", "drift_bp_long"])
|
||||||
|
# 完整深度。挂在无条件漂移那条路径上,所以每根 K 线的四个固定延迟点
|
||||||
|
# 都有一份,信号根上再补一份 actual 点
|
||||||
|
self.blog = BookLog(d / "shadow_books.jsonl.gz")
|
||||||
|
self.tape = TapeLog(d / "shadow_tape.jsonl.gz")
|
||||||
|
|
||||||
|
def _spawn(self, coro, what: str) -> None:
|
||||||
|
"""起一个后台任务,但异常要吼出来。
|
||||||
|
|
||||||
|
裸 create_task 的异常只在对象被 GC 时才由 asyncio 打一句
|
||||||
|
「Task exception was never retrieved」,很容易整晚没人发现。
|
||||||
|
这套东西最怕的就是不崩不报错的静默退化。
|
||||||
|
"""
|
||||||
|
async def guard():
|
||||||
|
try:
|
||||||
|
await coro
|
||||||
|
except asyncio.CancelledError:
|
||||||
|
raise
|
||||||
|
except Exception as e:
|
||||||
|
import traceback
|
||||||
|
print(f" [异常] {what}: {type(e).__name__}: {e}", flush=True)
|
||||||
|
traceback.print_exc()
|
||||||
|
|
||||||
|
t = asyncio.create_task(guard())
|
||||||
|
self._tasks.add(t)
|
||||||
|
t.add_done_callback(self._tasks.discard)
|
||||||
|
|
||||||
|
def _restart_pool(self) -> None:
|
||||||
|
"""进程池坏了之后重建。
|
||||||
|
|
||||||
|
用 spawn 而非 fork:此刻进程里已经有活跃的 WS 连接,fork 会把连接
|
||||||
|
状态一起复制进子进程。spawn 启动慢几秒,但只在故障时走这条路。
|
||||||
|
"""
|
||||||
|
import multiprocessing
|
||||||
|
self.n_broken += 1
|
||||||
|
try:
|
||||||
|
self.pool.shutdown(wait=False, cancel_futures=True)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
self.pool = ProcessPoolExecutor(
|
||||||
|
max_workers=self.workers,
|
||||||
|
mp_context=multiprocessing.get_context("spawn"))
|
||||||
|
print(f" [进程池] 已重建(第 {self.n_broken} 次)", flush=True)
|
||||||
|
|
||||||
# ---------- 启动 ----------
|
# ---------- 启动 ----------
|
||||||
|
|
||||||
@@ -167,7 +356,11 @@ class Shadow:
|
|||||||
from hummingbot.connector.derivative.bitget_perpetual.bitget_perpetual_derivative import (
|
from hummingbot.connector.derivative.bitget_perpetual.bitget_perpetual_derivative import (
|
||||||
BitgetPerpetualDerivative,
|
BitgetPerpetualDerivative,
|
||||||
)
|
)
|
||||||
from patched_candles import PatchedBitgetPerpetualCandles
|
from patched_candles import (PatchedBitgetPerpetualCandles,
|
||||||
|
assert_patch_effective)
|
||||||
|
|
||||||
|
# 覆盖失效是静默的(悄悄退回慢 1.06 秒,不报错),所以在启动就验一次
|
||||||
|
assert_patch_effective()
|
||||||
|
|
||||||
for s in SYMS:
|
for s in SYMS:
|
||||||
self.feeds_l[s] = PatchedBitgetPerpetualCandles(
|
self.feeds_l[s] = PatchedBitgetPerpetualCandles(
|
||||||
@@ -187,6 +380,7 @@ class Shadow:
|
|||||||
trading_required=False)
|
trading_required=False)
|
||||||
await self.connector.start_network()
|
await self.connector.start_network()
|
||||||
print(" 连接器已启动,等盘口与历史回填", flush=True)
|
print(" 连接器已启动,等盘口与历史回填", flush=True)
|
||||||
|
self._hook_trades()
|
||||||
|
|
||||||
t0 = time.time()
|
t0 = time.time()
|
||||||
while time.time() - t0 < 600:
|
while time.time() - t0 < 600:
|
||||||
@@ -200,6 +394,39 @@ class Shadow:
|
|||||||
f"1m {[len(self.feeds_l[s]._candles) for s in SYMS]} 根 · "
|
f"1m {[len(self.feeds_l[s]._candles) for s in SYMS]} 根 · "
|
||||||
f"5m {[len(self.feeds_h[s]._candles) for s in SYMS]} 根", flush=True)
|
f"5m {[len(self.feeds_h[s]._candles) for s in SYMS]} 根", flush=True)
|
||||||
|
|
||||||
|
def _hook_trades(self) -> None:
|
||||||
|
"""给每个盘口挂成交监听。
|
||||||
|
|
||||||
|
盘口对象可能还没建好(订阅是异步的),所以挂不上的先记下来,由
|
||||||
|
watch_bars 那圈重试;一直挂不上会在心跳里显示成交笔数为 0。
|
||||||
|
"""
|
||||||
|
from hummingbot.core.event.event_forwarder import EventForwarder
|
||||||
|
from hummingbot.core.event.events import OrderBookEvent
|
||||||
|
from hummingbot.core.data_type.common import TradeType
|
||||||
|
|
||||||
|
def make(sym: str):
|
||||||
|
def cb(ev) -> None:
|
||||||
|
self.tape.add(sym, ev.type == TradeType.BUY,
|
||||||
|
float(ev.price), float(ev.amount))
|
||||||
|
return EventForwarder(cb)
|
||||||
|
|
||||||
|
for s in SYMS:
|
||||||
|
if s in self._hooked: # 重复挂会让同一笔成交被记两次
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
ob = self.connector.get_order_book(f"{s}-USDT")
|
||||||
|
except Exception:
|
||||||
|
ob = None
|
||||||
|
if ob is None:
|
||||||
|
continue
|
||||||
|
fwd = make(s)
|
||||||
|
ob.add_listener(OrderBookEvent.TradeEvent, fwd)
|
||||||
|
self._trade_fwd[s] = fwd
|
||||||
|
self._hooked.add(s)
|
||||||
|
miss = [s for s in SYMS if s not in self._hooked]
|
||||||
|
print(f" 成交流已挂 {sorted(self._hooked)}"
|
||||||
|
+ (f",待重试 {miss}" if miss else ""), flush=True)
|
||||||
|
|
||||||
def _snapshot(self, sym: str):
|
def _snapshot(self, sym: str):
|
||||||
try:
|
try:
|
||||||
ob = self.connector.get_order_book(f"{sym}-USDT")
|
ob = self.connector.get_order_book(f"{sym}-USDT")
|
||||||
@@ -207,25 +434,41 @@ class Shadow:
|
|||||||
return None
|
return None
|
||||||
if ob is None:
|
if ob is None:
|
||||||
return None
|
return None
|
||||||
bids = [(float(r.price), float(r.amount))
|
# 存成 apply_numpy_snapshot 要的 [价, 量, update_id] 三列,
|
||||||
for r, _ in zip(ob.bid_entries(), range(BOOK_DEPTH))]
|
# 回查时才能直接搭 OrderBook 调框架的吃单查询
|
||||||
asks = [(float(r.price), float(r.amount))
|
bids = np.array([(float(r.price), float(r.amount), i)
|
||||||
for r, _ in zip(ob.ask_entries(), range(BOOK_DEPTH))]
|
for i, (r, _) in enumerate(
|
||||||
if not bids or not asks:
|
zip(ob.bid_entries(), range(BOOK_DEPTH)))])
|
||||||
|
asks = np.array([(float(r.price), float(r.amount), i)
|
||||||
|
for i, (r, _) in enumerate(
|
||||||
|
zip(ob.ask_entries(), range(BOOK_DEPTH)))])
|
||||||
|
if not len(bids) or not len(asks):
|
||||||
return None
|
return None
|
||||||
return bids, asks
|
return bids, asks
|
||||||
|
|
||||||
# ---------- 三个循环 ----------
|
# ---------- 三个循环 ----------
|
||||||
|
|
||||||
async def sample_books(self) -> None:
|
async def sample_books(self) -> None:
|
||||||
|
"""按截止时刻补睡,且对齐到墙钟 100ms 网格。
|
||||||
|
|
||||||
|
补睡是因为「干完活再睡固定时长」的实际周期是 100ms 加采样耗时,
|
||||||
|
名义 10Hz 到不了 10Hz。
|
||||||
|
|
||||||
|
对齐是因为回查目标都是 `kline_ts + n×500ms`,而 kline_ts 是整分钟,
|
||||||
|
所以目标必然落在墙钟 100ms 的整数倍上。采样相位若随启动时刻漂移,
|
||||||
|
每个回查点就会固定晚半个采样周期(实测 52ms)——四个固定延迟点
|
||||||
|
同向偏置,虽不改曲线形状,但白白多算了 50ms 的漂移。
|
||||||
|
"""
|
||||||
period = 1.0 / BOOK_HZ
|
period = 1.0 / BOOK_HZ
|
||||||
|
nxt = math.ceil(time.time() / period) * period
|
||||||
while not self.stop.is_set():
|
while not self.stop.is_set():
|
||||||
t = int(time.time() * 1000)
|
t = int(time.time() * 1000)
|
||||||
for s in SYMS:
|
for s in SYMS:
|
||||||
snap = self._snapshot(s)
|
snap = self._snapshot(s)
|
||||||
if snap:
|
if snap:
|
||||||
self.books.push(s, t, snap[0], snap[1])
|
self.books.push(s, t, snap[0], snap[1])
|
||||||
await asyncio.sleep(period)
|
nxt += period
|
||||||
|
await asyncio.sleep(max(0.0, nxt - time.time()))
|
||||||
|
|
||||||
async def watch_bars(self) -> None:
|
async def watch_bars(self) -> None:
|
||||||
last = {s: (int(self.feeds_l[s]._candles[-1][0])
|
last = {s: (int(self.feeds_l[s]._candles[-1][0])
|
||||||
@@ -239,7 +482,11 @@ class Shadow:
|
|||||||
if last[s] is not None and newest > last[s]:
|
if last[s] is not None and newest > last[s]:
|
||||||
t_data = int(time.time() * 1000)
|
t_data = int(time.time() * 1000)
|
||||||
kts = newest * 1000 if newest < 1e12 else newest
|
kts = newest * 1000 if newest < 1e12 else newest
|
||||||
asyncio.create_task(self.on_bar(s, kts, t_data))
|
# 刚收盘那根的成交聚合先落盘,再算信号
|
||||||
|
self.tape.flush_bar(s, kts)
|
||||||
|
if len(self._hooked) < len(SYMS):
|
||||||
|
self._hook_trades() # 换根时才重试,避免重复挂
|
||||||
|
self._spawn(self.on_bar(s, kts, t_data), f"on_bar {s}")
|
||||||
last[s] = newest
|
last[s] = newest
|
||||||
await asyncio.sleep(0.01)
|
await asyncio.sleep(0.01)
|
||||||
|
|
||||||
@@ -252,15 +499,28 @@ class Shadow:
|
|||||||
df_h = df_h[df_h["timestamp"] < kline_ts]
|
df_h = df_h[df_h["timestamp"] < kline_ts]
|
||||||
baseline = self._new_bar_open(sym, kline_ts)
|
baseline = self._new_bar_open(sym, kline_ts)
|
||||||
|
|
||||||
|
lag_med, lag_ok = self._probe_lag(sym, t_data - kline_ts)
|
||||||
|
|
||||||
t0 = time.perf_counter()
|
t0 = time.perf_counter()
|
||||||
payload = (df_l[NUM_COLS].values.tolist(),
|
payload = (df_l[NUM_COLS].values.tolist(),
|
||||||
df_h[NUM_COLS].values.tolist())
|
df_h[NUM_COLS].values.tolist(), baseline)
|
||||||
loop = asyncio.get_running_loop()
|
loop = asyncio.get_running_loop()
|
||||||
from shadow_signal import compute_packed
|
from shadow_signal import compute_packed
|
||||||
res = await loop.run_in_executor(self.pool, compute_packed, payload)
|
try:
|
||||||
|
res = await loop.run_in_executor(self.pool, compute_packed, payload)
|
||||||
|
except BrokenProcessPool as e:
|
||||||
|
# 不重建的话,之后每一根都会走到这里,采集静默停摆到跑完为止
|
||||||
|
print(f" [{sym}] 进程池损坏 {e},重建后跳过本根", flush=True)
|
||||||
|
self._restart_pool()
|
||||||
|
return
|
||||||
compute_ms = int((time.perf_counter() - t0) * 1000)
|
compute_ms = int((time.perf_counter() - t0) * 1000)
|
||||||
t_signal = int(time.time() * 1000)
|
t_signal = int(time.time() * 1000)
|
||||||
|
|
||||||
|
hits = res.get("hits", [])
|
||||||
|
atr_pct = res.get("atr_pct")
|
||||||
|
atr_bp = round(atr_pct * 1e4, 3) if atr_pct else ""
|
||||||
|
n_pass = sum(h["pass_all"] for h in hits)
|
||||||
|
|
||||||
self.n_bars += 1
|
self.n_bars += 1
|
||||||
self.w_lat.writerow({
|
self.w_lat.writerow({
|
||||||
"sym": sym, "kline_ts": kline_ts, "t_close_ms": kline_ts,
|
"sym": sym, "kline_ts": kline_ts, "t_close_ms": kline_ts,
|
||||||
@@ -268,13 +528,18 @@ class Shadow:
|
|||||||
"lag_data_ms": t_data - kline_ts,
|
"lag_data_ms": t_data - kline_ts,
|
||||||
"lag_signal_ms": t_signal - kline_ts,
|
"lag_signal_ms": t_signal - kline_ts,
|
||||||
"compute_ms": compute_ms, "n_bars": res.get("n_bars", 0),
|
"compute_ms": compute_ms, "n_bars": res.get("n_bars", 0),
|
||||||
"n_hits": len(res.get("hits", []))})
|
"n_hits": len(hits), "n_pass": n_pass, "atr_bp": atr_bp,
|
||||||
|
"lag_med_ms": lag_med, "lag_ok": int(lag_ok)})
|
||||||
self.f_lat.flush()
|
self.f_lat.flush()
|
||||||
|
|
||||||
|
if baseline is not None and np.isfinite(baseline):
|
||||||
|
# 无条件漂移:每根都记,不管有没有信号
|
||||||
|
self._spawn(self._drift_later(sym, kline_ts, baseline),
|
||||||
|
f"drift {sym}")
|
||||||
|
|
||||||
if res.get("error"):
|
if res.get("error"):
|
||||||
print(f" [{sym}] 信号计算出错 {res['error']}", flush=True)
|
print(f" [{sym}] 信号计算出错 {res['error']}", flush=True)
|
||||||
return
|
return
|
||||||
hits = res.get("hits", [])
|
|
||||||
if not hits:
|
if not hits:
|
||||||
return
|
return
|
||||||
if baseline is None or not np.isfinite(baseline):
|
if baseline is None or not np.isfinite(baseline):
|
||||||
@@ -283,20 +548,73 @@ class Shadow:
|
|||||||
|
|
||||||
for h in hits:
|
for h in hits:
|
||||||
self.n_signal += 1
|
self.n_signal += 1
|
||||||
print(f" ★ [{sym}] {kline_ts} 方向 {h['direction']:+d} "
|
self.n_pass += h["pass_all"]
|
||||||
f"h1_agree={h['h1_agree']} · 数据 {t_data - kline_ts}ms "
|
mark = "★" if h["pass_all"] else "·"
|
||||||
|
print(f" {mark} [{sym}] {kline_ts} 方向 {h['direction']:+d} "
|
||||||
|
f"同向{h['h1_agree']} 阶梯{h['ladder_ok']} 门控{h['gate_ok']} "
|
||||||
|
f"(ATR {atr_bp or 'na'}bp) · 数据 {t_data - kline_ts}ms "
|
||||||
f"信号 {t_signal - kline_ts}ms", flush=True)
|
f"信号 {t_signal - kline_ts}ms", flush=True)
|
||||||
# 最远的回查点在 t_close+5s,此刻尚未发生;等它过去再一次性落盘
|
# 最远的回查点在 t_close+5s,此刻尚未发生;等它过去再一次性落盘
|
||||||
asyncio.create_task(
|
self._spawn(
|
||||||
self._record_later(sym, kline_ts, h, t_data, t_signal, baseline))
|
self._record_later(sym, kline_ts, h, t_data, t_signal,
|
||||||
|
baseline, atr_pct, lag_ok),
|
||||||
|
f"record {sym}")
|
||||||
|
|
||||||
async def _record_later(self, sym: str, kline_ts: int, hit: dict,
|
def _probe_lag(self, sym: str, lag_ms: int) -> tuple[float, bool]:
|
||||||
t_data: int, t_signal: int, baseline: float) -> None:
|
"""记一根的到达延迟,返回 (滚动中位数, 该币是否健康)。
|
||||||
target = kline_ts + int(max(DELAYS_S) * 1000) + 500
|
|
||||||
wait = target / 1000.0 - time.time()
|
不健康时应停止开新仓;影子期不下单,故只落到 lag_ok 字段并告警。
|
||||||
|
"""
|
||||||
|
self.lag_hist[sym].append(lag_ms)
|
||||||
|
ok = lag_healthy(self.lag_hist[sym])
|
||||||
|
med = float(np.median(self.lag_hist[sym]))
|
||||||
|
if ok != self.lag_ok[sym]:
|
||||||
|
state = "恢复" if ok else f"退化,超 {LAG_ALARM_MS:.0f}ms 阈值,停开新仓"
|
||||||
|
print(f" [lag] {sym} {state}:近 {len(self.lag_hist[sym])} 根"
|
||||||
|
f"中位 {med:.0f}ms", flush=True)
|
||||||
|
self.lag_ok[sym] = ok
|
||||||
|
return round(med, 1), ok
|
||||||
|
|
||||||
|
async def _wait_for_delays(self, kline_ts: int) -> None:
|
||||||
|
"""最远回查点是 t_close+5s,等它过去(多留 0.5s 给采样)。"""
|
||||||
|
wait = (kline_ts + int(max(DELAYS_S) * 1000) + 500) / 1000.0 - time.time()
|
||||||
if wait > 0:
|
if wait > 0:
|
||||||
await asyncio.sleep(wait)
|
await asyncio.sleep(wait)
|
||||||
self._record(sym, kline_ts, hit, t_data, t_signal, baseline)
|
|
||||||
|
async def _record_later(self, sym: str, kline_ts: int, hit: dict,
|
||||||
|
t_data: int, t_signal: int, baseline: float,
|
||||||
|
atr_pct: float | None, lag_ok: bool) -> None:
|
||||||
|
await self._wait_for_delays(kline_ts)
|
||||||
|
self._record(sym, kline_ts, hit, t_data, t_signal, baseline,
|
||||||
|
atr_pct, lag_ok)
|
||||||
|
|
||||||
|
async def _drift_later(self, sym: str, kline_ts: int,
|
||||||
|
baseline: float) -> None:
|
||||||
|
await self._wait_for_delays(kline_ts)
|
||||||
|
for label, delay_ms in self._points(None):
|
||||||
|
target = kline_ts + delay_ms
|
||||||
|
snap = self.books.at(sym, target)
|
||||||
|
if snap is None:
|
||||||
|
continue
|
||||||
|
book_ts, bids, asks = snap
|
||||||
|
mid = (float(bids[0][0]) + float(asks[0][0])) / 2.0
|
||||||
|
self.blog.write(sym, kline_ts, label, delay_ms, target,
|
||||||
|
book_ts, bids, asks)
|
||||||
|
self.w_drf.writerow({
|
||||||
|
"sym": sym, "kline_ts": kline_ts, "delay_label": label,
|
||||||
|
"delay_ms": delay_ms, "book_ts": book_ts,
|
||||||
|
"book_lag_ms": book_ts - target, "baseline_px": baseline,
|
||||||
|
"mid": mid,
|
||||||
|
"drift_bp_long": round((mid - baseline) / baseline * 1e4, 4)})
|
||||||
|
self.f_drf.flush()
|
||||||
|
self.blog.flush() # 每根冲刷一次,进程被杀最多丢一根
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _points(t_signal_delay: int | None) -> list[tuple[str, int]]:
|
||||||
|
pts = [(f"{d}s", int(d * 1000)) for d in DELAYS_S]
|
||||||
|
if t_signal_delay is not None:
|
||||||
|
pts.insert(0, ("actual", t_signal_delay))
|
||||||
|
return pts
|
||||||
|
|
||||||
def _new_bar_open(self, sym: str, kline_ts: int) -> float | None:
|
def _new_bar_open(self, sym: str, kline_ts: int) -> float | None:
|
||||||
"""次根开盘价 = 回测假设的成交价。"""
|
"""次根开盘价 = 回测假设的成交价。"""
|
||||||
@@ -309,25 +627,56 @@ class Shadow:
|
|||||||
return float(row[1]) if ts == kline_ts else None
|
return float(row[1]) if ts == kline_ts else None
|
||||||
|
|
||||||
def _record(self, sym: str, kline_ts: int, hit: dict,
|
def _record(self, sym: str, kline_ts: int, hit: dict,
|
||||||
t_data: int, t_signal: int, baseline: float) -> None:
|
t_data: int, t_signal: int, baseline: float,
|
||||||
points = [("actual", t_signal - kline_ts)]
|
atr_pct: float | None, lag_ok: bool) -> None:
|
||||||
points += [(f"{d}s", int(d * 1000)) for d in DELAYS_S]
|
|
||||||
d_sign = hit["direction"]
|
d_sign = hit["direction"]
|
||||||
|
|
||||||
for label, delay_ms in points:
|
for label, delay_ms in self._points(t_signal - kline_ts):
|
||||||
snap = self.books.at(sym, kline_ts + delay_ms)
|
target = kline_ts + delay_ms
|
||||||
|
snap = self.books.at(sym, target)
|
||||||
if snap is None:
|
if snap is None:
|
||||||
continue
|
continue
|
||||||
_, bids, asks = snap
|
book_ts, bids, asks = snap
|
||||||
best_bid, best_ask = bids[0][0], asks[0][0]
|
best_bid, best_ask = float(bids[0][0]), float(asks[0][0])
|
||||||
mid = (best_bid + best_ask) / 2.0
|
mid = (best_bid + best_ask) / 2.0
|
||||||
# 多头吃卖盘,空头吃买盘
|
|
||||||
side = asks if d_sign > 0 else bids
|
|
||||||
best_px = best_ask if d_sign > 0 else best_bid
|
best_px = best_ask if d_sign > 0 else best_bid
|
||||||
|
ob = book_from(bids, asks)
|
||||||
|
if label == "actual":
|
||||||
|
# 四个固定点已由无条件漂移那条路径落过,只补这一个
|
||||||
|
self.blog.write(sym, kline_ts, label, delay_ms, target,
|
||||||
|
book_ts, bids, asks)
|
||||||
|
|
||||||
for notional in NOTIONALS:
|
for notional in NOTIONALS:
|
||||||
fill, filled = walk_book(side, notional)
|
# 名义额按基准价折成基础币再下单——真实委托是基础币计价的,
|
||||||
|
# 框架的 get_vwap_for_volume 也收基础币量。名义额那一栏留着
|
||||||
|
# 是为了跨币可比(1 BTC 和 1 SOL 没法横向比)
|
||||||
|
base_amt = notional / baseline
|
||||||
|
r = ob.get_vwap_for_volume(d_sign > 0, base_amt)
|
||||||
|
fill = float(r.result_price)
|
||||||
|
depth_ok = int(float(r.result_volume) >= base_amt * 0.999)
|
||||||
if not np.isfinite(fill):
|
if not np.isfinite(fill):
|
||||||
|
# 25 档吃不下这个量,框架直接给 nan。记一行标明深度不足,
|
||||||
|
# 免得「某个仓位档在薄盘时段整段消失」看不出来
|
||||||
|
self.w_sig.writerow({
|
||||||
|
"sym": sym, "kline_ts": kline_ts, "direction": d_sign,
|
||||||
|
"h1_agree": hit["h1_agree"],
|
||||||
|
"ladder_ok": hit["ladder_ok"],
|
||||||
|
"gate_ok": hit["gate_ok"], "pass_all": hit["pass_all"],
|
||||||
|
"lag_ok": int(lag_ok),
|
||||||
|
"atr_pct": atr_pct if atr_pct else "",
|
||||||
|
"atr_bp": round(atr_pct * 1e4, 3) if atr_pct else "",
|
||||||
|
"t_close_ms": kline_ts, "t_data_ms": t_data,
|
||||||
|
"t_signal_ms": t_signal,
|
||||||
|
"lag_data_ms": t_data - kline_ts,
|
||||||
|
"lag_signal_ms": t_signal - kline_ts,
|
||||||
|
"delay_label": label, "delay_ms": delay_ms,
|
||||||
|
"book_ts": book_ts, "book_lag_ms": book_ts - target,
|
||||||
|
"notional": notional, "base_amt": round(base_amt, 8),
|
||||||
|
"baseline_px": baseline, "mid": mid,
|
||||||
|
"best_px": best_px, "fill_px": "",
|
||||||
|
"filled": round(float(r.result_volume), 8),
|
||||||
|
"depth_ok": 0, "slip_bp": "", "drift_bp": "",
|
||||||
|
"spread_bp": "", "impact_bp": ""})
|
||||||
continue
|
continue
|
||||||
slip = d_sign * (fill - baseline) / baseline * 1e4
|
slip = d_sign * (fill - baseline) / baseline * 1e4
|
||||||
drift = d_sign * (mid - baseline) / baseline * 1e4
|
drift = d_sign * (mid - baseline) / baseline * 1e4
|
||||||
@@ -336,14 +685,21 @@ class Shadow:
|
|||||||
self.w_sig.writerow({
|
self.w_sig.writerow({
|
||||||
"sym": sym, "kline_ts": kline_ts,
|
"sym": sym, "kline_ts": kline_ts,
|
||||||
"direction": d_sign, "h1_agree": hit["h1_agree"],
|
"direction": d_sign, "h1_agree": hit["h1_agree"],
|
||||||
|
"ladder_ok": hit["ladder_ok"], "gate_ok": hit["gate_ok"],
|
||||||
|
"pass_all": hit["pass_all"], "lag_ok": int(lag_ok),
|
||||||
|
"atr_pct": atr_pct if atr_pct else "",
|
||||||
|
"atr_bp": round(atr_pct * 1e4, 3) if atr_pct else "",
|
||||||
"t_close_ms": kline_ts, "t_data_ms": t_data,
|
"t_close_ms": kline_ts, "t_data_ms": t_data,
|
||||||
"t_signal_ms": t_signal,
|
"t_signal_ms": t_signal,
|
||||||
"lag_data_ms": t_data - kline_ts,
|
"lag_data_ms": t_data - kline_ts,
|
||||||
"lag_signal_ms": t_signal - kline_ts,
|
"lag_signal_ms": t_signal - kline_ts,
|
||||||
"delay_label": label, "delay_ms": delay_ms,
|
"delay_label": label, "delay_ms": delay_ms,
|
||||||
"notional": notional, "baseline_px": baseline,
|
"book_ts": book_ts, "book_lag_ms": book_ts - target,
|
||||||
|
"notional": notional, "base_amt": round(base_amt, 8),
|
||||||
|
"baseline_px": baseline,
|
||||||
"mid": mid, "best_px": best_px, "fill_px": fill,
|
"mid": mid, "best_px": best_px, "fill_px": fill,
|
||||||
"filled": round(filled, 2),
|
"filled": round(float(r.result_volume), 8),
|
||||||
|
"depth_ok": depth_ok,
|
||||||
"slip_bp": round(slip, 4), "drift_bp": round(drift, 4),
|
"slip_bp": round(slip, 4), "drift_bp": round(drift, 4),
|
||||||
"spread_bp": round(spread, 4),
|
"spread_bp": round(spread, 4),
|
||||||
"impact_bp": round(impact, 4)})
|
"impact_bp": round(impact, 4)})
|
||||||
@@ -353,8 +709,22 @@ class Shadow:
|
|||||||
while not self.stop.is_set():
|
while not self.stop.is_set():
|
||||||
await asyncio.sleep(300)
|
await asyncio.sleep(300)
|
||||||
depth = {s: len(self.books.buf[s]) for s in SYMS}
|
depth = {s: len(self.books.buf[s]) for s in SYMS}
|
||||||
print(f" [心跳] 已处理 {self.n_bars} 根 · 命中 {self.n_signal} 个 "
|
lag = {s: (f"{np.median(h):.0f}ms" if h else "na")
|
||||||
f"· 盘口缓冲 {depth}", flush=True)
|
+ ("" if self.lag_ok[s] else "!")
|
||||||
|
for s, h in self.lag_hist.items()}
|
||||||
|
print(f" [心跳] 已处理 {self.n_bars} 根 · 命中 {self.n_signal} 个"
|
||||||
|
f"(过全部滤网 {self.n_pass}) · lag {lag} · 盘口缓冲 {depth}"
|
||||||
|
f" · 回查超容差 {self.books.n_stale} 次"
|
||||||
|
f" · 在途任务 {len(self._tasks)}"
|
||||||
|
f" · 盘口落盘 {self.blog.n} 份"
|
||||||
|
f" · 成交 {self.tape.n_trades} 笔{'' if self.tape.n_trades else ' ⚠监听未生效'}",
|
||||||
|
flush=True)
|
||||||
|
# 五分钟一根都没进来,说明管道断了。不喊一声就只能靠人翻日志
|
||||||
|
if self.n_bars == self._hb_last_bars:
|
||||||
|
print(f" ⚠ [停滞] 距上次心跳未处理任何 K 线"
|
||||||
|
f"(进程池重建 {self.n_broken} 次),管道可能已断",
|
||||||
|
flush=True)
|
||||||
|
self._hb_last_bars = self.n_bars
|
||||||
|
|
||||||
async def run(self) -> None:
|
async def run(self) -> None:
|
||||||
await self.start()
|
await self.start()
|
||||||
@@ -370,9 +740,12 @@ class Shadow:
|
|||||||
for f in list(self.feeds_l.values()) + list(self.feeds_h.values()):
|
for f in list(self.feeds_l.values()) + list(self.feeds_h.values()):
|
||||||
f.stop()
|
f.stop()
|
||||||
await self.connector.stop_network()
|
await self.connector.stop_network()
|
||||||
self.f_sig.close()
|
for f in (self.f_sig, self.f_lat, self.f_drf):
|
||||||
self.f_lat.close()
|
f.close()
|
||||||
print(f"\n收工:{self.n_bars} 根 · {self.n_signal} 个信号", flush=True)
|
self.blog.close()
|
||||||
|
self.tape.close()
|
||||||
|
print(f"\n收工:{self.n_bars} 根 · {self.n_signal} 个信号"
|
||||||
|
f"(过全部滤网 {self.n_pass})", flush=True)
|
||||||
|
|
||||||
|
|
||||||
async def main_async(workers: int, hours: float, pool) -> None:
|
async def main_async(workers: int, hours: float, pool) -> None:
|
||||||
|
|||||||
+179
-53
@@ -1,27 +1,44 @@
|
|||||||
"""影子交易器的报表:首日延迟门槛 + 滑点对延迟曲线。
|
"""影子交易器的报表:延迟门槛 + 滑点对延迟曲线。
|
||||||
|
|
||||||
两份产物对应计划里的两件事。
|
### 判据常数一律从研究侧 import,不在这里写死
|
||||||
|
|
||||||
### 延迟门槛(提前止损用)
|
`BUDGET_BP` 等常数留在 `research/lib/shadow_budget.py`。理由是这些数会变——
|
||||||
|
2026-08-27 一天之内预算就动了四次(3.91 → 11.06 → 14.25 → 15.19bp),
|
||||||
|
费率也改了一次。本文件曾经写死过 BTC −0.13 / ETH 4.02 / SOL 2.92,那三个数
|
||||||
|
由六处差异叠加而来(只有同向没有阶梯、费率按 6bp 双边 taker、TP=3.0、
|
||||||
|
余量没除 taker 名义额、无 ATR 门控,且 BTC/ETH/SOL 恰是 ATR 最低的三个币)。
|
||||||
|
正确值是 8.58 / 20.64 / 16.83——**ETH 差了五倍**。
|
||||||
|
|
||||||
跑满 24 小时先看这个。若**总延迟已令预期漂移超过余量**,说明方案在这台机器
|
这件事要紧是因为下面的判读是自动停机开关:用 4.02 当 ETH 的预算,真实滑点
|
||||||
上就不成立,不必等两周样本再停。余量取 bitget_baseline.py 的实测值:
|
只要到 2.4bp 就会报「需要压延迟或放弃」,会误杀一个可行的策略。
|
||||||
BTC −0.13bp(本就为负,只作参照)、ETH +4.02bp、SOL +2.92bp。
|
|
||||||
|
|
||||||
漂移按随机游走折算:σ_1m · √(t/60)。这是下限——入场条件是「收盘突破转强」,
|
### 什么时候能判什么
|
||||||
那一刻价格正朝我们方向跑,延迟造成的是系统性追价,不会正负抵消。所以实测
|
|
||||||
滑点理应比这个折算值更差,两者对照本身就是个校验。
|
| | 一天的样本量 | 够不够 |
|
||||||
|
|---|---|---|
|
||||||
|
| 延迟 | 1440 根/币 | 够,统计上很厚 |
|
||||||
|
| 滑点 | 门控后 4~6 笔/天 | **不够**,判据要 30 笔以上,即一周起步 |
|
||||||
|
|
||||||
|
所以首日只能判延迟和管道通不通。滑点那一节在样本不足时会明说。
|
||||||
|
|
||||||
|
### 延迟门槛(提前止损用)
|
||||||
|
|
||||||
|
若**总延迟已令预期漂移超过预算**,说明方案在这台机器上就不成立,不必等
|
||||||
|
两周样本再停。漂移按随机游走折算 σ_1m · √(t/60)。这是下限——入场条件是
|
||||||
|
「收盘突破转强」,那一刻价格正朝我们方向跑,延迟造成的是系统性追价,
|
||||||
|
不会正负抵消。所以实测滑点理应比折算值更差,两者对照本身就是个校验。
|
||||||
|
|
||||||
### 滑点对延迟曲线
|
### 滑点对延迟曲线
|
||||||
|
|
||||||
把延迟当自变量:0.5s / 1s / 2s / 5s 各一个滑点值,外加「actual」= 本机实际
|
把延迟当自变量:0.5s / 1s / 2s / 5s 各一个滑点值,外加「actual」= 本机实际
|
||||||
算完的时刻。这样即便本机算得慢,也能读出「若延迟压到 X 秒,滑点是多少」,
|
算完的时刻。同信号内的受控对比,能直接读出「若延迟压到 X 秒,滑点是多少」,
|
||||||
决策不被当前实现拖累。
|
决策不被当前实现拖累。
|
||||||
|
|
||||||
.venv/bin/python research/live/shadow_report.py
|
.venv/bin/python research/live/shadow_report.py
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sys
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
@@ -29,8 +46,16 @@ import pandas as pd
|
|||||||
|
|
||||||
HERE = Path(__file__).resolve().parent
|
HERE = Path(__file__).resolve().parent
|
||||||
OUT = HERE.parent / "out"
|
OUT = HERE.parent / "out"
|
||||||
|
sys.path.insert(0, str(HERE.parent))
|
||||||
|
|
||||||
|
from lib.shadow_budget import ( # noqa: E402
|
||||||
|
ATR_GATE_BP, BUDGET_PORTFOLIO_2026, LAG_ALARM_MS, budget_of, lag_healthy,
|
||||||
|
verdict,
|
||||||
|
)
|
||||||
|
|
||||||
SYMS = ("BTC", "ETH", "SOL")
|
SYMS = ("BTC", "ETH", "SOL")
|
||||||
BUDGET_BP = {"BTC": -0.13, "ETH": 4.02, "SOL": 2.92}
|
ORDER = ["0.5s", "1.0s", "2.0s", "5.0s", "actual"]
|
||||||
|
MIN_N = 30 # 滑点判据的最低笔数,低于此只报数不下结论
|
||||||
|
|
||||||
|
|
||||||
def vol_bp() -> dict[str, float]:
|
def vol_bp() -> dict[str, float]:
|
||||||
@@ -46,13 +71,38 @@ def vol_bp() -> dict[str, float]:
|
|||||||
return v
|
return v
|
||||||
|
|
||||||
|
|
||||||
|
def lag_health(lat: pd.DataFrame) -> None:
|
||||||
|
"""运行时 lag 探针的回看。补丁后实测 506~642ms,理论下限约 500ms。"""
|
||||||
|
print("\n\n########## 二、lag 探针(>%.0fms 该停开仓)##########"
|
||||||
|
% LAG_ALARM_MS)
|
||||||
|
print(f"{'币':<5}{'根数':>6}{'中位ms':>9}{'P90ms':>8}{'最差30根中位':>14}"
|
||||||
|
f"{'超阈根数':>10}{'判定':>8}")
|
||||||
|
for s in SYMS:
|
||||||
|
g = lat[lat["sym"] == s].sort_values("kline_ts")
|
||||||
|
if g.empty:
|
||||||
|
continue
|
||||||
|
x = g["lag_data_ms"].to_numpy(float)
|
||||||
|
roll = pd.Series(x).rolling(30).median()
|
||||||
|
worst = float(np.nanmax(roll)) if roll.notna().any() else float("nan")
|
||||||
|
ok = lag_healthy(x)
|
||||||
|
print(f"{s:<5}{len(g):>6}{np.median(x):>9.0f}"
|
||||||
|
f"{np.percentile(x, 90):>8.0f}{worst:>14.0f}"
|
||||||
|
f"{int((x > LAG_ALARM_MS).sum()):>10}"
|
||||||
|
f"{'健康' if ok else '退化':>8}")
|
||||||
|
if "lag_ok" in lat.columns:
|
||||||
|
bad = int((lat["lag_ok"] == 0).sum())
|
||||||
|
if bad:
|
||||||
|
print(f"\n 采集期间有 {bad} 根被判不健康,那些根上的信号"
|
||||||
|
f"(lag_ok=0)在真实运行下不会开仓,统计时应排除")
|
||||||
|
|
||||||
|
|
||||||
def latency_gate(lat: pd.DataFrame, vols: dict) -> None:
|
def latency_gate(lat: pd.DataFrame, vols: dict) -> None:
|
||||||
print("########## 一、延迟门槛 ##########")
|
print("########## 一、延迟门槛 ##########")
|
||||||
span_h = (lat["t_close_ms"].max() - lat["t_close_ms"].min()) / 3.6e6
|
span_h = (lat["t_close_ms"].max() - lat["t_close_ms"].min()) / 3.6e6
|
||||||
print(f"样本跨度 {span_h:.1f} 小时 · 共 {len(lat)} 根\n")
|
print(f"样本跨度 {span_h:.1f} 小时 · 共 {len(lat)} 根\n")
|
||||||
print(f"{'币':<5}{'根数':>6}{'数据ms':>9}{'计算ms':>9}{'总延迟ms':>10}"
|
print(f"{'币':<5}{'根数':>6}{'数据ms':>9}{'计算ms':>9}{'总延迟ms':>10}"
|
||||||
f"{'P90ms':>8}{'折算漂移bp':>12}{'余量bp':>9}{'占余量':>9}")
|
f"{'P90ms':>8}{'折算漂移bp':>12}{'预算bp':>9}{'占预算':>9}")
|
||||||
verdicts = {}
|
rows = {}
|
||||||
for s in SYMS:
|
for s in SYMS:
|
||||||
g = lat[lat["sym"] == s]
|
g = lat[lat["sym"] == s]
|
||||||
if g.empty:
|
if g.empty:
|
||||||
@@ -63,48 +113,110 @@ def latency_gate(lat: pd.DataFrame, vols: dict) -> None:
|
|||||||
p90 = float(np.percentile(g["lag_signal_ms"], 90))
|
p90 = float(np.percentile(g["lag_signal_ms"], 90))
|
||||||
vol = vols.get(s)
|
vol = vols.get(s)
|
||||||
drift = vol * np.sqrt(t / 60_000) if vol else float("nan")
|
drift = vol * np.sqrt(t / 60_000) if vol else float("nan")
|
||||||
b = BUDGET_BP[s]
|
b = budget_of(s)
|
||||||
share = drift / b if b > 0 else float("nan")
|
share = drift / b if np.isfinite(b) and b > 0 else float("nan")
|
||||||
verdicts[s] = (drift, b)
|
rows[s] = (drift, b)
|
||||||
txt = f"{share * 100:.0f}%" if b > 0 else "—(负)"
|
txt = f"{share * 100:.0f}%" if np.isfinite(share) else "—"
|
||||||
print(f"{s:<5}{len(g):>6}{d:>9.0f}{c:>9.0f}{t:>10.0f}{p90:>8.0f}"
|
print(f"{s:<5}{len(g):>6}{d:>9.0f}{c:>9.0f}{t:>10.0f}{p90:>8.0f}"
|
||||||
f"{drift:>12.2f}{b:>9.2f}{txt:>9}")
|
f"{drift:>12.2f}{b:>9.2f}{txt:>9}")
|
||||||
|
|
||||||
print("\n判读:")
|
print("\n判读(预算来自 lib/shadow_budget,2026 年口径):")
|
||||||
for s, (drift, b) in verdicts.items():
|
for s, (drift, b) in rows.items():
|
||||||
if b <= 0:
|
if not np.isfinite(b):
|
||||||
print(f" {s}: 余量本就为负,不作交易标的,仅作延迟参照")
|
print(f" {s}: 当前环境预算不足,不作交易标的,仅作延迟参照")
|
||||||
|
elif not np.isfinite(drift):
|
||||||
|
# 不特判的话 nan > b 是 False,会一路落到「尚有空间」说反话
|
||||||
|
print(f" {s}: 缺 1m 波动率缓存,折算不出漂移,无法判读")
|
||||||
elif drift > b:
|
elif drift > b:
|
||||||
print(f" {s}: 折算漂移 {drift:.2f}bp 已超余量 {b:.2f}bp —— 停下改方案")
|
print(f" {s}: 折算漂移 {drift:.2f}bp 已超预算 {b:.2f}bp —— 停下改方案")
|
||||||
elif drift > b * 0.6:
|
elif drift > b * 0.6:
|
||||||
print(f" {s}: 折算漂移 {drift:.2f}bp 吃掉余量 {b:.2f}bp 的六成以上,"
|
print(f" {s}: 折算漂移 {drift:.2f}bp 吃掉预算 {b:.2f}bp 的六成以上,"
|
||||||
f"需要压延迟或放弃")
|
f"需要压延迟或放弃")
|
||||||
else:
|
else:
|
||||||
print(f" {s}: 折算漂移 {drift:.2f}bp 对余量 {b:.2f}bp 尚有空间,继续收集")
|
print(f" {s}: 折算漂移 {drift:.2f}bp 对预算 {b:.2f}bp 尚有空间,继续收集")
|
||||||
|
|
||||||
|
|
||||||
|
def book_quality(sig: pd.DataFrame, drf: pd.DataFrame) -> None:
|
||||||
|
"""回查到的盘口比目标时刻晚多少。晚太多的已在采集侧丢弃,这里做复核。"""
|
||||||
|
frames = [d for d in (sig, drf) if not d.empty and "book_lag_ms" in d]
|
||||||
|
if not frames:
|
||||||
|
return
|
||||||
|
x = pd.concat([d["book_lag_ms"] for d in frames]).astype(float)
|
||||||
|
print("\n\n########## 二·五、盘口回查质量 ##########")
|
||||||
|
print(f" 回查 {len(x)} 次 · 中位 {x.median():.0f}ms · "
|
||||||
|
f"P90 {np.percentile(x, 90):.0f}ms · 最大 {x.max():.0f}ms")
|
||||||
|
print(f" 10Hz 采样下这个值应在 0~100ms。它是所有延迟点的同向偏置,"
|
||||||
|
f"不改变曲线形状,但要确认没有异常长尾")
|
||||||
|
|
||||||
|
|
||||||
|
def drift_split(sig: pd.DataFrame, drf: pd.DataFrame) -> None:
|
||||||
|
"""条件漂移 vs 无条件漂移。两者的差就是「系统性追价」的大小。"""
|
||||||
|
print("\n\n########## 三、条件漂移 vs 无条件漂移 ##########")
|
||||||
|
if drf.empty:
|
||||||
|
print(" 尚无逐根漂移数据(shadow_drift.csv 由本轮起才开始记)")
|
||||||
|
return
|
||||||
|
print(" 无条件 = 每根 K 线,方向未知故取 |漂移|;"
|
||||||
|
"条件 = 信号根按下单方向定号")
|
||||||
|
print(f"\n{'延迟':<8}{'无条件n':>9}{'无条件|漂移|':>14}"
|
||||||
|
f"{'条件n':>7}{'条件漂移':>10}{'追价差':>9}")
|
||||||
|
cond = sig[(sig["notional"] == sig["notional"].min())] if not sig.empty \
|
||||||
|
else sig
|
||||||
|
for lb in ORDER:
|
||||||
|
u = drf[drf["delay_label"] == lb]["drift_bp_long"].abs()
|
||||||
|
c = cond[cond["delay_label"] == lb]["drift_bp"] if not cond.empty \
|
||||||
|
else pd.Series(dtype=float)
|
||||||
|
if u.empty and c.empty:
|
||||||
|
continue
|
||||||
|
um = u.mean() if not u.empty else float("nan")
|
||||||
|
cm = c.mean() if not c.empty else float("nan")
|
||||||
|
print(f"{lb:<8}{len(u):>9}{um:>14.2f}{len(c):>7}{cm:>10.2f}"
|
||||||
|
f"{cm - um:>9.2f}")
|
||||||
|
if len(cond) and len(cond[cond["delay_label"] == "1.0s"]) < MIN_N:
|
||||||
|
print(f"\n 条件侧样本不足 {MIN_N},差值还读不出方向")
|
||||||
|
|
||||||
|
|
||||||
def slippage_curve(sig: pd.DataFrame) -> None:
|
def slippage_curve(sig: pd.DataFrame) -> None:
|
||||||
print("\n\n########## 二、滑点对延迟曲线 ##########")
|
print("\n\n########## 四、滑点对延迟曲线 ##########")
|
||||||
if sig.empty:
|
if sig.empty:
|
||||||
print(" 尚无信号样本")
|
print(" 尚无信号样本")
|
||||||
return
|
return
|
||||||
n_sig = sig.groupby(["sym", "kline_ts", "direction"]).ngroups
|
|
||||||
n_agree = sig[sig["h1_agree"] == 1].groupby(
|
|
||||||
["sym", "kline_ts", "direction"]).ngroups
|
|
||||||
print(f"信号总数 {n_sig}(其中 h1_agree=1 的 {n_agree} 个)\n")
|
|
||||||
|
|
||||||
order = ["0.5s", "1.0s", "2.0s", "5.0s", "actual"]
|
def n_of(df):
|
||||||
for scope, sub in (("全部信号", sig),
|
return df.groupby(["sym", "kline_ts", "direction"]).ngroups
|
||||||
("仅 h1_agree=1(主口径)", sig[sig["h1_agree"] == 1])):
|
|
||||||
|
has_flags = "pass_all" in sig.columns
|
||||||
|
if not has_flags:
|
||||||
|
print(" ⚠ 数据来自旧版采集(只有 h1_agree,无阶梯与 ATR 门控)。"
|
||||||
|
"这批不是我们要交易的那批信号,只能作管道验证,不能对预算判读。\n")
|
||||||
|
main_scope, main_name = sig[sig["h1_agree"] == 1], "仅 h1_agree=1(旧口径)"
|
||||||
|
else:
|
||||||
|
# depth_ok=0 是 25 档吃不满该仓位,均价按部分成交算会**低估**冲击
|
||||||
|
ok = (sig["pass_all"] == 1) & (sig["lag_ok"] == 1)
|
||||||
|
if "depth_ok" in sig.columns:
|
||||||
|
thin = int((sig["depth_ok"] == 0).sum())
|
||||||
|
ok &= sig["depth_ok"] == 1
|
||||||
|
if thin:
|
||||||
|
print(f" ({thin} 行深度吃不满,已排除;这些行会低估冲击)")
|
||||||
|
print(f"信号总数 {n_of(sig)} · 同向 {n_of(sig[sig['h1_agree'] == 1])}"
|
||||||
|
f" · 同向+阶梯 "
|
||||||
|
f"{n_of(sig[(sig['h1_agree'] == 1) & (sig['ladder_ok'] == 1)])}"
|
||||||
|
f" · 三项全过 {n_of(sig[sig['pass_all'] == 1])}"
|
||||||
|
f" · 再要求 lag 健康 {n_of(sig[ok])}")
|
||||||
|
print(f"(门控阈值 ATR ≥ {ATR_GATE_BP:.0f}bp,是费率的函数不是市场常数)\n")
|
||||||
|
main_scope = sig[ok]
|
||||||
|
main_name = "三项滤网全过 + lag 健康 + 深度吃满(主口径)"
|
||||||
|
|
||||||
|
scopes = [("全部信号(含不会下单的,仅作提前读数)", sig),
|
||||||
|
(main_name, main_scope)]
|
||||||
|
for scope, sub in scopes:
|
||||||
if sub.empty:
|
if sub.empty:
|
||||||
|
print(f"--- {scope} ---\n 尚无样本\n")
|
||||||
continue
|
continue
|
||||||
print(f"--- {scope} ---")
|
print(f"--- {scope} ---")
|
||||||
print(f"{'延迟':<8}{'仓位':>9}{'n':>5}{'滑点均值bp':>12}"
|
print(f"{'延迟':<8}{'仓位':>9}{'n':>5}{'滑点均值bp':>12}"
|
||||||
f"{'中位bp':>9}{'漂移bp':>9}{'价差bp':>9}{'冲击bp':>9}")
|
f"{'中位bp':>9}{'漂移bp':>9}{'价差bp':>9}{'冲击bp':>9}")
|
||||||
for lb in order:
|
for lb in ORDER:
|
||||||
g0 = sub[sub["delay_label"] == lb]
|
g0 = sub[sub["delay_label"] == lb]
|
||||||
if g0.empty:
|
|
||||||
continue
|
|
||||||
for nt in sorted(sub["notional"].unique()):
|
for nt in sorted(sub["notional"].unique()):
|
||||||
g = g0[g0["notional"] == nt]
|
g = g0[g0["notional"] == nt]
|
||||||
if g.empty:
|
if g.empty:
|
||||||
@@ -117,19 +229,36 @@ def slippage_curve(sig: pd.DataFrame) -> None:
|
|||||||
f"{g['impact_bp'].mean():>9.2f}")
|
f"{g['impact_bp'].mean():>9.2f}")
|
||||||
print()
|
print()
|
||||||
|
|
||||||
print("--- 分币种(仅 h1_agree=1,仓位 5000)---")
|
print("--- 分币种判读(主口径,仓位 5000)---")
|
||||||
m = sig[(sig["h1_agree"] == 1) & (sig["notional"] == 5000.0)]
|
m = main_scope[main_scope["notional"] == 5000.0] if not main_scope.empty \
|
||||||
|
else main_scope
|
||||||
if m.empty:
|
if m.empty:
|
||||||
print(" 尚无样本")
|
print(" 尚无样本")
|
||||||
return
|
return
|
||||||
print(f"{'币':<5}{'延迟':<8}{'n':>5}{'滑点均值bp':>12}{'余量bp':>9}")
|
print(f"{'币':<5}{'延迟':<8}{'n':>5}{'滑点中位bp':>12}{'预算bp':>9} 判读")
|
||||||
for s in SYMS:
|
for s in SYMS:
|
||||||
for lb in order:
|
for lb in ORDER:
|
||||||
g = m[(m["sym"] == s) & (m["delay_label"] == lb)]
|
g = m[(m["sym"] == s) & (m["delay_label"] == lb)]
|
||||||
if g.empty:
|
if g.empty:
|
||||||
continue
|
continue
|
||||||
print(f"{s:<5}{lb:<8}{len(g):>5}{g['slip_bp'].mean():>12.2f}"
|
med = float(g["slip_bp"].median())
|
||||||
f"{BUDGET_BP[s]:>9.2f}")
|
b = budget_of(s)
|
||||||
|
note = verdict(med, s) if len(g) >= MIN_N \
|
||||||
|
else f"n={len(g)} < {MIN_N},不下结论"
|
||||||
|
print(f"{s:<5}{lb:<8}{len(g):>5}{med:>12.2f}{b:>9.2f} {note}")
|
||||||
|
|
||||||
|
n_main = len(m[m["delay_label"] == "actual"])
|
||||||
|
if n_main < MIN_N:
|
||||||
|
print(f"\n ⚠ 主口径仅 {n_main} 笔。门控后约 4~6 笔/天/全部币种,"
|
||||||
|
f"滑点判据要 {MIN_N} 笔以上——**一周起步**。首日只能判延迟和管道。")
|
||||||
|
print(f" 单币样本薄时可先看组合口径:2026 预算 {BUDGET_PORTFOLIO_2026}bp")
|
||||||
|
|
||||||
|
|
||||||
|
def _load(name: str) -> pd.DataFrame:
|
||||||
|
f = OUT / name
|
||||||
|
if not f.exists() or f.stat().st_size == 0:
|
||||||
|
return pd.DataFrame()
|
||||||
|
return pd.read_csv(f)
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
def main() -> None:
|
||||||
@@ -137,20 +266,17 @@ def main() -> None:
|
|||||||
print("1m 收益标准差(bp/分钟,Bitget 实测):"
|
print("1m 收益标准差(bp/分钟,Bitget 实测):"
|
||||||
+ " ".join(f"{s} {v:.2f}" for s, v in vols.items()) + "\n")
|
+ " ".join(f"{s} {v:.2f}" for s, v in vols.items()) + "\n")
|
||||||
|
|
||||||
f_lat = OUT / "shadow_latency.csv"
|
lat = _load("shadow_latency.csv")
|
||||||
if f_lat.exists() and f_lat.stat().st_size > 0:
|
if lat.empty:
|
||||||
lat = pd.read_csv(f_lat)
|
|
||||||
if not lat.empty:
|
|
||||||
latency_gate(lat, vols)
|
|
||||||
else:
|
|
||||||
print("尚无延迟数据")
|
print("尚无延迟数据")
|
||||||
|
|
||||||
f_sig = OUT / "shadow_signals.csv"
|
|
||||||
if f_sig.exists() and f_sig.stat().st_size > 0:
|
|
||||||
sig = pd.read_csv(f_sig)
|
|
||||||
slippage_curve(sig)
|
|
||||||
else:
|
else:
|
||||||
print("\n尚无信号数据")
|
latency_gate(lat, vols)
|
||||||
|
lag_health(lat)
|
||||||
|
|
||||||
|
sig, drf = _load("shadow_signals.csv"), _load("shadow_drift.csv")
|
||||||
|
book_quality(sig, drf)
|
||||||
|
drift_split(sig, drf)
|
||||||
|
slippage_curve(sig)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
@@ -3,15 +3,30 @@
|
|||||||
单次调用约 0.26s 的纯 CPU,且 chanlun 是纯 Python 受 GIL 限制,放进
|
单次调用约 0.26s 的纯 CPU,且 chanlun 是纯 Python 受 GIL 限制,放进
|
||||||
Hummingbot 的 asyncio 循环里会把行情处理一起卡住,所以必须隔离到独立进程。
|
Hummingbot 的 asyncio 循环里会把行情处理一起卡住,所以必须隔离到独立进程。
|
||||||
|
|
||||||
口径与 [aggregate_robustness.run_once] 逐行对齐:同样的 build_htf_zones →
|
## 口径必须与预算同源,缺一项数就不可比
|
||||||
find_fast_bsp3 → attach_htf_context(h1) 链路,同样的 h1_agree 过滤。两边
|
|
||||||
必须一致,否则影子测出来的滑点没法和 3.9bp 预算对照。
|
|
||||||
|
|
||||||
与回测的唯一差别是这里只关心**最后一根已收盘 K 线**上有没有信号——
|
预算(`lib/shadow_budget.BUDGET_BP`)算在 step42 的这套滤网上,
|
||||||
实盘只能在当下下单,历史信号无意义。
|
本文件逐行对齐 `step42_exit_tp_1m.run_one`:
|
||||||
|
|
||||||
未过滤信号也一并返回:过滤后样本太稀,先用未过滤的当提前读数,
|
同向 h1_agree == 1
|
||||||
两者都记,靠 h1_agree 字段区分。
|
中枢阶梯 多头要求当前中枢整体高于前一个(zd > 前 zg),空头反之
|
||||||
|
ATR 门控 atr_pct ≥ ATR_GATE_BP(当前 8bp)
|
||||||
|
|
||||||
|
早先这里只有 h1_agree。缺阶梯与门控测的就不是我们要交易的那批信号,
|
||||||
|
而这一项改常数解决不了——必须改信号路径本身。
|
||||||
|
|
||||||
|
门控阈值是**费率的函数**不是市场常数(低 ATR 信号的毛质量反而最好,
|
||||||
|
断崖只在扣费后出现),换 VIP 档或换交易所要重扫,不要抄 8bp。
|
||||||
|
|
||||||
|
## 与回测的两点差别
|
||||||
|
|
||||||
|
其一,这里只关心**最后一根已收盘 K 线**上有没有信号——实盘只能在当下下单。
|
||||||
|
其二,`atr_pct` 的分母取次根开盘价,与 `exit_model.walk_exits` 一致,
|
||||||
|
所以调用方必须把次根开盘价传进来。
|
||||||
|
|
||||||
|
未通过滤网的信号也一并返回并打上标志:过滤后样本很稀(门控后 8 个币
|
||||||
|
合计约 38 笔/周),未过滤的可作提前读数。但**统计主口径只能用 pass_all**,
|
||||||
|
在我们根本不会下单的根上测滑点会把判据算宽。
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
@@ -23,15 +38,20 @@ for _v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
|
|||||||
os.environ.setdefault(_v, "1")
|
os.environ.setdefault(_v, "1")
|
||||||
|
|
||||||
|
|
||||||
def compute(df_l, df_h) -> dict:
|
def compute(df_l, df_h, entry_px: float | None = None) -> dict:
|
||||||
"""在 df_l 的最后一根上找信号。df_l/df_h 都只含已收盘 K 线。
|
"""在 df_l 的最后一根上找信号。df_l/df_h 都只含已收盘 K 线。
|
||||||
|
|
||||||
|
entry_px 是次根开盘价(回测 entry_delay=1 的成交价),用作 atr_pct 的
|
||||||
|
分母。取不到时退回用信号根收盘价,并在返回里标 atr_ref="close"。
|
||||||
|
|
||||||
返回 dict:
|
返回 dict:
|
||||||
last_idx 最后一根在 chanlun 处理后 dataframe 里的下标
|
last_idx 最后一根在 chanlun 处理后 dataframe 里的下标
|
||||||
n_bars 实际参与计算的根数
|
n_bars 实际参与计算的根数
|
||||||
hits 命中列表,每项 {direction, h1_agree}
|
atr_pct 信号根 ATR / 次根开盘价
|
||||||
|
hits 命中列表,每项含方向与三个滤网标志、pass_all
|
||||||
error 出错时的说明,正常为 None
|
error 出错时的说明,正常为 None
|
||||||
"""
|
"""
|
||||||
|
import numpy as np
|
||||||
import pandas as pd
|
import pandas as pd
|
||||||
|
|
||||||
try:
|
try:
|
||||||
@@ -40,45 +60,76 @@ def compute(df_l, df_h) -> dict:
|
|||||||
from lib.fx_signal import extract_fx_signals, signals_to_frame
|
from lib.fx_signal import extract_fx_signals, signals_to_frame
|
||||||
from lib.nested_bsp import attach_htf_context, htf_fx_timeline
|
from lib.nested_bsp import attach_htf_context, htf_fx_timeline
|
||||||
from lib.nested_level import build_htf_zones
|
from lib.nested_level import build_htf_zones
|
||||||
|
from lib.shadow_budget import ATR_GATE_BP
|
||||||
|
|
||||||
chan_l = TF_DF(df_l, 1, "1m")
|
chan_l = TF_DF(df_l, 1, "1m")
|
||||||
cdf = chan_l.dataframe
|
cdf = chan_l.dataframe
|
||||||
last = len(cdf) - 1
|
last = len(cdf) - 1
|
||||||
base = {"last_idx": last, "n_bars": int(len(df_l)), "hits": [],
|
base = {"last_idx": last, "n_bars": int(len(df_l)), "hits": [],
|
||||||
"error": None}
|
"atr_pct": None, "atr_ref": None, "error": None}
|
||||||
|
|
||||||
zones = build_htf_zones(cdf, "1m", chan=chan_l)
|
# ATR 门控。分母与 exit_model.walk_exits 一致,取次根开盘价。
|
||||||
|
# 放在任何早退之前——无信号的根也要记,才能在线看到门控的真实刷除率
|
||||||
|
atr = float(cdf["atr"].to_numpy(dtype=float)[last]) \
|
||||||
|
if "atr" in cdf.columns else float("nan")
|
||||||
|
ref = entry_px if (entry_px and np.isfinite(entry_px)) else \
|
||||||
|
float(cdf["close"].to_numpy(dtype=float)[last])
|
||||||
|
atr_pct = atr / ref if (np.isfinite(atr) and ref) else float("nan")
|
||||||
|
gate_ok = bool(np.isfinite(atr_pct) and atr_pct * 1e4 >= ATR_GATE_BP)
|
||||||
|
base["atr_pct"] = None if not np.isfinite(atr_pct) else float(atr_pct)
|
||||||
|
base["atr_ref"] = "next_open" if (entry_px and np.isfinite(entry_px)) \
|
||||||
|
else "close"
|
||||||
|
|
||||||
|
zones = build_htf_zones(cdf, "1m", chan=chan_l).reset_index(drop=True)
|
||||||
if zones.empty:
|
if zones.empty:
|
||||||
return base
|
return base
|
||||||
sig = find_fast_bsp3(cdf, zones.reset_index(drop=True))
|
|
||||||
|
# 中枢阶梯:当前中枢是否整体脱离前一个。与 step42 同一算法
|
||||||
|
z = zones.copy()
|
||||||
|
prev_zg, prev_zd = z["zg"].shift(), z["zd"].shift()
|
||||||
|
z["z_above"], z["z_below"] = z["zd"] > prev_zg, z["zg"] < prev_zd
|
||||||
|
z["zone_i"] = np.arange(len(z))
|
||||||
|
|
||||||
|
sig = find_fast_bsp3(cdf, zones)
|
||||||
if sig is None or sig.empty:
|
if sig is None or sig.empty:
|
||||||
return base
|
return base
|
||||||
|
sig = sig.merge(z[["zone_i", "z_above", "z_below"]],
|
||||||
|
on="zone_i", how="left")
|
||||||
|
|
||||||
# 只留落在最后一根上的信号,其余是历史,实盘下不了
|
# 5m 同向。算不出时 h1_agree 记 0,该信号自然不会通过 pass_all
|
||||||
cur = sig[sig["entry_idx"].astype(int) == last]
|
|
||||||
if cur.empty:
|
|
||||||
return base
|
|
||||||
|
|
||||||
# 5m 同向过滤:算得出就标 h1_agree,算不出就当未过滤照记
|
|
||||||
agree_map: dict[int, int] = {}
|
|
||||||
if df_h is not None and len(df_h) > 0:
|
if df_h is not None and len(df_h) > 0:
|
||||||
chan_h = TF_DF(df_h, 1, "5m")
|
chan_h = TF_DF(df_h, 1, "5m")
|
||||||
hdf = chan_h.dataframe
|
hdf = chan_h.dataframe
|
||||||
tl = htf_fx_timeline(
|
tl = htf_fx_timeline(
|
||||||
signals_to_frame(extract_fx_signals(chan_h, hdf)), hdf)
|
signals_to_frame(extract_fx_signals(chan_h, hdf)), hdf)
|
||||||
full = attach_htf_context(sig, cdf, tl, "h1")
|
sig = attach_htf_context(sig, cdf, tl, "h1")
|
||||||
f_cur = full[full["entry_idx"].astype(int) == last]
|
else:
|
||||||
for _, r in f_cur.iterrows():
|
sig["h1_agree"] = 0
|
||||||
agree_map[int(r["direction"])] = int(r.get("h1_agree", 0))
|
|
||||||
|
|
||||||
base["hits"] = [{"direction": int(r["direction"]),
|
cur = sig[sig["entry_idx"].astype(int) == last]
|
||||||
"h1_agree": agree_map.get(int(r["direction"]), 0)}
|
if cur.empty:
|
||||||
for _, r in cur.iterrows()]
|
return base
|
||||||
|
|
||||||
|
hits = []
|
||||||
|
for _, r in cur.iterrows():
|
||||||
|
d = int(r["direction"])
|
||||||
|
push = r["z_above"] if d == 1 else r["z_below"]
|
||||||
|
ladder_ok = bool(pd.notna(push) and bool(push))
|
||||||
|
# attach_htf_context 在入场时刻之前没有大级别分型时写 NaN。
|
||||||
|
# 不能写成 `int(x or 0)`——NaN 是真值,会走到 int(nan) 抛异常,
|
||||||
|
# 整根的信号就此丢掉,只留一行报错
|
||||||
|
raw = r.get("h1_agree", 0)
|
||||||
|
agree = int(raw) if pd.notna(raw) else 0
|
||||||
|
hits.append({"direction": d, "h1_agree": agree,
|
||||||
|
"ladder_ok": int(ladder_ok), "gate_ok": int(gate_ok),
|
||||||
|
"pass_all": int(agree == 1 and ladder_ok and gate_ok)})
|
||||||
|
base["hits"] = hits
|
||||||
return base
|
return base
|
||||||
except Exception as e: # 子进程里异常必须带回主进程,否则只见超时不见原因
|
except Exception as e: # 子进程里异常必须带回主进程,否则只见超时不见原因
|
||||||
import traceback
|
import traceback
|
||||||
return {"last_idx": -1, "n_bars": int(len(df_l)) if df_l is not None else 0,
|
return {"last_idx": -1, "n_bars": int(len(df_l)) if df_l is not None else 0,
|
||||||
"hits": [], "error": f"{type(e).__name__}: {e}",
|
"hits": [], "atr_pct": None, "atr_ref": None,
|
||||||
|
"error": f"{type(e).__name__}: {e}",
|
||||||
"traceback": traceback.format_exc()}
|
"traceback": traceback.format_exc()}
|
||||||
|
|
||||||
|
|
||||||
@@ -102,8 +153,8 @@ def _rebuild(rows) -> "object":
|
|||||||
|
|
||||||
|
|
||||||
def compute_packed(payload: tuple) -> dict:
|
def compute_packed(payload: tuple) -> dict:
|
||||||
"""ProcessPoolExecutor 的入口:收 (l_rows, h_rows) 两组数值行。"""
|
"""ProcessPoolExecutor 的入口:收 (l_rows, h_rows, entry_px)。"""
|
||||||
l_rows, h_rows = payload
|
l_rows, h_rows, entry_px = payload
|
||||||
df_l = _rebuild(l_rows)
|
df_l = _rebuild(l_rows)
|
||||||
df_h = _rebuild(h_rows) if h_rows else None
|
df_h = _rebuild(h_rows) if h_rows else None
|
||||||
return compute(df_l, df_h)
|
return compute(df_l, df_h, entry_px)
|
||||||
|
|||||||
@@ -0,0 +1,205 @@
|
|||||||
|
"""验证 live 信号路径与 step42 的批量过滤等价。
|
||||||
|
|
||||||
|
live 侧每根只看最后一根、且只喂 2000 根窗口;研究侧一次性跑全量。两者
|
||||||
|
用同一套滤网(同向 + 中枢阶梯 + ATR 门控),但**不保证逐笔一致**——
|
||||||
|
缠论结构依赖历史,2000 根窗口是 step39 定的命中率饱和点,不是无损截断。
|
||||||
|
|
||||||
|
每个信号根上比三种口径:
|
||||||
|
|
||||||
|
批量 全量历史 + 完整 5m。这是预算的来源,是基准
|
||||||
|
剔partial 窗口 2000 根 1m + 800 根**已收盘** 5m
|
||||||
|
含partial 窗口,5m 末尾保留那根**尚未收盘**的。这是 live 现行做法
|
||||||
|
|
||||||
|
## 结论:partial 根要保留,不能剔
|
||||||
|
|
||||||
|
live 的 `df_h[df_h["timestamp"] < kline_ts]` 里 kline_ts 是 1m 的收盘时刻,
|
||||||
|
而正在走的那根 5m 开盘更早,于是被保留下来——五根里有四根如此。乍看像是
|
||||||
|
「把未收盘的根当完整根用」的口径错误,实测**反过来**:
|
||||||
|
|
||||||
|
BTC 25/25、ETH 24/25、SOL 23/25 与批量一致(合计 96%)
|
||||||
|
剔掉则只有 21/25、21/25、18/25(合计 80%)
|
||||||
|
|
||||||
|
原因是批量口径里那根 5m 是存在的。缠论的包含处理与分型检测吃整条序列,
|
||||||
|
凭空少一根会把结构整体挪位;保留一根「开盘价正确、高低点尚不完整」的
|
||||||
|
近似根,比直接删掉更接近批量。
|
||||||
|
|
||||||
|
这也暴露了研究侧的一处残留:批量的 HTF 结构用到了那根 5m 的**最终**高低点,
|
||||||
|
而实盘在该时刻不可能知道。`htf_fx_timeline` 的 `confirm_ts += period` 只挡住了
|
||||||
|
分型**选取**上的未来函数,挡不住结构构建。live 用 partial 根逼近,落在 96%,
|
||||||
|
差的那 4% 是这条残留的下界,不是可以修掉的 bug。
|
||||||
|
|
||||||
|
.venv/bin/python research/live/verify_signal_path.py --symbol BTC
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import sys
|
||||||
|
import warnings
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
warnings.filterwarnings("ignore")
|
||||||
|
HERE = Path(__file__).resolve().parent
|
||||||
|
sys.path.insert(0, str(HERE))
|
||||||
|
sys.path.insert(0, str(HERE.parent))
|
||||||
|
sys.path.insert(0, str(HERE.parents[1]))
|
||||||
|
|
||||||
|
import numpy as np # noqa: E402
|
||||||
|
import pandas as pd # noqa: E402
|
||||||
|
|
||||||
|
WINDOW_L, WINDOW_H = 2000, 800
|
||||||
|
|
||||||
|
|
||||||
|
HTF_MS = 300_000
|
||||||
|
|
||||||
|
|
||||||
|
def partial_htf_bar(df_l: pd.DataFrame, kline_ts: int) -> dict | None:
|
||||||
|
"""用 1m 合成「此刻正在走的那根 5m」,复现 live 曾经喂进去的 partial 根。"""
|
||||||
|
bucket = kline_ts // HTF_MS * HTF_MS
|
||||||
|
if bucket >= kline_ts: # 正好落在 5m 边界,没有未收盘的根
|
||||||
|
return None
|
||||||
|
part = df_l[(df_l["timestamp"] >= bucket) & (df_l["timestamp"] < kline_ts)]
|
||||||
|
if part.empty:
|
||||||
|
return None
|
||||||
|
date = pd.to_datetime(bucket, unit="ms", utc=True) \
|
||||||
|
.tz_convert("Asia/Shanghai")
|
||||||
|
return {"timestamp": bucket, "date": date,
|
||||||
|
"open": float(part["open"].iloc[0]),
|
||||||
|
"high": float(part["high"].max()), "low": float(part["low"].min()),
|
||||||
|
"close": float(part["close"].iloc[-1]),
|
||||||
|
"volume": float(part["volume"].sum())}
|
||||||
|
|
||||||
|
|
||||||
|
def load(sym: str, tf: str, days: int) -> pd.DataFrame:
|
||||||
|
f = HERE / "cache" / f"bitget_{sym}_{tf}_{days}d.feather"
|
||||||
|
if not f.exists():
|
||||||
|
raise SystemExit(f"缺数据 {f}")
|
||||||
|
return pd.read_feather(f)
|
||||||
|
|
||||||
|
|
||||||
|
def batch_flags(df_l: pd.DataFrame, df_h: pd.DataFrame) -> pd.DataFrame:
|
||||||
|
"""step42_exit_tp_1m.run_one 的滤网,逐行照搬。"""
|
||||||
|
from chanlun import TF_DF
|
||||||
|
from lib.fast_bsp3 import find_fast_bsp3
|
||||||
|
from lib.fx_signal import extract_fx_signals, signals_to_frame
|
||||||
|
from lib.nested_bsp import attach_htf_context, htf_fx_timeline
|
||||||
|
from lib.nested_level import build_htf_zones
|
||||||
|
from lib.shadow_budget import ATR_GATE_BP
|
||||||
|
|
||||||
|
chan_l = TF_DF(df_l, 1, "1m")
|
||||||
|
cdf = chan_l.dataframe
|
||||||
|
zones = build_htf_zones(cdf, "1m", chan=chan_l).reset_index(drop=True)
|
||||||
|
z = zones.copy()
|
||||||
|
pg, pdn = z["zg"].shift(), z["zd"].shift()
|
||||||
|
z["z_above"], z["z_below"] = z["zd"] > pg, z["zg"] < pdn
|
||||||
|
z["zone_i"] = np.arange(len(z))
|
||||||
|
|
||||||
|
chan_h = TF_DF(df_h, 1, "5m")
|
||||||
|
hdf = chan_h.dataframe
|
||||||
|
tl = htf_fx_timeline(
|
||||||
|
signals_to_frame(extract_fx_signals(chan_h, hdf)), hdf)
|
||||||
|
|
||||||
|
sig = find_fast_bsp3(cdf, zones)
|
||||||
|
sig = sig.merge(z[["zone_i", "z_above", "z_below"]], on="zone_i", how="left")
|
||||||
|
sig = attach_htf_context(sig, cdf, tl, "h1")
|
||||||
|
|
||||||
|
d = sig["direction"].astype(int)
|
||||||
|
push = np.where(d == 1, sig["z_above"], sig["z_below"])
|
||||||
|
idx = sig["entry_idx"].astype(int).to_numpy()
|
||||||
|
entry = cdf["open"].to_numpy(float)[np.minimum(idx + 1, len(cdf) - 1)]
|
||||||
|
atr_pct = cdf["atr"].to_numpy(float)[idx] / entry
|
||||||
|
|
||||||
|
out = pd.DataFrame({
|
||||||
|
"entry_idx": idx,
|
||||||
|
"ts": cdf["timestamp"].to_numpy()[idx],
|
||||||
|
"direction": d.to_numpy(),
|
||||||
|
"h1_agree": sig["h1_agree"].fillna(0).astype(int).to_numpy(),
|
||||||
|
"ladder_ok": pd.Series(push).fillna(False).astype(int).to_numpy(),
|
||||||
|
"atr_bp": atr_pct * 1e4,
|
||||||
|
})
|
||||||
|
out["gate_ok"] = (out["atr_bp"] >= ATR_GATE_BP).astype(int)
|
||||||
|
out["pass_all"] = ((out["h1_agree"] == 1) & (out["ladder_ok"] == 1)
|
||||||
|
& (out["gate_ok"] == 1)).astype(int)
|
||||||
|
return out, cdf
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--symbol", default="BTC")
|
||||||
|
ap.add_argument("--days", type=int, default=30)
|
||||||
|
ap.add_argument("--checks", type=int, default=12)
|
||||||
|
a = ap.parse_args()
|
||||||
|
|
||||||
|
df_l, df_h = load(a.symbol, "1m", a.days), load(a.symbol, "5m", a.days)
|
||||||
|
print(f"[{a.symbol}] 1m {len(df_l)} 根 / 5m {len(df_h)} 根,跑批量滤网…",
|
||||||
|
flush=True)
|
||||||
|
batch, cdf = batch_flags(df_l, df_h)
|
||||||
|
n = len(batch)
|
||||||
|
print(f" 原始 B4/S4 {n} 个 · 同向 {int((batch.h1_agree == 1).sum())}"
|
||||||
|
f" · 同向+阶梯 "
|
||||||
|
f"{int(((batch.h1_agree == 1) & (batch.ladder_ok == 1)).sum())}"
|
||||||
|
f" · 三项全过 {int(batch.pass_all.sum())}")
|
||||||
|
print(f" ATR 中位 {batch.atr_bp.median():.2f}bp · "
|
||||||
|
f"门控刷掉 {(1 - batch.gate_ok.mean()) * 100:.1f}%\n")
|
||||||
|
|
||||||
|
# 挑最近的若干个信号根做窗口复现
|
||||||
|
from shadow_signal import compute
|
||||||
|
cand = batch[batch["entry_idx"] >= WINDOW_L].tail(a.checks)
|
||||||
|
if cand.empty:
|
||||||
|
raise SystemExit("窗口内没有可核对的信号")
|
||||||
|
|
||||||
|
def run_window(r, with_partial: bool):
|
||||||
|
i = int(r["entry_idx"])
|
||||||
|
kline_ts = int(r["ts"]) + 60_000 # 信号根的收盘时刻
|
||||||
|
wl = df_l[df_l["timestamp"] < kline_ts].tail(WINDOW_L)
|
||||||
|
wh = df_h[df_h["timestamp"] + HTF_MS <= kline_ts].tail(WINDOW_H)
|
||||||
|
if with_partial:
|
||||||
|
p = partial_htf_bar(df_l, kline_ts)
|
||||||
|
if p is not None:
|
||||||
|
wh = pd.concat([wh, pd.DataFrame([p])], ignore_index=True)
|
||||||
|
entry_px = float(cdf["open"].to_numpy(float)[min(i + 1, len(cdf) - 1)])
|
||||||
|
res = compute(wl.reset_index(drop=True), wh.reset_index(drop=True),
|
||||||
|
entry_px)
|
||||||
|
if res.get("error"):
|
||||||
|
raise SystemExit(f"compute 报错,测试本身有问题:{res['error']}\n"
|
||||||
|
f"{res.get('traceback', '')}")
|
||||||
|
return next((h for h in res["hits"]
|
||||||
|
if h["direction"] == int(r["direction"])), None)
|
||||||
|
|
||||||
|
def fmt(h):
|
||||||
|
if h is None:
|
||||||
|
return f"{'未复现':>18}"
|
||||||
|
return (f"{h['h1_agree']:>6}{h['ladder_ok']:>5}{h['gate_ok']:>5}")
|
||||||
|
|
||||||
|
print(f"{'K线时刻':<15}{'方向':>4}{' 批量':>18}{' 窗口':>18}"
|
||||||
|
f"{' 含未收盘':>18}{' 截断':>7}{'partial':>9}")
|
||||||
|
print(f"{'':<15}{'':>4}{'同向 阶梯 门控':>20}{'同向 阶梯 门控':>20}"
|
||||||
|
f"{'同向 阶梯 门控':>20}")
|
||||||
|
n_trunc = n_part = n_ok = n_bad = 0
|
||||||
|
for _, r in cand.iterrows():
|
||||||
|
h_ok = run_window(r, False)
|
||||||
|
h_bad = run_window(r, True)
|
||||||
|
t = pd.to_datetime(r["ts"], unit="ms", utc=True) \
|
||||||
|
.tz_convert("Asia/Shanghai").strftime("%m-%d %H:%M")
|
||||||
|
ref = (int(r.h1_agree), int(r.ladder_ok), int(r.gate_ok))
|
||||||
|
got = None if h_ok is None else (h_ok["h1_agree"], h_ok["ladder_ok"],
|
||||||
|
h_ok["gate_ok"])
|
||||||
|
bad = None if h_bad is None else (h_bad["h1_agree"], h_bad["ladder_ok"],
|
||||||
|
h_bad["gate_ok"])
|
||||||
|
n_trunc += got != ref
|
||||||
|
n_part += bad != got
|
||||||
|
n_ok += got == ref
|
||||||
|
n_bad += bad == ref
|
||||||
|
print(f"{t:<15}{int(r.direction):>+4}"
|
||||||
|
f"{ref[0]:>6}{ref[1]:>5}{ref[2]:>5}"
|
||||||
|
f"{fmt(h_ok):>18}{fmt(h_bad):>18}"
|
||||||
|
f"{'' if got == ref else '差':>7}"
|
||||||
|
f"{'' if bad == got else '差':>9}")
|
||||||
|
n = len(cand)
|
||||||
|
print(f"\n 与批量(预算口径)一致:")
|
||||||
|
print(f" 剔掉未收盘 5m 根 {n_ok}/{n}")
|
||||||
|
print(f" 保留未收盘 5m 根 {n_bad}/{n} ← live 现行做法")
|
||||||
|
print(f" 两种窗口口径互不相同 {n_part}/{n}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
Reference in New Issue
Block a user