影子测量改用框架吃单原语，并补齐容量与 maker 成交率两项测算
吃单查询换成 Hummingbot 的 OrderBook.get_vwap_for_volume：手写的 walk_book 返回的是按计价币吃单的加权均价，但框架的 get_price_for_quote_volume 返回 边际价、get_vwap_for_volume 收基础币量，两者语义不同。改为按基础币下单 （真实委托与 PositionExecutor.amount 均是基础币计价），深度不足由 query_volume/result_volume 判定，框架此时返回 nan 而非一个看似正常的 部分成交均价。 落盘完整盘口（双边 50 档）。此前只记三个固定名义额的成交价，这批数据的 寿命就等于那几个档位的寿命；存完整深度后任意资金量级的冲击都能离线重算。 仓位档同时从 1k/5k/20k 提到十万量级，此前低估真实仓位约两个数量级。 订阅成交流，按根按价位聚合。买卖分开存——多头在目标位挂卖出靠主动买盘 成交，混在一起会把成交率高估约一倍。BTC 每根总成交额中位与 210 天历史 的 volume×close 差 0.3%，可确认采集完整。 新增两项测算： - 冲击不是绑定约束。32 万仓位单边冲击 0.19~2.39bp，对 8.58~20.64bp 的 预算只占 1.6~14.2%，冲击反推的资金上限 100~500 万。 - maker 成交率才是。止盈位被首次触及时，限价在该根价格区间中的位置 中位 k=0.28（63.9 万次触及，三币一致）；合并每根成交额后，32 万仓位 的全额成交率仅 30.1%/15.6%/1.5%。要 80% 全额成交，仓位须 ≤ 4.7 万 /1.4 万/0.26 万——比冲击反推的上限低 40~370 倍。 回测把这些止盈按「全额成交在目标价」计，故预算所依据的收益流本身需重估。 Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
jack
+458
-85
@@ -10,8 +10,25 @@
|
||||
为什么不能用 paper trade 的成交:那是 Hummingbot 自己的撮合模型模拟的,
|
||||
测出来是模型行为不是市场行为。
|
||||
|
||||
口径对齐 aggregate_robustness.py:回测假设成交在**信号次根的开盘价**
|
||||
口径对齐 step42_exit_tp_1m.py:回测假设成交在**信号次根的开盘价**
|
||||
(entry_delay=1),所以基准价就是换根后新一根的 open。滑点为正表示比回测差。
|
||||
滤网(同向 + 中枢阶梯 + ATR 门控)在 shadow_signal.py 里,必须与预算同源。
|
||||
|
||||
### 统计口径三条硬要求
|
||||
|
||||
1. **主口径只用 pass_all 的信号根**。未过滤的照记但只作提前读数——
|
||||
在我们根本不会下单的根上测滑点会把判据算宽
|
||||
2. **条件漂移与无条件漂移分开报**。所以每根 K 线都记一份漂移
|
||||
(shadow_drift.csv),不只信号根。两者的差就是「系统性追价」的大小
|
||||
3. **出场腿按 maker/taker 分开**。止盈挂限价不吃滑点,把那 60% 混进
|
||||
平均值会低估真实成本。出场腿属持仓管理,尚未实现
|
||||
|
||||
### lag 探针:超阈值要停开仓,不能只打日志
|
||||
|
||||
补丁只防得住「上游代码变了」,防不住 Bitget 再改一次消息格式。每根记
|
||||
本地接收 − K 线收盘,近 30 根取中位数,超 800ms 即判该币不健康。
|
||||
要停开仓是因为这种退化是**经济性且静默**的:不崩不报错,只让收益慢慢
|
||||
变差,几周后才从统计里看得出来。影子期不下单,故落到 lag_ok 字段上。
|
||||
|
||||
### 盘口滚动缓冲把延迟变成自变量
|
||||
|
||||
@@ -38,22 +55,32 @@ from __future__ import annotations
|
||||
import argparse
|
||||
import asyncio
|
||||
import csv
|
||||
import gzip
|
||||
import json
|
||||
import math
|
||||
import time
|
||||
from collections import deque
|
||||
from concurrent.futures import ProcessPoolExecutor
|
||||
from concurrent.futures.process import BrokenProcessPool
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from lib.shadow_budget import LAG_ALARM_MS, LAG_WINDOW, lag_healthy
|
||||
|
||||
SYMS = ("BTC", "ETH", "SOL")
|
||||
# 多存一根:deque 尾部是尚未收盘的当前根,剔除后正好剩 step39 定下的窗口
|
||||
LTF_BARS, HTF_BARS = 2001, 801 # 有效窗口 2000 / 800,命中率在此饱和
|
||||
BOOK_HZ = 10 # 盘口采样 10Hz
|
||||
BOOK_KEEP_S = 30 # 缓冲保留 30 秒,够回查到 +5s
|
||||
BOOK_DEPTH = 25
|
||||
BOOK_TOL_MS = 250 # 回查容差:10Hz 正常 ≤100ms,留些余量
|
||||
BOOK_DEPTH = 50 # 双边各 50 档，实测能撑 78 万~261 万美元
|
||||
DELAYS_S = (0.5, 1.0, 2.0, 5.0) # 回查点
|
||||
NOTIONALS = (1_000.0, 5_000.0, 20_000.0)
|
||||
# 便利视图用的仓位档。**真正的答案在 shadow_books.jsonl.gz 里**——完整盘口
|
||||
# 落了盘，任意资金量级的冲击都能离线算，换个规模不必重测。这里的档位只是
|
||||
# 为了让 CSV 直接可读，覆盖到按 2ATR 止损反推的十万量级真实仓位
|
||||
NOTIONALS = (50_000.0, 100_000.0, 320_000.0, 530_000.0)
|
||||
NUM_COLS = ["timestamp", "open", "high", "low", "close", "volume"]
|
||||
|
||||
|
||||
@@ -62,6 +89,119 @@ def out_dir() -> Path:
|
||||
return p if p.is_dir() else Path(__file__).resolve().parents[1] / "out"
|
||||
|
||||
|
||||
def _writer(path: Path, cols: list[str]):
|
||||
"""追加模式打开；表头对不上就先把旧文件归档。
|
||||
|
||||
不校验的话，列一改，DictWriter 会按新顺序把行写到旧表头下面——
|
||||
读出来整片错位，而且没有任何报错。长跑靠追加续命，这个校验是必需的。
|
||||
"""
|
||||
if path.exists() and path.stat().st_size > 0:
|
||||
with path.open(newline="") as fh:
|
||||
old = next(csv.reader(fh), [])
|
||||
if old != cols:
|
||||
arch = path.parent / "archive"
|
||||
arch.mkdir(exist_ok=True)
|
||||
dst = arch / f"{path.stem}_{time.strftime('%Y%m%d_%H%M%S')}.csv"
|
||||
path.rename(dst)
|
||||
print(f" [CSV] {path.name} 表头已变，旧数据归档为 {dst.name}",
|
||||
flush=True)
|
||||
fresh = not path.exists() or path.stat().st_size == 0
|
||||
f = path.open("a", newline="")
|
||||
w = csv.DictWriter(f, fieldnames=cols)
|
||||
if fresh:
|
||||
w.writeheader()
|
||||
return f, w
|
||||
|
||||
|
||||
class BookLog:
|
||||
"""把完整盘口快照落成 gzip JSONL。
|
||||
|
||||
只记「某几个仓位档的成交价」的话，这批数据的寿命就等于那几个档位的寿命：
|
||||
换一次资金规模就得重跑一周。存完整深度后，任意仓位的冲击都能离线重算，
|
||||
一次采集回答所有资金量级的问题——包括容量上限那个必须现在就算、
|
||||
不该等实盘暴露的数。
|
||||
|
||||
用 gzip 追加（多个 gzip 成员首尾相接仍可正常解压），进程被杀也只丢最后
|
||||
一个缓冲块，不会毁掉整个文件。
|
||||
"""
|
||||
|
||||
def __init__(self, path: Path) -> None:
|
||||
self.path = path
|
||||
self.fh = gzip.open(path, "at", encoding="utf-8")
|
||||
self.n = 0
|
||||
|
||||
def write(self, sym: str, kline_ts: int, label: str, delay_ms: int,
|
||||
target: int, book_ts: int, bids: np.ndarray,
|
||||
asks: np.ndarray) -> None:
|
||||
# 只留价与量两列，update_id 对离线分析没用。round 到 10 位避免
|
||||
# float repr 把文件撑大一倍
|
||||
rec = {"sym": sym, "kline_ts": kline_ts, "label": label,
|
||||
"delay_ms": delay_ms, "target": target, "book_ts": book_ts,
|
||||
"bids": [[round(float(p), 10), round(float(a), 10)]
|
||||
for p, a, *_ in bids],
|
||||
"asks": [[round(float(p), 10), round(float(a), 10)]
|
||||
for p, a, *_ in asks]}
|
||||
self.fh.write(json.dumps(rec, separators=(",", ":")) + "\n")
|
||||
self.n += 1
|
||||
|
||||
def flush(self) -> None:
|
||||
self.fh.flush()
|
||||
|
||||
def close(self) -> None:
|
||||
try:
|
||||
self.fh.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
class TapeLog:
|
||||
"""按 K 线、按价位聚合成交量，用来判 maker 腿能不能全额成交。
|
||||
|
||||
回测假设 3ATR 和 8ATR 的限价单全额成交。十万量级挂在那里，全成交还是
|
||||
部分成交是完全不同的事——部分成交会把分批出场的收益结构改掉，而这个
|
||||
问题盘口深度回答不了：深度说的是「现在有多少人挂着」，成交率问的是
|
||||
「之后有多少人打过来」。只有成交流能回答。
|
||||
|
||||
**买卖必须分开存。** 多头在 3ATR 挂卖出止盈，成交靠的是主动**买盘**
|
||||
打上来；把双边成交量合在一起会把成交率高估约一倍。
|
||||
|
||||
聚合到「根 × 价位」而不是逐笔：判据是「本根内有多少量在 ≥ 限价处成交」，
|
||||
逐笔的时序对这个判据没有增量信息，而聚合能把体量压下两个数量级。
|
||||
"""
|
||||
|
||||
def __init__(self, path: Path) -> None:
|
||||
self.fh = gzip.open(path, "at", encoding="utf-8")
|
||||
# sym -> side('b'/'s') -> price -> 累计基础币量
|
||||
self.acc: dict[str, dict[str, dict[float, float]]] = {}
|
||||
self.n_trades = 0
|
||||
|
||||
def add(self, sym: str, is_buy: bool, price: float, amount: float) -> None:
|
||||
d = self.acc.setdefault(sym, {"b": {}, "s": {}})
|
||||
side = d["b"] if is_buy else d["s"]
|
||||
side[price] = side.get(price, 0.0) + amount
|
||||
self.n_trades += 1
|
||||
|
||||
def flush_bar(self, sym: str, bar_ts: int) -> None:
|
||||
"""一根走完就把这根的聚合结果落盘并清空。"""
|
||||
d = self.acc.get(sym)
|
||||
if not d or (not d["b"] and not d["s"]):
|
||||
return
|
||||
rec = {"sym": sym, "bar_ts": bar_ts,
|
||||
"buys": {f"{p:.10g}": round(v, 10)
|
||||
for p, v in sorted(d["b"].items())},
|
||||
"sells": {f"{p:.10g}": round(v, 10)
|
||||
for p, v in sorted(d["s"].items())}}
|
||||
self.fh.write(json.dumps(rec, separators=(",", ":")) + "\n")
|
||||
self.fh.flush()
|
||||
self.acc[sym] = {"b": {}, "s": {}}
|
||||
|
||||
def close(self) -> None:
|
||||
try:
|
||||
self.fh.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def hb_to_research(cdf: pd.DataFrame) -> pd.DataFrame:
|
||||
"""Hummingbot 的 candles_df 转成 research/lib/data.py 的列结构。
|
||||
|
||||
@@ -77,38 +217,32 @@ def hb_to_research(cdf: pd.DataFrame) -> pd.DataFrame:
|
||||
.sort_values("timestamp").reset_index(drop=True)
|
||||
|
||||
|
||||
def walk_book(levels: list[tuple[float, float]], notional: float
|
||||
) -> tuple[float, float]:
|
||||
"""吃单到 notional(计价币)为止,返回 (加权成交价, 实际吃到的额度)。
|
||||
def book_from(bids: np.ndarray, asks: np.ndarray):
|
||||
"""用缓冲里的快照临时搭一个 OrderBook，以便调用框架自带的吃单查询。
|
||||
|
||||
深度不足时返回吃到的部分,由调用方按 filled < notional 判断是否可信。
|
||||
自己手写吃单曾经踩过两个坑，框架版都没有：`get_vwap_for_volume` 返回的
|
||||
是真加权均价（市价单的实际成交价），而 `get_price_for_quote_volume` 返回
|
||||
的是**边际价**，用后者会高估冲击；深度不足时框架返回 nan 而不是一个
|
||||
「看起来很正常」的部分成交均价，靠 query_volume/result_volume 判断。
|
||||
"""
|
||||
if not levels:
|
||||
return float("nan"), 0.0
|
||||
got = 0.0
|
||||
cost = 0.0
|
||||
qty = 0.0
|
||||
for px, sz in levels:
|
||||
avail = px * sz
|
||||
take = min(avail, notional - got)
|
||||
if take <= 0:
|
||||
break
|
||||
q = take / px
|
||||
cost += q * px
|
||||
qty += q
|
||||
got += take
|
||||
if got >= notional - 1e-9:
|
||||
break
|
||||
if qty <= 0:
|
||||
return float("nan"), 0.0
|
||||
return cost / qty, got
|
||||
from hummingbot.core.data_type.order_book import OrderBook
|
||||
ob = OrderBook()
|
||||
ob.apply_numpy_snapshot(bids, asks)
|
||||
return ob
|
||||
|
||||
|
||||
class BookBuffer:
|
||||
"""每币一份滚动盘口。按时间戳回查,取第一个不早于目标时刻的快照。"""
|
||||
"""每币一份滚动盘口。按时间戳回查,取第一个不早于目标时刻的快照。
|
||||
|
||||
回查必须有容差上界。10Hz 下正常落在目标后 100ms 内（所有延迟点同向
|
||||
偏约 +50ms，不影响曲线形状），但采样一旦卡顿，标着「0.5s」的那行可能
|
||||
用的是 +3s 的盘口——数据看不出异常，判读却已经错了。超容差宁可丢弃，
|
||||
并且把快照实际时刻写进 CSV，让这件事事后可查。
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.buf: dict[str, deque] = {s: deque() for s in SYMS}
|
||||
self.n_stale = 0 # 因超容差被丢弃的回查次数
|
||||
|
||||
def push(self, sym: str, t_ms: int, bids: list, asks: list) -> None:
|
||||
d = self.buf[sym]
|
||||
@@ -118,12 +252,13 @@ class BookBuffer:
|
||||
d.popleft()
|
||||
|
||||
def at(self, sym: str, t_ms: int) -> tuple | None:
|
||||
best = None
|
||||
for snap in self.buf[sym]:
|
||||
if snap[0] >= t_ms:
|
||||
best = snap
|
||||
break
|
||||
return best
|
||||
if snap[0] - t_ms > BOOK_TOL_MS:
|
||||
self.n_stale += 1
|
||||
return None
|
||||
return snap
|
||||
return None
|
||||
|
||||
|
||||
class Shadow:
|
||||
@@ -136,30 +271,84 @@ class Shadow:
|
||||
self.feeds_h: dict = {}
|
||||
self.connector = None
|
||||
self.stop = asyncio.Event()
|
||||
# 持有 fire-and-forget 任务的强引用。只 create_task 不留引用的话，
|
||||
# 任务可能在完成前被 GC 掉，asyncio 官方文档明确警告过这一点
|
||||
self._tasks: set = set()
|
||||
self.n_broken = 0
|
||||
self._hb_last_bars = 0
|
||||
# 成交监听：已挂上的币，以及必须持有的 forwarder 强引用
|
||||
# （PubSub 只存弱引用，不持有的话监听会被 GC 静默摘掉）
|
||||
self._hooked: set[str] = set()
|
||||
self._trade_fwd: dict = {}
|
||||
self.n_signal = 0
|
||||
self.n_pass = 0
|
||||
self.n_bars = 0
|
||||
# lag 探针的滚动窗口，逐币独立：一个币的行情退化不该连累其他币
|
||||
self.lag_hist: dict[str, deque] = {
|
||||
s: deque(maxlen=LAG_WINDOW) for s in SYMS}
|
||||
self.lag_ok: dict[str, bool] = {s: True for s in SYMS}
|
||||
d = out_dir()
|
||||
# 追加模式:长跑期间若重启,已收集的样本不该被清掉
|
||||
p_sig = d / "shadow_signals.csv"
|
||||
new_sig = not p_sig.exists() or p_sig.stat().st_size == 0
|
||||
self.f_sig = p_sig.open("a", newline="")
|
||||
self.w_sig = csv.DictWriter(self.f_sig, fieldnames=[
|
||||
"sym", "kline_ts", "direction", "h1_agree",
|
||||
self.f_sig, self.w_sig = _writer(d / "shadow_signals.csv", [
|
||||
"sym", "kline_ts", "direction",
|
||||
"h1_agree", "ladder_ok", "gate_ok", "pass_all", "lag_ok",
|
||||
"atr_pct", "atr_bp",
|
||||
"t_close_ms", "t_data_ms", "t_signal_ms",
|
||||
"lag_data_ms", "lag_signal_ms",
|
||||
"delay_label", "delay_ms", "notional",
|
||||
"baseline_px", "mid", "best_px", "fill_px", "filled",
|
||||
"delay_label", "delay_ms", "book_ts", "book_lag_ms",
|
||||
"notional", "base_amt",
|
||||
"baseline_px", "mid", "best_px", "fill_px", "filled", "depth_ok",
|
||||
"slip_bp", "drift_bp", "spread_bp", "impact_bp"])
|
||||
if new_sig:
|
||||
self.w_sig.writeheader()
|
||||
p_lat = d / "shadow_latency.csv"
|
||||
new_lat = not p_lat.exists() or p_lat.stat().st_size == 0
|
||||
self.f_lat = p_lat.open("a", newline="")
|
||||
self.w_lat = csv.DictWriter(self.f_lat, fieldnames=[
|
||||
self.f_lat, self.w_lat = _writer(d / "shadow_latency.csv", [
|
||||
"sym", "kline_ts", "t_close_ms", "t_data_ms", "t_signal_ms",
|
||||
"lag_data_ms", "lag_signal_ms", "compute_ms", "n_bars", "n_hits"])
|
||||
if new_lat:
|
||||
self.w_lat.writeheader()
|
||||
"lag_data_ms", "lag_signal_ms", "compute_ms", "n_bars", "n_hits",
|
||||
"n_pass", "atr_bp", "lag_med_ms", "lag_ok"])
|
||||
# 无条件漂移：每根都记，用来和信号根上的条件漂移对照
|
||||
self.f_drf, self.w_drf = _writer(d / "shadow_drift.csv", [
|
||||
"sym", "kline_ts", "delay_label", "delay_ms",
|
||||
"book_ts", "book_lag_ms", "baseline_px", "mid", "drift_bp_long"])
|
||||
# 完整深度。挂在无条件漂移那条路径上，所以每根 K 线的四个固定延迟点
|
||||
# 都有一份，信号根上再补一份 actual 点
|
||||
self.blog = BookLog(d / "shadow_books.jsonl.gz")
|
||||
self.tape = TapeLog(d / "shadow_tape.jsonl.gz")
|
||||
|
||||
def _spawn(self, coro, what: str) -> None:
|
||||
"""起一个后台任务，但异常要吼出来。
|
||||
|
||||
裸 create_task 的异常只在对象被 GC 时才由 asyncio 打一句
|
||||
「Task exception was never retrieved」，很容易整晚没人发现。
|
||||
这套东西最怕的就是不崩不报错的静默退化。
|
||||
"""
|
||||
async def guard():
|
||||
try:
|
||||
await coro
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as e:
|
||||
import traceback
|
||||
print(f" [异常] {what}: {type(e).__name__}: {e}", flush=True)
|
||||
traceback.print_exc()
|
||||
|
||||
t = asyncio.create_task(guard())
|
||||
self._tasks.add(t)
|
||||
t.add_done_callback(self._tasks.discard)
|
||||
|
||||
def _restart_pool(self) -> None:
|
||||
"""进程池坏了之后重建。
|
||||
|
||||
用 spawn 而非 fork：此刻进程里已经有活跃的 WS 连接，fork 会把连接
|
||||
状态一起复制进子进程。spawn 启动慢几秒，但只在故障时走这条路。
|
||||
"""
|
||||
import multiprocessing
|
||||
self.n_broken += 1
|
||||
try:
|
||||
self.pool.shutdown(wait=False, cancel_futures=True)
|
||||
except Exception:
|
||||
pass
|
||||
self.pool = ProcessPoolExecutor(
|
||||
max_workers=self.workers,
|
||||
mp_context=multiprocessing.get_context("spawn"))
|
||||
print(f" [进程池] 已重建（第 {self.n_broken} 次）", flush=True)
|
||||
|
||||
# ---------- 启动 ----------
|
||||
|
||||
@@ -167,7 +356,11 @@ class Shadow:
|
||||
from hummingbot.connector.derivative.bitget_perpetual.bitget_perpetual_derivative import (
|
||||
BitgetPerpetualDerivative,
|
||||
)
|
||||
from patched_candles import PatchedBitgetPerpetualCandles
|
||||
from patched_candles import (PatchedBitgetPerpetualCandles,
|
||||
assert_patch_effective)
|
||||
|
||||
# 覆盖失效是静默的（悄悄退回慢 1.06 秒，不报错），所以在启动就验一次
|
||||
assert_patch_effective()
|
||||
|
||||
for s in SYMS:
|
||||
self.feeds_l[s] = PatchedBitgetPerpetualCandles(
|
||||
@@ -187,6 +380,7 @@ class Shadow:
|
||||
trading_required=False)
|
||||
await self.connector.start_network()
|
||||
print(" 连接器已启动,等盘口与历史回填", flush=True)
|
||||
self._hook_trades()
|
||||
|
||||
t0 = time.time()
|
||||
while time.time() - t0 < 600:
|
||||
@@ -200,6 +394,39 @@ class Shadow:
|
||||
f"1m {[len(self.feeds_l[s]._candles) for s in SYMS]} 根 · "
|
||||
f"5m {[len(self.feeds_h[s]._candles) for s in SYMS]} 根", flush=True)
|
||||
|
||||
def _hook_trades(self) -> None:
|
||||
"""给每个盘口挂成交监听。
|
||||
|
||||
盘口对象可能还没建好（订阅是异步的），所以挂不上的先记下来，由
|
||||
watch_bars 那圈重试；一直挂不上会在心跳里显示成交笔数为 0。
|
||||
"""
|
||||
from hummingbot.core.event.event_forwarder import EventForwarder
|
||||
from hummingbot.core.event.events import OrderBookEvent
|
||||
from hummingbot.core.data_type.common import TradeType
|
||||
|
||||
def make(sym: str):
|
||||
def cb(ev) -> None:
|
||||
self.tape.add(sym, ev.type == TradeType.BUY,
|
||||
float(ev.price), float(ev.amount))
|
||||
return EventForwarder(cb)
|
||||
|
||||
for s in SYMS:
|
||||
if s in self._hooked: # 重复挂会让同一笔成交被记两次
|
||||
continue
|
||||
try:
|
||||
ob = self.connector.get_order_book(f"{s}-USDT")
|
||||
except Exception:
|
||||
ob = None
|
||||
if ob is None:
|
||||
continue
|
||||
fwd = make(s)
|
||||
ob.add_listener(OrderBookEvent.TradeEvent, fwd)
|
||||
self._trade_fwd[s] = fwd
|
||||
self._hooked.add(s)
|
||||
miss = [s for s in SYMS if s not in self._hooked]
|
||||
print(f" 成交流已挂 {sorted(self._hooked)}"
|
||||
+ (f"，待重试 {miss}" if miss else ""), flush=True)
|
||||
|
||||
def _snapshot(self, sym: str):
|
||||
try:
|
||||
ob = self.connector.get_order_book(f"{sym}-USDT")
|
||||
@@ -207,25 +434,41 @@ class Shadow:
|
||||
return None
|
||||
if ob is None:
|
||||
return None
|
||||
bids = [(float(r.price), float(r.amount))
|
||||
for r, _ in zip(ob.bid_entries(), range(BOOK_DEPTH))]
|
||||
asks = [(float(r.price), float(r.amount))
|
||||
for r, _ in zip(ob.ask_entries(), range(BOOK_DEPTH))]
|
||||
if not bids or not asks:
|
||||
# 存成 apply_numpy_snapshot 要的 [价, 量, update_id] 三列，
|
||||
# 回查时才能直接搭 OrderBook 调框架的吃单查询
|
||||
bids = np.array([(float(r.price), float(r.amount), i)
|
||||
for i, (r, _) in enumerate(
|
||||
zip(ob.bid_entries(), range(BOOK_DEPTH)))])
|
||||
asks = np.array([(float(r.price), float(r.amount), i)
|
||||
for i, (r, _) in enumerate(
|
||||
zip(ob.ask_entries(), range(BOOK_DEPTH)))])
|
||||
if not len(bids) or not len(asks):
|
||||
return None
|
||||
return bids, asks
|
||||
|
||||
# ---------- 三个循环 ----------
|
||||
|
||||
async def sample_books(self) -> None:
|
||||
"""按截止时刻补睡，且对齐到墙钟 100ms 网格。
|
||||
|
||||
补睡是因为「干完活再睡固定时长」的实际周期是 100ms 加采样耗时，
|
||||
名义 10Hz 到不了 10Hz。
|
||||
|
||||
对齐是因为回查目标都是 `kline_ts + n×500ms`，而 kline_ts 是整分钟，
|
||||
所以目标必然落在墙钟 100ms 的整数倍上。采样相位若随启动时刻漂移，
|
||||
每个回查点就会固定晚半个采样周期（实测 52ms）——四个固定延迟点
|
||||
同向偏置，虽不改曲线形状，但白白多算了 50ms 的漂移。
|
||||
"""
|
||||
period = 1.0 / BOOK_HZ
|
||||
nxt = math.ceil(time.time() / period) * period
|
||||
while not self.stop.is_set():
|
||||
t = int(time.time() * 1000)
|
||||
for s in SYMS:
|
||||
snap = self._snapshot(s)
|
||||
if snap:
|
||||
self.books.push(s, t, snap[0], snap[1])
|
||||
await asyncio.sleep(period)
|
||||
nxt += period
|
||||
await asyncio.sleep(max(0.0, nxt - time.time()))
|
||||
|
||||
async def watch_bars(self) -> None:
|
||||
last = {s: (int(self.feeds_l[s]._candles[-1][0])
|
||||
@@ -239,7 +482,11 @@ class Shadow:
|
||||
if last[s] is not None and newest > last[s]:
|
||||
t_data = int(time.time() * 1000)
|
||||
kts = newest * 1000 if newest < 1e12 else newest
|
||||
asyncio.create_task(self.on_bar(s, kts, t_data))
|
||||
# 刚收盘那根的成交聚合先落盘，再算信号
|
||||
self.tape.flush_bar(s, kts)
|
||||
if len(self._hooked) < len(SYMS):
|
||||
self._hook_trades() # 换根时才重试，避免重复挂
|
||||
self._spawn(self.on_bar(s, kts, t_data), f"on_bar {s}")
|
||||
last[s] = newest
|
||||
await asyncio.sleep(0.01)
|
||||
|
||||
@@ -252,15 +499,28 @@ class Shadow:
|
||||
df_h = df_h[df_h["timestamp"] < kline_ts]
|
||||
baseline = self._new_bar_open(sym, kline_ts)
|
||||
|
||||
lag_med, lag_ok = self._probe_lag(sym, t_data - kline_ts)
|
||||
|
||||
t0 = time.perf_counter()
|
||||
payload = (df_l[NUM_COLS].values.tolist(),
|
||||
df_h[NUM_COLS].values.tolist())
|
||||
df_h[NUM_COLS].values.tolist(), baseline)
|
||||
loop = asyncio.get_running_loop()
|
||||
from shadow_signal import compute_packed
|
||||
res = await loop.run_in_executor(self.pool, compute_packed, payload)
|
||||
try:
|
||||
res = await loop.run_in_executor(self.pool, compute_packed, payload)
|
||||
except BrokenProcessPool as e:
|
||||
# 不重建的话，之后每一根都会走到这里，采集静默停摆到跑完为止
|
||||
print(f" [{sym}] 进程池损坏 {e}，重建后跳过本根", flush=True)
|
||||
self._restart_pool()
|
||||
return
|
||||
compute_ms = int((time.perf_counter() - t0) * 1000)
|
||||
t_signal = int(time.time() * 1000)
|
||||
|
||||
hits = res.get("hits", [])
|
||||
atr_pct = res.get("atr_pct")
|
||||
atr_bp = round(atr_pct * 1e4, 3) if atr_pct else ""
|
||||
n_pass = sum(h["pass_all"] for h in hits)
|
||||
|
||||
self.n_bars += 1
|
||||
self.w_lat.writerow({
|
||||
"sym": sym, "kline_ts": kline_ts, "t_close_ms": kline_ts,
|
||||
@@ -268,13 +528,18 @@ class Shadow:
|
||||
"lag_data_ms": t_data - kline_ts,
|
||||
"lag_signal_ms": t_signal - kline_ts,
|
||||
"compute_ms": compute_ms, "n_bars": res.get("n_bars", 0),
|
||||
"n_hits": len(res.get("hits", []))})
|
||||
"n_hits": len(hits), "n_pass": n_pass, "atr_bp": atr_bp,
|
||||
"lag_med_ms": lag_med, "lag_ok": int(lag_ok)})
|
||||
self.f_lat.flush()
|
||||
|
||||
if baseline is not None and np.isfinite(baseline):
|
||||
# 无条件漂移：每根都记，不管有没有信号
|
||||
self._spawn(self._drift_later(sym, kline_ts, baseline),
|
||||
f"drift {sym}")
|
||||
|
||||
if res.get("error"):
|
||||
print(f" [{sym}] 信号计算出错 {res['error']}", flush=True)
|
||||
return
|
||||
hits = res.get("hits", [])
|
||||
if not hits:
|
||||
return
|
||||
if baseline is None or not np.isfinite(baseline):
|
||||
@@ -283,20 +548,73 @@ class Shadow:
|
||||
|
||||
for h in hits:
|
||||
self.n_signal += 1
|
||||
print(f" ★ [{sym}] {kline_ts} 方向 {h['direction']:+d} "
|
||||
f"h1_agree={h['h1_agree']} · 数据 {t_data - kline_ts}ms "
|
||||
self.n_pass += h["pass_all"]
|
||||
mark = "★" if h["pass_all"] else "·"
|
||||
print(f" {mark} [{sym}] {kline_ts} 方向 {h['direction']:+d} "
|
||||
f"同向{h['h1_agree']} 阶梯{h['ladder_ok']} 门控{h['gate_ok']} "
|
||||
f"(ATR {atr_bp or 'na'}bp) · 数据 {t_data - kline_ts}ms "
|
||||
f"信号 {t_signal - kline_ts}ms", flush=True)
|
||||
# 最远的回查点在 t_close+5s，此刻尚未发生；等它过去再一次性落盘
|
||||
asyncio.create_task(
|
||||
self._record_later(sym, kline_ts, h, t_data, t_signal, baseline))
|
||||
self._spawn(
|
||||
self._record_later(sym, kline_ts, h, t_data, t_signal,
|
||||
baseline, atr_pct, lag_ok),
|
||||
f"record {sym}")
|
||||
|
||||
async def _record_later(self, sym: str, kline_ts: int, hit: dict,
|
||||
t_data: int, t_signal: int, baseline: float) -> None:
|
||||
target = kline_ts + int(max(DELAYS_S) * 1000) + 500
|
||||
wait = target / 1000.0 - time.time()
|
||||
def _probe_lag(self, sym: str, lag_ms: int) -> tuple[float, bool]:
|
||||
"""记一根的到达延迟，返回 (滚动中位数, 该币是否健康)。
|
||||
|
||||
不健康时应停止开新仓；影子期不下单，故只落到 lag_ok 字段并告警。
|
||||
"""
|
||||
self.lag_hist[sym].append(lag_ms)
|
||||
ok = lag_healthy(self.lag_hist[sym])
|
||||
med = float(np.median(self.lag_hist[sym]))
|
||||
if ok != self.lag_ok[sym]:
|
||||
state = "恢复" if ok else f"退化，超 {LAG_ALARM_MS:.0f}ms 阈值，停开新仓"
|
||||
print(f" [lag] {sym} {state}：近 {len(self.lag_hist[sym])} 根"
|
||||
f"中位 {med:.0f}ms", flush=True)
|
||||
self.lag_ok[sym] = ok
|
||||
return round(med, 1), ok
|
||||
|
||||
async def _wait_for_delays(self, kline_ts: int) -> None:
|
||||
"""最远回查点是 t_close+5s，等它过去（多留 0.5s 给采样）。"""
|
||||
wait = (kline_ts + int(max(DELAYS_S) * 1000) + 500) / 1000.0 - time.time()
|
||||
if wait > 0:
|
||||
await asyncio.sleep(wait)
|
||||
self._record(sym, kline_ts, hit, t_data, t_signal, baseline)
|
||||
|
||||
async def _record_later(self, sym: str, kline_ts: int, hit: dict,
|
||||
t_data: int, t_signal: int, baseline: float,
|
||||
atr_pct: float | None, lag_ok: bool) -> None:
|
||||
await self._wait_for_delays(kline_ts)
|
||||
self._record(sym, kline_ts, hit, t_data, t_signal, baseline,
|
||||
atr_pct, lag_ok)
|
||||
|
||||
async def _drift_later(self, sym: str, kline_ts: int,
|
||||
baseline: float) -> None:
|
||||
await self._wait_for_delays(kline_ts)
|
||||
for label, delay_ms in self._points(None):
|
||||
target = kline_ts + delay_ms
|
||||
snap = self.books.at(sym, target)
|
||||
if snap is None:
|
||||
continue
|
||||
book_ts, bids, asks = snap
|
||||
mid = (float(bids[0][0]) + float(asks[0][0])) / 2.0
|
||||
self.blog.write(sym, kline_ts, label, delay_ms, target,
|
||||
book_ts, bids, asks)
|
||||
self.w_drf.writerow({
|
||||
"sym": sym, "kline_ts": kline_ts, "delay_label": label,
|
||||
"delay_ms": delay_ms, "book_ts": book_ts,
|
||||
"book_lag_ms": book_ts - target, "baseline_px": baseline,
|
||||
"mid": mid,
|
||||
"drift_bp_long": round((mid - baseline) / baseline * 1e4, 4)})
|
||||
self.f_drf.flush()
|
||||
self.blog.flush() # 每根冲刷一次，进程被杀最多丢一根
|
||||
|
||||
@staticmethod
|
||||
def _points(t_signal_delay: int | None) -> list[tuple[str, int]]:
|
||||
pts = [(f"{d}s", int(d * 1000)) for d in DELAYS_S]
|
||||
if t_signal_delay is not None:
|
||||
pts.insert(0, ("actual", t_signal_delay))
|
||||
return pts
|
||||
|
||||
def _new_bar_open(self, sym: str, kline_ts: int) -> float | None:
|
||||
"""次根开盘价 = 回测假设的成交价。"""
|
||||
@@ -309,25 +627,56 @@ class Shadow:
|
||||
return float(row[1]) if ts == kline_ts else None
|
||||
|
||||
def _record(self, sym: str, kline_ts: int, hit: dict,
|
||||
t_data: int, t_signal: int, baseline: float) -> None:
|
||||
points = [("actual", t_signal - kline_ts)]
|
||||
points += [(f"{d}s", int(d * 1000)) for d in DELAYS_S]
|
||||
t_data: int, t_signal: int, baseline: float,
|
||||
atr_pct: float | None, lag_ok: bool) -> None:
|
||||
d_sign = hit["direction"]
|
||||
|
||||
for label, delay_ms in points:
|
||||
snap = self.books.at(sym, kline_ts + delay_ms)
|
||||
for label, delay_ms in self._points(t_signal - kline_ts):
|
||||
target = kline_ts + delay_ms
|
||||
snap = self.books.at(sym, target)
|
||||
if snap is None:
|
||||
continue
|
||||
_, bids, asks = snap
|
||||
best_bid, best_ask = bids[0][0], asks[0][0]
|
||||
book_ts, bids, asks = snap
|
||||
best_bid, best_ask = float(bids[0][0]), float(asks[0][0])
|
||||
mid = (best_bid + best_ask) / 2.0
|
||||
# 多头吃卖盘,空头吃买盘
|
||||
side = asks if d_sign > 0 else bids
|
||||
best_px = best_ask if d_sign > 0 else best_bid
|
||||
ob = book_from(bids, asks)
|
||||
if label == "actual":
|
||||
# 四个固定点已由无条件漂移那条路径落过，只补这一个
|
||||
self.blog.write(sym, kline_ts, label, delay_ms, target,
|
||||
book_ts, bids, asks)
|
||||
|
||||
for notional in NOTIONALS:
|
||||
fill, filled = walk_book(side, notional)
|
||||
# 名义额按基准价折成基础币再下单——真实委托是基础币计价的，
|
||||
# 框架的 get_vwap_for_volume 也收基础币量。名义额那一栏留着
|
||||
# 是为了跨币可比（1 BTC 和 1 SOL 没法横向比）
|
||||
base_amt = notional / baseline
|
||||
r = ob.get_vwap_for_volume(d_sign > 0, base_amt)
|
||||
fill = float(r.result_price)
|
||||
depth_ok = int(float(r.result_volume) >= base_amt * 0.999)
|
||||
if not np.isfinite(fill):
|
||||
# 25 档吃不下这个量，框架直接给 nan。记一行标明深度不足，
|
||||
# 免得「某个仓位档在薄盘时段整段消失」看不出来
|
||||
self.w_sig.writerow({
|
||||
"sym": sym, "kline_ts": kline_ts, "direction": d_sign,
|
||||
"h1_agree": hit["h1_agree"],
|
||||
"ladder_ok": hit["ladder_ok"],
|
||||
"gate_ok": hit["gate_ok"], "pass_all": hit["pass_all"],
|
||||
"lag_ok": int(lag_ok),
|
||||
"atr_pct": atr_pct if atr_pct else "",
|
||||
"atr_bp": round(atr_pct * 1e4, 3) if atr_pct else "",
|
||||
"t_close_ms": kline_ts, "t_data_ms": t_data,
|
||||
"t_signal_ms": t_signal,
|
||||
"lag_data_ms": t_data - kline_ts,
|
||||
"lag_signal_ms": t_signal - kline_ts,
|
||||
"delay_label": label, "delay_ms": delay_ms,
|
||||
"book_ts": book_ts, "book_lag_ms": book_ts - target,
|
||||
"notional": notional, "base_amt": round(base_amt, 8),
|
||||
"baseline_px": baseline, "mid": mid,
|
||||
"best_px": best_px, "fill_px": "",
|
||||
"filled": round(float(r.result_volume), 8),
|
||||
"depth_ok": 0, "slip_bp": "", "drift_bp": "",
|
||||
"spread_bp": "", "impact_bp": ""})
|
||||
continue
|
||||
slip = d_sign * (fill - baseline) / baseline * 1e4
|
||||
drift = d_sign * (mid - baseline) / baseline * 1e4
|
||||
@@ -336,14 +685,21 @@ class Shadow:
|
||||
self.w_sig.writerow({
|
||||
"sym": sym, "kline_ts": kline_ts,
|
||||
"direction": d_sign, "h1_agree": hit["h1_agree"],
|
||||
"ladder_ok": hit["ladder_ok"], "gate_ok": hit["gate_ok"],
|
||||
"pass_all": hit["pass_all"], "lag_ok": int(lag_ok),
|
||||
"atr_pct": atr_pct if atr_pct else "",
|
||||
"atr_bp": round(atr_pct * 1e4, 3) if atr_pct else "",
|
||||
"t_close_ms": kline_ts, "t_data_ms": t_data,
|
||||
"t_signal_ms": t_signal,
|
||||
"lag_data_ms": t_data - kline_ts,
|
||||
"lag_signal_ms": t_signal - kline_ts,
|
||||
"delay_label": label, "delay_ms": delay_ms,
|
||||
"notional": notional, "baseline_px": baseline,
|
||||
"book_ts": book_ts, "book_lag_ms": book_ts - target,
|
||||
"notional": notional, "base_amt": round(base_amt, 8),
|
||||
"baseline_px": baseline,
|
||||
"mid": mid, "best_px": best_px, "fill_px": fill,
|
||||
"filled": round(filled, 2),
|
||||
"filled": round(float(r.result_volume), 8),
|
||||
"depth_ok": depth_ok,
|
||||
"slip_bp": round(slip, 4), "drift_bp": round(drift, 4),
|
||||
"spread_bp": round(spread, 4),
|
||||
"impact_bp": round(impact, 4)})
|
||||
@@ -353,8 +709,22 @@ class Shadow:
|
||||
while not self.stop.is_set():
|
||||
await asyncio.sleep(300)
|
||||
depth = {s: len(self.books.buf[s]) for s in SYMS}
|
||||
print(f" [心跳] 已处理 {self.n_bars} 根 · 命中 {self.n_signal} 个 "
|
||||
f"· 盘口缓冲 {depth}", flush=True)
|
||||
lag = {s: (f"{np.median(h):.0f}ms" if h else "na")
|
||||
+ ("" if self.lag_ok[s] else "!")
|
||||
for s, h in self.lag_hist.items()}
|
||||
print(f" [心跳] 已处理 {self.n_bars} 根 · 命中 {self.n_signal} 个"
|
||||
f"(过全部滤网 {self.n_pass}) · lag {lag} · 盘口缓冲 {depth}"
|
||||
f" · 回查超容差 {self.books.n_stale} 次"
|
||||
f" · 在途任务 {len(self._tasks)}"
|
||||
f" · 盘口落盘 {self.blog.n} 份"
|
||||
f" · 成交 {self.tape.n_trades} 笔{'' if self.tape.n_trades else ' ⚠监听未生效'}",
|
||||
flush=True)
|
||||
# 五分钟一根都没进来，说明管道断了。不喊一声就只能靠人翻日志
|
||||
if self.n_bars == self._hb_last_bars:
|
||||
print(f" ⚠ [停滞] 距上次心跳未处理任何 K 线"
|
||||
f"（进程池重建 {self.n_broken} 次），管道可能已断",
|
||||
flush=True)
|
||||
self._hb_last_bars = self.n_bars
|
||||
|
||||
async def run(self) -> None:
|
||||
await self.start()
|
||||
@@ -370,9 +740,12 @@ class Shadow:
|
||||
for f in list(self.feeds_l.values()) + list(self.feeds_h.values()):
|
||||
f.stop()
|
||||
await self.connector.stop_network()
|
||||
self.f_sig.close()
|
||||
self.f_lat.close()
|
||||
print(f"\n收工:{self.n_bars} 根 · {self.n_signal} 个信号", flush=True)
|
||||
for f in (self.f_sig, self.f_lat, self.f_drf):
|
||||
f.close()
|
||||
self.blog.close()
|
||||
self.tape.close()
|
||||
print(f"\n收工:{self.n_bars} 根 · {self.n_signal} 个信号"
|
||||
f"(过全部滤网 {self.n_pass})", flush=True)
|
||||
|
||||
|
||||
async def main_async(workers: int, hours: float, pool) -> None:
|
||||
|
||||
Reference in New Issue
Block a user