Adaptiv Drill-Down: Daqiqalardan xom savdolargacha o'zgaruvchan granulyarlikda backtest

Daqiqalik sham (candle) grafiklari backtestlar uchun standart granulyarlik hisoblanadi. Ammo bitta daqiqalik sham ichida narx turlicha harakatlanishi mumkin: ba'zan 0,01% ga, ba'zan esa 2% ga. Stop-loss va take-profit ikkalasi ham bitta daqiqalik shamning [low, high] oralig'iga to'g'ri kelganda, backtest qaysi biri birinchi ishga tushganini bilmaydi. Bu — bajarilish noaniqligi (fill ambiguity) muammosidir.

Sodda yechim — butun backtest uchun soniyalik ma'lumotlarga o'tish. Ammo ikki yil davomida bu ~1 million daqiqalik bar o'rniga ~63 million soniyalik bar degani. Saqlash joyi 60 baravar oshadi, tezlik esa mutanosib ravishda pasayadi.

Adaptiv drill-down bu muammoni hal qiladi: nozik granulyarlikdan faqat haqiqatan zarur bo'lgan joyda foydalanish.

Muammo: Katta shamlarda bajarilish noaniqligi

Aniq bir vaziyatni ko'rib chiqaylik. Strategiya 3000 USDT da long pozitsiya ochdi. Stop-loss: 2970 (-1%). Take-profit: 3060 (+2%).

14:37 dagi daqiqalik sham:

• Open: 3010
• High: 3065
• Low: 2965
• Close: 3050

SL (2970) ham, TP (3060) ham [2965, 3065] oralig'iga to'g'ri keladi. Qaysi biri birinchi ishga tushdi?

Mumkin bo'lgan natijalar:

• Narx avval pastga tushdi -> SL ishga tushdi -> -1% zarar
• Narx avval yuqoriga ko'tarildi -> TP ishga tushdi -> +2% foyda

Bitta savdodagi farq: 3 foiz punkti. 10x leverage bilan — 30%. Yuzlab savdolari bo'lgan backtest uchun bajarilish noaniqligini noto'g'ri hal qilish natijalarni tizimli ravishda buzadi.

Freymvorklar buni standart bo'yicha qanday hal qiladi

Backtest dvijoklarining ko'pchiligi ikkita evristikadan birini ishlatadi:

1. Optimistik: avval TP ishga tushadi -> shishirilgan natijalar
2. Pessimistik: avval SL ishga tushadi -> pasaytirilgan natijalar

Ikkala yondashuv ham taxmin qilishdir. Haqiqiy ma'lumotlar soniya yoki hatto millisoniya darajasida mavjud, va ko'rish mumkin bo'lganda taxmin qilishning hech qanday asosi yo'q.

Drill-Down: To'rt darajali strategiya

Drill-down g'oyasi: daqiqalik darajadan boshlab, faqat noaniqlik bo'lgandagina — narx harakati yoki hajm sakrashlari tufayli — pastroq darajaga "burg'ulab tushish".

Level 1: 1m (minute candles)
  -> If SL or TP is unambiguously outside the [low, high] range — resolve on the spot
  -> If both are within the range — drill down

Level 2: 1s (second candles)
  -> Load 60 second bars for this minute
  -> Walk through second by second: which triggered first?
  -> If a second bar is ambiguous, OR price_move >= min_pct, OR volume >= median_1s * vol_mult — drill down

Level 3: 100ms (millisecond candles)
  -> Load up to 10 bars of 100ms for this second
  -> Walk through 100ms by 100ms
  -> If a 100ms bar is ambiguous, OR price_move >= min_pct, OR volume >= median_100ms * vol_mult — drill down

Level 4: Raw trades
  -> Load individual trades for this 100ms bucket
  -> Resolve the fill at trade-by-trade level — maximum possible precision

Drill-Down kerak bo'lmagan holatlar

Holatlarning 95% da drill-down talab qilinmaydi. Odatiy stsenariylar:

Aniq SL: shamning high'i TP ga yetmaydi, low SL ni buzadi -> SL ishga tushdi, drill-down kerak emas.

Aniq TP: low SL ga yetmaydi, high TP ni buzadi -> TP ishga tushdi, drill-down kerak emas.

Ikkalasi ham ishga tushmadi: ikkala daraja ham oraliqdan tashqarida -> pozitsiya ochiq qoladi.

Gap aniqlash: keyingi shamning ochilishi SL yoki TP orqali sakraydi -> ochilish narxi bo'yicha bajarilish, drill-down yo'q.

Drill-down faqat ~5% barlar uchun kerak — ikkala daraja ham bitta shamning oralig'iga to'g'ri kelganda.

class AdaptiveFillSimulator:
    """
    Four-level drill-down for determining fill order.
    """
    def __init__(self, data_loader):
        self.loader = data_loader
        self.cache_1s = {}  # Cache of second data by month

    def check_fill(self, timestamp, candle_1m, sl_price, tp_price, side):
        """
        Checks whether SL or TP triggered on the given minute candle.

        Returns: ('sl', fill_price) | ('tp', fill_price) | None
        """
        low, high = candle_1m['low'], candle_1m['high']

        open_price = candle_1m['open']
        if side == 'long':
            if open_price <= sl_price:
                return ('sl', open_price)
            if open_price >= tp_price:
                return ('tp', open_price)
        else:
            if open_price >= sl_price:
                return ('sl', open_price)
            if open_price <= tp_price:
                return ('tp', open_price)

        sl_hit = self._level_hit(sl_price, low, high, side, 'sl')
        tp_hit = self._level_hit(tp_price, low, high, side, 'tp')

        if sl_hit and not tp_hit:
            return ('sl', sl_price)
        if tp_hit and not sl_hit:
            return ('tp', tp_price)
        if not sl_hit and not tp_hit:
            return None

        return self._drill_down_1s(timestamp, sl_price, tp_price, side)

    def _drill_down_1s(self, minute_ts, sl_price, tp_price, side):
        """Level 2: second-by-second pass."""
        bars_1s = self.loader.load_1s_for_minute(minute_ts)

        if bars_1s is None or len(bars_1s) == 0:
            return self._pessimistic_fill(side, sl_price, tp_price)

        for bar in bars_1s:
            sl_hit = self._level_hit(sl_price, bar['low'], bar['high'], side, 'sl')
            tp_hit = self._level_hit(tp_price, bar['low'], bar['high'], side, 'tp')

            if sl_hit and not tp_hit:
                return ('sl', sl_price)
            if tp_hit and not sl_hit:
                return ('tp', tp_price)
            if sl_hit and tp_hit:
                result = self._drill_down_100ms(bar['timestamp'], sl_price, tp_price, side)
                if result:
                    return result

        return self._pessimistic_fill(side, sl_price, tp_price)

    def _pessimistic_fill(self, side, sl_price, tp_price):
        """Pessimistic assumption: SL for longs, TP for shorts."""
        if side == 'long':
            return ('sl', sl_price)
        else:
            return ('sl', sl_price)

Unumdorlik

~400 savdosi bo'lgan 2 yillik backtestda drill-down taxminan 20 ta shamda chaqiriladi. Umumiy ortiqcha yuk — butun backtest uchun 1 soniyadan kam.

Adaptiv ma'lumotlarni saqlash

Drill-down soniyalik va millisoniyalik ma'lumotlarni talab qiladi. Ammo hamma narsani maksimal granulyarlikda saqlash amaliy emas:

2 yil davomidagi to'liq 1s arxiv taxminan 13 GB ni tashkil qiladi. 100ms — 100 GB dan ortiq. Hamma narsani saqlash mumkin, ammo isrofgarchilik, chunki drill-down bu ma'lumotlarning 1% dan kamini ishlatadi.

Hot-Second aniqlash

Asosiy kuzatuv: narx sezilarli darajada harakatlanadigan soniyalar kichik qismni tashkil qiladi. Agar narx bir soniya ichida 0,1% dan kam o'zgargan bo'lsa — o'sha soniya uchun 100ms bo'linishini saqlashning ma'nosi yo'q.

Hot-second aniqlash: ma'lumotlarni yuklab olish va qayta ishlash paytida biz har bir soniyani tahlil qilamiz va 100ms shamlarni faqat "hot" soniyalar uchun — ya'ni narx harakati chegaradan oshgan soniyalar uchun — yaratamiz.

def process_trades_adaptive(
    trades: pd.DataFrame,
    min_price_change_pct: float = 1.0,
) -> tuple[pd.DataFrame, pd.DataFrame]:
    """
    Processes raw trades into an adaptive structure:
    - 1s candles for all seconds
    - 100ms candles only for "hot" seconds

    Args:
        trades: DataFrame with columns [timestamp, price, quantity]
        min_price_change_pct: threshold for drill-down to 100ms

    Returns:
        (df_1s, df_100ms_hot) — second candles and 100ms for hot seconds
    """
    trades['second'] = trades['timestamp'].dt.floor('1s')
    df_1s = trades.groupby('second').agg(
        open=('price', 'first'),
        high=('price', 'max'),
        low=('price', 'min'),
        close=('price', 'last'),
        volume=('quantity', 'sum'),
    )

    df_1s['price_change_pct'] = (df_1s['high'] - df_1s['low']) / df_1s['open'] * 100
    hot_seconds = df_1s[df_1s['price_change_pct'] >= min_price_change_pct].index

    hot_trades = trades[trades['second'].isin(hot_seconds)]
    hot_trades['bucket_100ms'] = hot_trades['timestamp'].dt.floor('100ms')

    df_100ms = hot_trades.groupby('bucket_100ms').agg(
        open=('price', 'first'),
        high=('price', 'max'),
        low=('price', 'min'),
        close=('price', 'last'),
        volume=('quantity', 'sum'),
    )

    return df_1s, df_100ms

Saqlash joyini tejash

Masalan — ETHUSDT odatiy oy davomida:

min_price_change_pct = 1.0% chegarasi bilan, hot soniyalar barcha soniyalarning 1% dan kamini tashkil qiladi. Ular uchun 100ms ma'lumotlari 550 MB soniyalik ma'lumotlarga ~50 MB qo'shadi — sezilmaydigan ortiqcha yuk.

Agar soniyalik ma'lumotlar ham adaptiv tarzda saqlansa (faqat bir daqiqa ichidagi harakat 0,1% dan oshganda), hajmni yana 3-5 baravar kamaytirish mumkin.

Parquet saqlash strukturasi

data/{SYMBOL}/
├── source.json                # Exchange source: {"exchange": "binance"} or {"exchange": "bybit"}
├── stats.json                 # Precomputed median volumes: {"median_volume_1s": ..., "median_volume_100ms": ...}
├── klines_1m/
│   ├── 2024-01.parquet       # ~1 MB
│   ├── 2024-02.parquet
│   └── ...
├── klines_1s/
│   ├── 2024-01.parquet       # ~550 MB
│   └── ...
├── klines_100ms_hot/
│   ├── 2024-01.parquet       # ~50 MB (hot seconds only)
│   └── ...
├── trades_hot/
│   ├── 2024-01.parquet       # Raw trades for hot 100ms buckets
│   └── ...
└── states_1m.parquet          # Precomputed rolling state cache (~112 MB)

Har bir fayl bir oylik ma'lumotni qamrab oladi. Soniyalik, millisoniyalik va savdo ma'lumotlari dangasa (lazy) yuklanadi — faqat drill-down ularni so'raganda. stats.json fayli hajmga asoslangan drill-down triggerlari uchun ishlatiladigan oldindan hisoblangan mediana hajmlarni o'z ichiga oladi.

Moliyaviy ma'lumotlar uchun Parquet optimallashtiruvi

Moliyaviy ma'lumotlarning o'ziga xos xususiyatlari bor: timestamp'lar monoton ravishda o'sadi, narxlar silliq o'zgaradi, hajmlar sezilarli darajada farq qiladi. Optimal sozlamalar:

import pyarrow as pa
import pyarrow.parquet as pq

schema = pa.schema([
    pa.field("timestamp", pa.int32()),    # Seconds from epoch — int32 is sufficient
    pa.field("open",      pa.float32()),
    pa.field("high",      pa.float32()),
    pa.field("low",       pa.float32()),
    pa.field("close",     pa.float32()),
    pa.field("volume",    pa.float32()),
])

column_encodings = {
    "timestamp": "DELTA_BINARY_PACKED",   # Monotonic int -> delta compression
    "open":      "BYTE_STREAM_SPLIT",     # Float -> byte-stream split
    "high":      "BYTE_STREAM_SPLIT",
    "low":       "BYTE_STREAM_SPLIT",
    "close":     "BYTE_STREAM_SPLIT",
    "volume":    "BYTE_STREAM_SPLIT",
}

def save_optimized_parquet(df, path):
    table = pa.Table.from_pandas(df, schema=schema)
    pq.write_table(
        table, path,
        compression="zstd",
        compression_level=9,
        use_dictionary=False,
        write_statistics=False,
        column_encoding=column_encodings,
    )

Nima uchun bu sozlamalar:

• Timestamp'lar uchun DELTA_BINARY_PACKED: ketma-ket timestamp'lar doimiy qiymatga farq qiladi (1m uchun 60, 1s uchun 1). Delta kodlash ularni deyarli nolgacha siqadi.
• Float uchun BYTE_STREAM_SPLIT: float32 baytlarini oqimlarga bo'ladi (barcha birinchi baytlar birga, barcha ikkinchi baytlar birga va h.k.). Silliq o'zgaradigan narxlar uchun bu standart kodlashdan 2-3 baravar yaxshi siqishga erishadi.
• ZSTD daraja 9: qulay siqishni ochish tezligi bilan yaxshi siqish.
• float64 o'rniga float32: narxlar va hajmlar uchun yetarli, xotirani 50% tejaydi.

Kesh bilan dangasa yuklash

Drill-down ma'lum bir daqiqa uchun soniyalik ma'lumotlarni so'raydi. Har bir so'rov uchun parquet faylini yuklash sekin. Yechim — oy bo'yicha LRU keshi bilan dangasa yuklash.

from functools import lru_cache
import pyarrow.parquet as pq
import pandas as pd

class AdaptiveDataLoader:
    """
    Lazy loader with cache: loads second data by month,
    keeps the last N months in memory.
    """
    def __init__(self, symbol: str, data_dir: str = "data", cache_months: int = 2):
        self.symbol = symbol
        self.data_dir = data_dir
        self.cache_months = cache_months
        self._cache_1s: dict[str, pd.DataFrame] = {}

    def load_1s_for_minute(self, minute_ts: pd.Timestamp) -> pd.DataFrame | None:
        """Load 1s data for a specific minute."""
        month_key = minute_ts.strftime("%Y-%m")

        if month_key not in self._cache_1s:
            self._load_month_1s(month_key)

        if month_key not in self._cache_1s:
            return None

        df = self._cache_1s[month_key]
        minute_start = minute_ts.floor('1min')
        minute_end = minute_start + pd.Timedelta(minutes=1)

        return df[(df.index >= minute_start) & (df.index < minute_end)]

    def load_100ms_for_second(self, second_ts: pd.Timestamp) -> pd.DataFrame | None:
        """Load 100ms data for a hot second."""
        month_key = second_ts.strftime("%Y-%m")
        path = f"{self.data_dir}/{self.symbol}/klines_100ms_hot/{month_key}.parquet"

        try:
            df = pd.read_parquet(path)
            second_start = second_ts.floor('1s')
            second_end = second_start + pd.Timedelta(seconds=1)
            return df[(df.index >= second_start) & (df.index < second_end)]
        except FileNotFoundError:
            return None

    def _load_month_1s(self, month_key: str):
        """Load a month of 1s data, evict old data from cache."""
        path = f"{self.data_dir}/{self.symbol}/klines_1s/{month_key}.parquet"
        try:
            df = pd.read_parquet(path)
            df.index = pd.to_datetime(df['timestamp'], unit='s')

            if len(self._cache_1s) >= self.cache_months:
                oldest = min(self._cache_1s.keys())
                del self._cache_1s[oldest]

            self._cache_1s[month_key] = df
        except FileNotFoundError:
            pass

Drill-Down'ni backtestingga qo'llash

Backtest sikliga integratsiya:

def backtest_with_adaptive_fill(
    states: pd.DataFrame,
    strategy_params: dict,
    data_loader: AdaptiveDataLoader,
) -> list:
    """
    Backtest with adaptive drill-down for fill simulation.
    """
    fill_sim = AdaptiveFillSimulator(data_loader)
    trades = []
    position = None

    for i in range(len(states)):
        row = states.iloc[i]
        ts = states.index[i]

        candle_1m = {
            'open': row['open'], 'high': row['high'],
            'low': row['low'], 'close': row['close'],
            'timestamp': ts,
        }

        if position is not None:
            fill = fill_sim.check_fill(
                ts, candle_1m,
                position['sl'], position['tp'],
                position['side'],
            )

            if fill is not None:
                fill_type, fill_price = fill
                trades.append({
                    'entry_time': position['entry_time'],
                    'exit_time': ts,
                    'side': position['side'],
                    'entry_price': position['entry_price'],
                    'exit_price': fill_price,
                    'exit_type': fill_type,
                    'drill_down': fill_sim.last_drill_depth,  # 0, 1, or 2
                })
                position = None
                continue

        signal = check_entry_signal(row, strategy_params)
        if signal and position is None:
            position = {
                'side': signal['side'],
                'entry_price': row['close'],
                'entry_time': ts,
                'sl': signal['sl'],
                'tp': signal['tp'],
            }

    return trades

Rolling State Cache bilan bog'liqlik

Drill-down agregatlangan parquet keshni to'ldiradi — ular turli muammolarni hal qiladi:

Ikkala yondashuv ham kundalik sham darajasida ko'rinmaydigan, ammo real backtesting uchun muhim bo'lgan xatolarni bartaraf etadi.

Xulosa: Bajarilishni simulyatsiya qilish yondashuvlarini taqqoslash

Drill-down 1m backtest tezligida to'liq 1s backtestning aniqligini ta'minlaydi. Asosiy kuzatuv: yuqori granulyarlik hamma joyda kerak emas — faqat qaror qabul qilish nuqtalarida.

Hajmga asoslangan Drill-Down

Dastlabki drill-down faqat narx harakatida ishga tushadi — sham oralig'i [low, high] bajarilish noaniqligini yaratish uchun yetarlicha keng bo'lganda. Ammo narx bardaqda qiziqarli narsa yuz berganining yagona belgisi emas.

Hajm sakrashlari xuddi shunday muhim triggerdir. Hajmi medianadan 500 baravar yuqori bo'lgan soniya odatda katta bozor buyurtmasiga, likvidatsiya kaskadiga yoki flash-krashga to'g'ri keladi. Shamning tanasi kichik ko'rinsa ham, o'sha soniya ichidagi haqiqiy narx yo'li juda beqaror bo'lishi mumkin — OHLC ko'rinishi yashiradigan ekstremumlarga tegib o'tgan bo'lishi mumkin.

Drill-down sharti endi OR-asoslangan: yo sezilarli narx harakati, YOKI anomal hajm sakrashi nozikroq granulyarlikka tushishni ishga tushiradi.

def is_hot(bar, median_volume, min_pct=0.1, vol_mult=500):
    """
    Determines if a bar warrants drill-down to the next level.
    Two independent triggers (OR logic):
      - price moved >= min_pct within the bar
      - volume exceeded median * vol_mult
    """
    price_move = (bar['high'] - bar['low']) / bar['open'] * 100
    return price_move >= min_pct or bar['volume'] >= median_volume * vol_mult

Bu faqat narxga asoslangan aniqlashga ko'rinmaydigan stsenariylarni tutadi: open=3000, close=3001 bo'lgan bar, ammo hajmi normadan 50 000 baravar yuqori bo'lsa, millisoniyalar ichida qisqacha 2950 va 3050 ga tegib o'tgan bo'lishi mumkin. Hajmga asoslangan drill-down bo'lmasa, backtest bu soniyani hech qachon yaqinroq ko'rib chiqmaydi.

Xom savdolar: to'rtinchi daraja

Dastlabki uch darajali ierarxiya (1m -> 1s -> 100ms) hali ham bo'shliq qoldiradi: bitta 100ms bucket ichida bir nechta savdo turli narxlarda amalga oshirilishi mumkin. high=3060 va low=2965 bo'lgan bucket uchun biz hali ham aniq ketma-ketlikni bilmaymiz.

Yechim: to'rtinchi va oxirgi daraja sifatida xom savdolarga qadar drill-down qilish.

1m candles (base)
  └─> 1s candles    (when 1s shows price_move >= min_pct OR volume >= median_1s * vol_mult)
      └─> 100ms candles  (when hot second detected)
          └─> Raw trades     (when 100ms shows price_move >= min_pct OR volume >= median_100ms * vol_mult)

Xom savdolar darajasida noaniqlik yo'q — har bir savdoning aniq narxi va timestamp'i bor. Bajarilish yakuniy ravishda hal qilinadi:

def resolve_from_trades(trades, sl_price, tp_price, side):
    """
    Walk through individual trades in chronological order.
    The first trade that crosses SL or TP determines the fill.
    """
    for trade in trades:
        price = trade['price']
        if side == 'long':
            if price <= sl_price:
                return ('sl', price)
            if price >= tp_price:
                return ('tp', price)
        else:  # short
            if price >= sl_price:
                return ('sl', price)
            if price <= tp_price:
                return ('tp', price)
    return None

Xom savdolar darajasi juda kam chaqiriladi — barcha barlarning 0,1% dan kami — ammo chaqirilganda, hech qanday shamga asoslangan taxminiy yaqinlashish tenglasha olmaydigan asosiy haqiqatni (ground truth) taqdim etadi.

Har bir o'tish uchun alohida chegaralar

Turli darajalar orasidagi o'tishlarning turli xususiyatlari bor. Bir soniya ichidagi 0,1% narx harakati muhim; bitta 100ms bucket ichidagi xuddi shu 0,1% ekstremal hisoblanadi. Xuddi shunday, hajm taqsimotlari har bir vaqt shkalasida farq qiladi.

Har bir daraja o'tishining endi o'zining min_pct va vol_mult parametrlari bor:

1s → 100ms:   --min-pct-1s 0.1   --vol-mult-1s 500
100ms → trades: --min-pct-100ms 0.1 --vol-mult-100ms 500

Bu har bir o'tishning sezgirligini mustaqil ravishda aniq sozlashga imkon beradi. Amalda, 100ms'dan savdolarga o'tish qattiqroq chegaradan foydalanishi mumkin, chunki bitta 100ms bucket uchun xom savdolarni yuklash tannarxi minimal.

@dataclass
class DrillDownConfig:
    min_pct_1s: float = 0.1
    vol_mult_1s: float = 500
    min_pct_100ms: float = 0.1
    vol_mult_100ms: float = 500

Doimiy mediana statistikasi

Hajmga asoslangan drill-down har bir vaqt shkalasidagi mediana hajmni bilishni talab qiladi. Har bir backtest uchun medianalarni real vaqtda hisoblash unumdorlik afzalliklarini yo'qqa chiqaradi. Yechim: medianalarni bir marta oldindan hisoblab, keshda saqlash.

Har bir simvol uchun 1s va 100ms granulyarligidagi mediana hajmlar tarixiy ma'lumotlardan hisoblanadi va stats.json faylida saqlanadi:

{
  "ETHUSDT": {
    "median_volume_1s": 12.5,
    "median_volume_100ms": 1.8
  },
  "BTCUSDT": {
    "median_volume_1s": 0.45,
    "median_volume_100ms": 0.06
  }
}

Statistika ma'lumotlar birinchi marta yuklab olinganda simvol bo'yicha bir marta hisoblanadi va barcha keyingi backtestlarda qayta ishlatiladi. Agar ma'lumotlar yangilansa (yangi oylar yuklab olinsa), statistika bosqichma-bosqich qayta hisoblanadi.

def compute_median_stats(symbol, data_dir):
    """Compute and cache median volume stats for a symbol."""
    stats_path = f"{data_dir}/{symbol}/stats.json"

    all_1s = load_all_months(f"{data_dir}/{symbol}/klines_1s/")
    median_1s = all_1s['volume'].median()

    all_100ms = load_all_months(f"{data_dir}/{symbol}/klines_100ms_hot/")
    median_100ms = all_100ms['volume'].median()

    stats = {
        "median_volume_1s": float(median_1s),
        "median_volume_100ms": float(median_100ms),
    }

    with open(stats_path, 'w') as f:
        json.dump(stats, f, indent=2)

    return stats

Ko'p birjali qo'llab-quvvatlash: Bybit

Barcha simvollar Binance'da mavjud emas. XAUTUSDT (oltin) kabi aktivlar uchun ma'lumotlar boshqa birjalardan kelishi kerak. Drill-down tizimi endi muqobil ma'lumotlar manbai sifatida Bybit'ni qo'llab-quvvatlaydi.

Bybit simvollari uchun barcha sham darajalari (1m, 1s, 100ms) va xom savdolar Bybit'ning xom savdo oqimidan quriladi. Jarayon bir xil — xom savdolar har bir vaqt shkalasida shamlarga agregatlanadi — ammo ma'lumotlar manbai farq qiladi.

data/{SYMBOL}/
├── source.json              # {"exchange": "bybit"} or {"exchange": "binance"}
├── klines_1m/
│   └── ...
├── klines_1s/
│   └── ...
├── klines_100ms_hot/
│   └── ...
└── trades_hot/              # Raw trades for hot 100ms buckets
    └── ...

Ma'lumotlar yuklovchisi source.json ni tekshiradi va tegishli yuklab olish quvurini ishlatadi. Backtest dvijogi nuqtai nazaridan, ma'lumotlar formati manba birjadan qat'i nazar bir xil — drill-down mantiqi birjadan mustaqil (exchange-agnostic).

Bu ayniqsa birjalararo strategiyalar yoki faqat ma'lum platformalarda savdo qilinadigan simvollar uchun muhimdir.

Xulosa

Adaptiv drill-down oddiy printsipning qo'llanilishi: hisoblash resurslari va saqlash joyini ma'lumotlarning ahamiyatiga mutanosib ravishda sarflash.

To'rt granulyarlik darajasi:

1. 1m — barlarning 95% i uchun asosiy o'tish
2. 1s — bajarilish noaniqligi yoki hajm sakrashlari paytida drill-down
3. 100ms — ekstremal harakat yoki anomal hajmga ega hot soniyalar uchun drill-down
4. Xom savdolar — hot 100ms bucketlar uchun drill-down, individual savdo darajasida bajarilishni hal qilish

To'rt saqlash darajasi:

1. Barcha 1m — to'liq arxiv, 2 yil uchun ~15 MB
2. Barcha 1s — to'liq yoki adaptiv arxiv, ~550 MB/oy
3. Faqat hot 100ms — soniyalarning <1%, ~50 MB/oy
4. Faqat hot savdolar — eng ekstremal 100ms bucketlar uchun xom savdolar

Ikkita drill-down triggeri (OR mantiqi):

• Narxga asoslangan: barning narx oralig'i min_pct dan oshadi
• Hajmga asoslangan: barning hajmi median * vol_mult dan oshadi

Natija: daqiqa darajasidagi tezlikda tik-simulyator aniqligiga ega backtest. Saqlash joyi eksponent emas, chiziqli ravishda o'sadi. Va birjadan mustaqil drill-down mantiqi bilan bir nechta birjalarni — Binance va Bybit — qo'llab-quvvatlash.

Ko'p vaqt oralig'idagi strategiyalar uchun oldindan hisoblangan kesh haqida ko'proq ma'lumot uchun Agregatlangan Parquet Cache maqolasiga qarang. Yuqori leverage bilan natijalarga funding stavkalarining ta'siri haqida — Funding stavkalari leverage'ingizni o'ldiradi.

Foydali havolalar

1. Apache Parquet — ma'lumotlarni saqlash formati
2. Apache Arrow — BYTE_STREAM_SPLIT kodlash
3. Zstandard — siqish algoritmi
4. Lopez de Prado — Advances in Financial Machine Learning
5. Binance — tarixiy bozor ma'lumotlari

Iqtibos

@article{soloviov2026adaptivedrilldown,
  author = {Soloviov, Eugen},
  title = {Adaptive Drill-Down: Backtest with Variable Granularity from Minutes to Raw Trades},
  year = {2026},
  url = {https://marketmaker.cc/ru/blog/post/adaptive-resolution-drill-down-backtest},
  description = {How adaptive data granularity speeds up backtests and saves storage: drill-down from 1m to 1s, 100ms, and raw trades only where price moved significantly or volume spiked.}
}