Plato tahlili: barqaror optimumni overfitting-dan qanday ajratish kerak

"Illyuziyasiz backtestlar" turkumining 6-maqolasi

📄 Bu maqola tadqiqot maqolasiga aylandi. Metrikalar rasmiylashtirilgan va takrorlanuvchi simulyatsiyalar bilan stress-testdan o'tkazilgan, Probability of Backtest Overfitting va Deflated Sharpe Ratio bilan solishtirilgan. Maqolani onlayn o'qing (interaktiv versiya + PDF) plateau.marketmaker.cc saytida, kod va ma'lumotlar github.com/suenot/plateau-robustness manzilida.

Siz study.optimize() ni ishga tushirdingiz, Optuna PnL +87% bo'lgan parametrlar to'plamini topdi. Siz hayajondasiz va strategiyani productionga tayyorlamoqdasiz. Ikki haftalik jonli treyding o'tgandan so'ng, PnL deyarli nolga teng. Nima bo'ldi?

Optimizator parametrlar maydonidan ignaning uchini topdi. Parametrlar savdo-sotiqning tarixiy ketma-ketligiga mukammal moslashgan — ammo bozor sharoitlaridagi eng kichik og'ish butun tuzilmani vayron qiladi. Bu klassik overfitting, va uni ishga tushirishdan oldin aniqlash mumkin edi.

Oldingi maqolada biz coordinate descent-ni bayes optimallashtirish bilan solishtirdik va Optuna optimumni nega samaraliroq topishini ko'rsatdik. Bugun — keyingi qadam: topilgan optimum shovqinga moslashtirishning natijasi emas, balki barqaror ekaniga qanday ishonch hosil qilish kerak.

"Eng yaxshi" parametrlarni topish nega ishning faqat yarmi

Optimizator haqiqiy optimumni izlab, keng ko'p o'lchamli parametrlar landshaftida harakatlanmoqda

Strategiya parametrlarini optimallashtirish — ko'p o'lchamli maydondagi maksimumni izlashdir. Muammo shundaki, maksimumlar ikki turda bo'ladi:

1. Plato — parametr o'zgarishlarida ham PnL doimiy yuqori bo'lib qoladigan keng tekis hudud. Hatto bozor sharoitlari samarali parametrlarni 10-20%-ga siljitsa ham, strategiya foyda keltirishda davom etadi.

2. O'tkir cho'qqi — PnL faqat parametrning aniq qiymatida yuqori bo'ladigan tor cho'qqi. Bir qadamlik siljish foydalilikni qulatadi. Bu deyarli aniq overfitting: optimizator barqaror qonuniyatni emas, tarixiy ma'lumotlarning artefaktini topdi.

Alpinistik metafora: plato — siz xavfsiz yura oladigan tog' platosi. O'tkir cho'qqi — faqat muvozanat saqlashingiz mumkin bo'lgan ignaning uchi.

O'tkir cho'qqi va tekis plato — vizual tushuncha

Chapda: barqaror plato (yumshoq yonbag'irlari bo'lgan keng stol tog'). O'ngda: notinch o'tkir cho'qqi (chuqur vodiylar bilan o'ralgan igna uchi)

O'qlar ikki strategiya parametri bo'lgan va rang PnL-ni ifodalaydigan kontur xaritasini tasavvur qiling. Ikki naqshni vizual ravishda oson farqlash mumkin:

Plato (barqaror optimum):

• Bir xil rangdagi keng hududlar
• PnL darajalari orasidagi silliq o'tishlar
• Bir-biridan uzoq joylashgan izoliniyalar
• Optimumdan +/-20%-ga siljish PnL-ni 10%-dan ko'proq o'zgartirmaydi

Issiqlik xaritasini tasavvur qiling: markazida — butun xaritaning taxminan uchdan biriga teng yorqin sariq to'rtburchak. Rang chekkalarga qarab asta-sekin to'q sariqqa, so'ng qizilga o'tadi. Optimum — nuqta emas, balki hudud.

O'tkir cho'qqi (overfitting):

• Sovuq ranglar bilan o'ralgan tor yorqin dog'
• Keskin o'tishlar: optimum yonida darhol qulash
• Tor halqalarga siqilgan izoliniyalar
• +/-5%-ga siljish PnL-ni 50% yoki undan ko'proq pasaytiradi

Xuddi shu issiqlik xaritasini tasavvur qiling, ammo markazida — ko'k va binafsha rang bilan darhol o'ralgan kichkina sariq nuqta. Yagona "to'g'ri" parametrlar kombinatsiyasi.

Parametr sezgirligi tahlili

PnL alohida parametr qiymatlariga qanday bog'liq ekanini ko'rsatuvchi slice plot-lar — keng chiziqlar barqarorlikni, tor klasterlar notinchlikni bildiradi

Bir o'lchamli tahlil: PnL va bitta parametr

Eng oddiy yondashuv — bitta parametrdan tashqari barchasini belgilash va PnL uning qiymatiga qanday bog'liqligini ko'rish. Optuna buning uchun plot_slice ni taqdim etadi:

import optuna
from optuna.visualization import plot_slice

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=500)

fig = plot_slice(study, params=["htf_entry_sell", "ltf_momentum", "stop_loss_pct"])
fig.show()

Slice plot-da nimalarga e'tibor berish kerak:

• Barqaror parametr: nuqtalar buluti optimum yaqinida keng gorizontal chiziq hosil qiladi. Eng yaxshi trial-lar parametr qiymatlarining keng diapazonida tarqalgan.
• Notinch parametr: eng yaxshi trial-lar tor diapazonda to'plangan. Parametrni bir-ikki qadamga siljitish — va foydalilik qulaydi.

Ikki o'lchamli tahlil: kontur diagrammalari (heatmap-lar)

Kontur diagrammasi ikki parametrning o'zaro ta'sirini bir vaqtda ko'rsatadi. Bu plato tahlili uchun asosiy vosita, chunki parametrlar kamdan-kam mustaqil harakat qiladi — kirish va chiqish chegaralari, timeframe-lar va pozitsiya hajmlari o'zaro bog'liq.

from optuna.visualization import plot_contour

fig = plot_contour(study, params=["htf_entry_sell", "htf_exit_buy"])
fig.show()

Barqaror parametrlar juftligi uchun kontur diagrammasi tepalikli tekislikning topografik xaritasiga o'xshaydi: silliq keng izoliniyalar, bir xil rangdagi katta hududlar. Notinch juftlik uchun kontur diagrammasi — vulqon konusining xaritasiga o'xshaydi: bitta nuqta atrofidagi tor konsentrik halqalar.

12 ta separatsiya parametri bo'lgan strategiya uchun bu $\binom{12}{2} = 66$ juftlik kontur diagrammalarini beradi. Ularning barchasini o'rganish shart emas — Optuna eng muhim deb baholagan parametrlardan boshlang.

Ko'p o'lchamli tahlil: parametrlar muhimligi reytingi

Optuna har bir parametrning maqsad funksiyasiga qo'shgan hissasini baholay oladi:

from optuna.visualization import plot_param_importances

fig = plot_param_importances(study)
fig.show()

Parametr muhimligi diagrammasi gorizontal gistogramma. Parametrlar PnL dispersiyasiga qo'shgan hissasi bo'yicha kamayish tartibida saralanadi. Eng yuqori 3-4 parametr odatda dispersiyaning 70-80%-ini tushuntiradi.

Qoida: agar parametr PnL dispersiyasining 2%-dan kamini tushuntirsa, uning qiymati natija uchun deyarli ahamiyatsiz — u ta'rifiga ko'ra barqaror. Plato tahlilini eng muhim 5 ta parametrga qarating.

Optuna vizualizatsiya vositalari

Muhimlik reytinglari bilan birga parametrlarning o'zaro ta'sir landshaftini ko'rsatuvchi kontur heatmap-lari

plot_slice — bir o'lchamli kesimlar

import optuna
from optuna.visualization import plot_slice

fig = plot_slice(study, params=[
    "htf_entry_sell", "htf_entry_buy",
    "ltf_momentum_threshold", "stop_loss_pct",
    "take_profit_pct", "trailing_stop_pct"
])
fig.update_layout(height=800, title="Parameter Slice Plots")
fig.show()

Natija — sochilish diagrammalari to'ri. Har bir kichik diagramma maqsad funksiyasi qiymatini (PnL, Y o'qi) bitta parametr qiymatiga (X o'qi) nisbatan ko'rsatadi. Nuqtalar — alohida trial-lar. Barqaror parametr uchun eng yaxshi nuqtalar (eng yuqori PnL) X ning keng diapazonida tarqalgan. Notinch parametr uchun — tor ustunda to'plangan.

plot_contour — ikki o'lchamli konturlar

from optuna.visualization import plot_contour

important_pairs = [
    ["htf_entry_sell", "htf_entry_buy"],
    ["htf_entry_sell", "stop_loss_pct"],
    ["ltf_momentum_threshold", "take_profit_pct"],
]

for params in important_pairs:
    fig = plot_contour(study, params=params)
    fig.update_layout(title=f"Contour: {params[0]} vs {params[1]}")
    fig.show()

Har bir kontur diagrammasi o'qlarda ikki parametri bo'lgan heatmap. Rang parametrlar maydonining berilgan hududidagi o'rtacha PnL-ni kodlaydi. Sariq/yashil — yuqori PnL, ko'k/binafsha — past. Izoliniyalar bir xil PnL-ga ega nuqtalarni bog'laydi.

plot_param_importances — parametrlar hissasi

from optuna.visualization import plot_param_importances

fig = plot_param_importances(
    study,
    evaluator=optuna.importance.FanovaImportanceEvaluator()
)
fig.show()

fANOVA (functional ANOVA) maqsad funksiyasining dispersiyasini parametrlar va ularning o'zaro ta'sirlariga ajratadi. Bu oddiy korrelyatsiyadan kuchliroq, chunki chiziqli bo'lmagan ta'sirlarni hisobga oladi.

Plato miqdoriy metrikalari

Sensitivity ratio, plato kengligi va robustness score — plato sifatini rasmiylashtiruvchi uchta metrika

Vizual baholash subyektiv. Bizga raqamlar kerak. Mana "plato" tushunchasini rasmiylashtiruvchi uchta metrika.

Sensitivity ratio

PnL o'zgarishining parametr o'zgarishiga nisbati:

$S_i = \frac{\Delta \text{PnL} / \text{PnL}_{opt}}{\Delta p_i / p_{i,opt}}$

bu yerda $\Delta \text{PnL}$ — $p_i$ parametri optimumdan $\Delta p_i$ ga chetlashganda PnL pasayishi.

Talqin:

• $S_i < 0.5$ — parametr barqaror: 10%-lik siljish 5%-dan kam PnL pasayishiga sabab bo'ladi
• $0.5 \leq S_i < 2.0$ — o'rtacha sezgirlik
• $S_i \geq 2.0$ — parametr notinch: 10%-lik siljish PnL-ni 20%+-ga qulatadi

Plato kengligi

PnL optimumning $X\%$ chegarasida qoladigan parametr hududining kengligi:

$W_i(X) = p_{i,max} - p_{i,min} \quad \text{subject to} \quad \text{PnL}(p_i) \geq (1 - X/100) \times \text{PnL}_{opt}$

Nisbiy plato kengligi:

$W_i^{rel}(X) = \frac{W_i(X)}{p_{i,max}^{range} - p_{i,min}^{range}}$

bu yerda maxraj — parametrning to'liq qidiruv diapazoni.

Talqin:

• $W_i^{rel}(10\%) > 0.3$ — plato 10%-lik chegarada diapazonning 30%-dan ko'prog'ini qamrab oladi. Barqaror parametr.
• $W_i^{rel}(10\%) < 0.05$ — plato diapazonning 5%-idan tor. Ogohlantiruvchi belgi.

Robustness score

Barcha parametrlar bo'yicha birlashtirilgan metrika:

$R = \prod_{i=1}^{k} \left( W_i^{rel}(10\%) \right)^{w_i}$

bu yerda $w_i$ — fANOVA dan olingan $i$ parametrining normallashtirilgan muhimligi ($\sum w_i = 1$).

Vaznlangan kengliklar ko'paytmasi — qattiq metrika: hatto bitta muhim parametrning platosi tor bo'lsa ham, $R$ past bo'ladi. Muhim bo'lmagan parametrlar (kichik $w_i$ bilan) deyarli ta'sir qilmaydi.

Talqin:

• $R > 0.1$ — strategiya barqaror
• $0.01 < R \leq 0.1$ — qo'shimcha tekshirish zarur (walk-forward)
• $R \leq 0.01$ — overfitting juda ehtimol

Avtomatlashtirilgan plato aniqlash uchun Python kodi

Barqaror platolar va notinch cho'qqilarni aniqlash uchun parametrlar landshaftini skanerlaydigan avtomatlashtirilgan tizim

import numpy as np
import optuna
from optuna.importance import FanovaImportanceEvaluator
from typing import Dict, List, Tuple

def compute_sensitivity_ratio(
    study: optuna.Study,
    param_name: str,
    n_steps: int = 20,
) -> float:
    """
    Compute sensitivity ratio for a single parameter.

    Fixes all parameters at their best values, varies param_name,
    estimates PnL drop through trial interpolation.
    """
    best_trial = study.best_trial
    best_value = best_trial.values[0]
    best_param = best_trial.params[param_name]

    all_trials = [t for t in study.trials if t.state == optuna.trial.TrialState.COMPLETE]
    all_trials.sort(key=lambda t: t.values[0], reverse=True)
    top_trials = all_trials[:max(10, len(all_trials) // 5)]

    param_values = np.array([t.params[param_name] for t in top_trials])
    pnl_values = np.array([t.values[0] for t in top_trials])

    if best_param == 0 or best_value == 0:
        return float('inf')

    from numpy.polynomial import polynomial as P
    coeffs = np.polyfit(param_values, pnl_values, deg=2)
    dpnl_dparam = 2 * coeffs[0] * best_param + coeffs[1]

    sensitivity = abs(dpnl_dparam * best_param / best_value)
    return sensitivity


def compute_plateau_width(
    study: optuna.Study,
    param_name: str,
    threshold_pct: float = 10.0,
) -> Tuple[float, float]:
    """
    Compute absolute and relative plateau width.

    Returns:
        (absolute_width, relative_width)
    """
    best_value = study.best_value
    threshold = best_value * (1 - threshold_pct / 100)

    trials = [t for t in study.trials if t.state == optuna.trial.TrialState.COMPLETE]
    good_trials = [t for t in trials if t.values[0] >= threshold]

    if not good_trials:
        return 0.0, 0.0

    good_params = [t.params[param_name] for t in good_trials]
    all_params = [t.params[param_name] for t in trials]

    plateau_min = min(good_params)
    plateau_max = max(good_params)
    absolute_width = plateau_max - plateau_min

    search_range = max(all_params) - min(all_params)
    relative_width = absolute_width / search_range if search_range > 0 else 0

    return absolute_width, relative_width


def compute_robustness_score(
    study: optuna.Study,
    threshold_pct: float = 10.0,
) -> Dict:
    """
    Compute combined robustness score.

    Returns:
        dict with per-parameter metrics and the final score
    """
    evaluator = FanovaImportanceEvaluator()
    importances = optuna.importance.get_param_importances(
        study, evaluator=evaluator
    )

    results = {}
    total_importance = sum(importances.values())

    for param_name, importance in importances.items():
        sensitivity = compute_sensitivity_ratio(study, param_name)
        abs_width, rel_width = compute_plateau_width(
            study, param_name, threshold_pct
        )

        weight = importance / total_importance
        results[param_name] = {
            "importance": importance,
            "weight": weight,
            "sensitivity_ratio": sensitivity,
            "plateau_width_abs": abs_width,
            "plateau_width_rel": rel_width,
        }

    log_score = sum(
        r["weight"] * np.log(max(r["plateau_width_rel"], 1e-10))
        for r in results.values()
    )
    robustness_score = np.exp(log_score)

    return {
        "robustness_score": robustness_score,
        "parameters": results,
        "verdict": (
            "robust" if robustness_score > 0.1
            else "check" if robustness_score > 0.01
            else "overfitting"
        ),
    }

Foydalanish

report = compute_robustness_score(study, threshold_pct=10.0)

print(f"Robustness score: {report['robustness_score']:.4f}")
print(f"Verdict: {report['verdict']}")
print()

for name, metrics in report["parameters"].items():
    print(f"  {name}:")
    print(f"    Importance:       {metrics['importance']:.3f}")
    print(f"    Sensitivity:      {metrics['sensitivity_ratio']:.2f}")
    print(f"    Plateau width:    {metrics['plateau_width_rel']:.1%}")
    print()

Misol natija:

Robustness score: 0.1482
Verdict: robust

  htf_entry_sell:
    Importance:       0.312
    Sensitivity:      0.38
    Plateau width:    42.5%

  htf_entry_buy:
    Importance:       0.251
    Sensitivity:      0.45
    Plateau width:    38.1%

  ltf_momentum_threshold:
    Importance:       0.187
    Sensitivity:      1.21
    Plateau width:    22.3%

  stop_loss_pct:
    Importance:       0.098
    Sensitivity:      0.67
    Plateau width:    31.0%

  take_profit_pct:
    Importance:       0.072
    Sensitivity:      0.89
    Plateau width:    28.4%

  trailing_delta:
    Importance:       0.031
    Sensitivity:      0.22
    Plateau width:    55.2%

Separatsiya strategiyalari bilan amaliy misollar

A strategiyasini (keng plato, barqaror), B strategiyasini (o'rtacha) va C strategiyasini (o'tkir cho'qqi, overfitted) solishtirish

12 ta separatsiya parametri bo'lgan uchta strategiyani ko'rib chiqamiz. Har bir strategiya 500 ta trial bilan Optuna optimallashtirishidan o'tdi.

A strategiyasi (~55% PnL, ~500 savdo, ~15% vaqt)

A strategiyasining parametrlari keng plato hosil qiladi. htf_entry_sell asosiy parametrini olaylik:

• Optimal qiymat: 0.020
• 0.015 da PnL: +51% (7% pasayish)
• 0.025 da PnL: +49% (11% pasayish)
• 0.010 da PnL: +43% (22% pasayish)
• 0.030 da PnL: +41% (25% pasayish)

Buni bir o'lchamli diagramma sifatida tasavvur qilsangiz (X o'qi — htf_entry_sell qiymati, Y o'qi — PnL), tekis cho'qqisi bo'lgan yumshoq parabolani ko'rasiz. 0.010-0.030 diapazoni — PnL optimumning +/-25% chegarasida qoladigan plato.

Sensitivity ratio: $S = \frac{0.11}{0.25} = 0.44$ — barqaror.

10%-lik chegarada plato kengligi: 0.013 dan 0.027 gacha, $W^{rel} = \frac{0.014}{0.04} = 35\%$.

B strategiyasi (~25% PnL, ~40 savdo, ~5% vaqt)

B strategiyasi kam sonli savdolar asosida optimallashtirilgan. htf_entry_sell parametri:

• Optimal qiymat: 0.018
• 0.015 da PnL: +24% (4% pasayish)
• 0.025 da PnL: +9% (64% pasayish)
• 0.012 da PnL: +11% (56% pasayish)

Diagrammada — assimetrik va tik chiziq. Plato faqat tor 0.015-0.020 diapazonida mavjud. Optimumning o'ng tomonida — jarlik.

Sensitivity ratio: $S = \frac{0.64}{0.39} = 1.64$ — o'rtacha sezgirlik, lekin 40 ta savdo bilan bu ogohlantiruvchi belgi. Kichik namuna + tor plato = overfitting ehtimoli yuqori.

10%-lik chegarada plato kengligi: 0.016 dan 0.020 gacha, $W^{rel} = \frac{0.004}{0.04} = 10\%$.

C strategiyasi (~300% PnL, ~400 savdo, ~45% vaqt)

C strategiyasi hayratlanarli PnL ko'rsatadi, ammo plato tahlili muammolarni ochib beradi:

• htf_entry_sell ning optimal qiymati: 0.022
• 0.020 da PnL: +295% (2% pasayish)
• 0.025 da PnL: +142% (53% pasayish)
• 0.019 da PnL: +128% (57% pasayish)

Diagrammada — xarakterli "igna": 0.022 da juda yuqori cho'qqi, barcha yo'nalishlarda keskin pasayish. Kontur diagrammasi sovuq ranglar bilan darhol o'ralgan yorqin dog'ni ko'rsatgan bo'lardi.

Sensitivity ratio: $S = \frac{0.53}{0.14} = 3.79$ — notinch. 400 ta savdoga qaramay, strategiya bitta parametrning aniq qiymatiga haddan tashqari bog'liq.

10%-lik chegarada plato kengligi: 0.021 dan 0.023 gacha, $W^{rel} = \frac{0.002}{0.04} = 5\%$.

Yakuniy jadval

Paradoks: PnL-i +300% bo'lgan C strategiyasining robustness score-i eng yomon. "Kamtarona" +55% bo'lgan A strategiyasi eng barqaror. Bu plato tahlilining odatiy natijasi: ta'sirchan raqamlar ko'pincha notinchlikni yashiradi.

Har bir strategiyaning ishonch intervallarini qo'shimcha ravishda Monte Carlo bootstrap orqali tekshirish mumkin — bu savdolarni qayta tanlashda PnL tarqalishini ko'rsatadi.

3D vizualizatsiya va heatmap-lar

Pol tekisligiga proyeksiyalangan kontur chiziqlari bilan ikki parametr bo'yicha PnL ning 3D sirt diagrammasi

Eng muhim parametr juftliklari uchun 3D sirt va heatmap qurish foydali. Bu landshaft shaklini intuitiv tushunishga imkon beradi.

import numpy as np
import matplotlib.pyplot as plt
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D

def plot_parameter_landscape(
    study: "optuna.Study",
    param_x: str,
    param_y: str,
    grid_size: int = 50,
):
    """
    Build a 3D surface plot and heatmap for a pair of parameters.
    """
    trials = [t for t in study.trials
              if t.state == optuna.trial.TrialState.COMPLETE]

    x_vals = np.array([t.params[param_x] for t in trials])
    y_vals = np.array([t.params[param_y] for t in trials])
    z_vals = np.array([t.values[0] for t in trials])

    from scipy.interpolate import griddata

    xi = np.linspace(x_vals.min(), x_vals.max(), grid_size)
    yi = np.linspace(y_vals.min(), y_vals.max(), grid_size)
    Xi, Yi = np.meshgrid(xi, yi)
    Zi = griddata((x_vals, y_vals), z_vals, (Xi, Yi), method='cubic')

    fig = plt.figure(figsize=(18, 7))

    ax1 = fig.add_subplot(121, projection='3d')
    surf = ax1.plot_surface(Xi, Yi, Zi, cmap=cm.viridis, alpha=0.85,
                            edgecolor='none')
    ax1.set_xlabel(param_x)
    ax1.set_ylabel(param_y)
    ax1.set_zlabel('PnL, %')
    ax1.set_title('3D Parameter Landscape')
    fig.colorbar(surf, ax=ax1, shrink=0.5)

    ax2 = fig.add_subplot(122)
    hm = ax2.pcolormesh(Xi, Yi, Zi, cmap=cm.viridis, shading='auto')
    contours = ax2.contour(Xi, Yi, Zi, levels=10, colors='white',
                           linewidths=0.8, alpha=0.7)
    ax2.clabel(contours, inline=True, fontsize=8, fmt='%.0f%%')

    best = study.best_trial
    ax2.scatter(best.params[param_x], best.params[param_y],
                color='red', s=100, marker='*', zorder=5, label='Optimum')

    ax2.set_xlabel(param_x)
    ax2.set_ylabel(param_y)
    ax2.set_title('Contour Heatmap')
    ax2.legend()
    fig.colorbar(hm, ax=ax2)

    plt.tight_layout()
    plt.savefig(f'landscape_{param_x}_vs_{param_y}.png', dpi=150)
    plt.show()

Barqaror strategiya uchun 3D sirt diagrammasi stol tog'iga o'xshaydi — yumshoq yonbag'irlari bo'lgan tekis cho'qqi. Notinch strategiya uchun — Matterhorn kabi o'tkir cho'qqi. Heatmap 3D ko'rinishni to'ldirib, izoliniyalar bilan birga xuddi shu ma'lumotni yuqoridan pastga proyeksiyada ko'rsatadi.

Ogohlantiruvchi belgilar: optimallashtirish natijalari shubhali bo'lganda

Optimallashtirish natijalarida mumkin bo'lgan overfitting-ni bildiruvchi ogohlantirish ko'rsatkichlari

Optimallashtirish haqiqiy qonuniyat o'rniga overfitting topganini bildiruvchi sakkizta belgi:

1. Asosiy parametr uchun sensitivity ratio > 2

Agar parametrning 10%-lik siljishida PnL 20%-dan ko'proq pasaysa — optimum notinch.

2. Qidiruv diapazonining 10%-idan kam plato kengligi

Agar "yaxshi" hudud o'rganilgan diapazonning 10%-idan kamini egallasa — optimizator ehtimol artefakt topgan.

3. Top-3 trial mediandan 2-3 baravar yuqori PnL beradi

Agar eng yaxshi trial-lar "tepalik cho'qqisi" o'rniga qolganlariga nisbatan outlier bo'lsa — bu plato emas.

top_3_mean = np.mean(sorted([t.values[0] for t in study.trials
                              if t.state == optuna.trial.TrialState.COMPLETE],
                             reverse=True)[:3])
median_pnl = np.median([t.values[0] for t in study.trials
                         if t.state == optuna.trial.TrialState.COMPLETE])

outlier_ratio = top_3_mean / median_pnl
if outlier_ratio > 2.5:
    print(f"WARNING: Top trials are {outlier_ratio:.1f}x above median — possible overfitting")

4. Yuqori PnL bilan savdolar soni kam (< 50)

Kichik namuna + yuqori PnL = baholashda yuqori dispersiya. 40 ta savdo bo'yicha plato tahlili o'zi ham ishonchsiz. Bunday strategiyalar uchun Monte Carlo bootstrap juda muhim.

5. Bitta "sehrli" parametrlar kombinatsiyasi

Agar kontur diagrammasi kulrang maydon o'rtasida bitta yorqin nuqtani ko'rsatsa — bu strategiya emas, balki ma'lumotlarga moslashtirilgan kombinatsiya.

6. Juda ko'p parametrlar

Har biri 10 qiymatga ega 12 ta parametr uchun qidiruv maydonida $10^{12}$ kombinatsiya mavjud. Optuna ~500 tasini o'rganadi. Bunday maydonda "yaxshi" artefakt topish ehtimoli yuqori. Parametrlar qancha ko'p bo'lsa, plato tahlili shuncha qattiq bo'lishi kerak.

7. PnL out-of-sample-da keskin pasayadi

Agar in-sample PnL +87% bo'lsa va walk-forward +12% ko'rsatsa — optimallashtirish parametrlarni o'qitish davriga moslashtirgan. Bu haqda ko'proq Walk-Forward optimallashtirish maqolasida.

8. Parametrlar diapazon chegaralariga "mixlangan"

Agar optimal qiymat qidiruv to'rining chegarasi bilan mos kelsa — optimum diapazondan tashqarida bo'lishi mumkin. Diapazonni kengaytiring va optimallashtirishni qayta ishga tushiring.

Avtomatlashtirilgan plato tahlili hisoboti

Har bir optimallashtirishdan keyin yaratiladigan yagona hisobotga barchasini birlashtirish:

import json
from datetime import datetime

def generate_plateau_report(
    study: "optuna.Study",
    strategy_name: str,
    n_trades: int,
    threshold_pct: float = 10.0,
) -> dict:
    """
    Generate a complete plateau analysis report.
    """
    robustness = compute_robustness_score(study, threshold_pct)

    red_flags = []

    sorted_params = sorted(
        robustness["parameters"].items(),
        key=lambda x: x[1]["importance"],
        reverse=True
    )
    for name, metrics in sorted_params[:3]:
        if metrics["sensitivity_ratio"] > 2.0:
            red_flags.append(
                f"High sensitivity for {name}: "
                f"S={metrics['sensitivity_ratio']:.2f}"
            )

    for name, metrics in robustness["parameters"].items():
        if metrics["plateau_width_rel"] < 0.05:
            red_flags.append(
                f"Narrow plateau for {name}: "
                f"W={metrics['plateau_width_rel']:.1%}"
            )

    all_values = sorted(
        [t.values[0] for t in study.trials
         if t.state == optuna.trial.TrialState.COMPLETE],
        reverse=True
    )
    if len(all_values) > 10:
        top3 = np.mean(all_values[:3])
        med = np.median(all_values)
        if med > 0 and top3 / med > 2.5:
            red_flags.append(
                f"Top trials are outliers: "
                f"{top3:.1f} vs median {med:.1f} "
                f"({top3/med:.1f}x)"
            )

    if n_trades < 50:
        red_flags.append(f"Low trade count: {n_trades}")

    report = {
        "strategy": strategy_name,
        "timestamp": datetime.now().isoformat(),
        "best_pnl": study.best_value,
        "n_trials": len(study.trials),
        "n_trades": n_trades,
        "robustness_score": robustness["robustness_score"],
        "verdict": robustness["verdict"],
        "red_flags": red_flags,
        "parameters": robustness["parameters"],
    }

    return report


report = generate_plateau_report(
    study, strategy_name="Strategy A", n_trades=491
)

print(json.dumps(report, indent=2, default=str))

Misol natija:

{
  "strategy": "Strategy A",
  "best_pnl": 55.2,
  "n_trials": 500,
  "n_trades": 491,
  "robustness_score": 0.1482,
  "verdict": "robust",
  "red_flags": [],
  "parameters": {
    "htf_entry_sell": {
      "importance": 0.312,
      "sensitivity_ratio": 0.44,
      "plateau_width_rel": 0.35
    }
  }
}

Walk-Forward tekshiruvi bilan bog'liqligi

Ikki bir-birini to'ldiruvchi tekshirish tizimi sifatida parametrik barqarorlik (plato tahlili) va vaqtinchalik barqarorlik (walk-forward)

Plato tahlili va walk-forward tekshiruvi (WFO) — bir-birini to'ldiruvchi usullar:

• Plato tahlili quyidagi savolga javob beradi: "Optimum kichik parametr siljishlariga qanchalik barqaror?" Bu parametrik barqarorlikni tekshirishdir.
• Walk-forward quyidagi savolga javob beradi: "Parametrlar optimizator ko'rmagan ma'lumotlarda ishlaydimi?" Bu vaqtinchalik barqarorlikni tekshirishdir.

Strategiya plato tahlilidan (keng plato) o'tishi mumkin, ammo walk-forward-da muvaffaqiyatsizlikka uchrashi mumkin (bozor rejimi o'zgardi). Va aksincha — sobit parametrlar bilan walk-forward-dan o'tishi mumkin, ammo notinch optimumga ega bo'lishi mumkin.

Tavsiya: har doim ikkala usuldan ham foydalaning. Agar strategiya plato tahlilidan ($R > 0.1$) va walk-forward-dan ($\text{PnL}_{OOS} > 50\% \times \text{PnL}_{IS}$) o'tsa — bu barqarorlikning kuchli signali. Batafsil ma'lumot Walk-Forward optimallashtirish maqolasida.

Har bir bosqichda PnL ishonch intervallarini baholash uchun Monte Carlo bootstrap ni qo'llang. Turli faol vaqtga ega strategiyalarni to'g'ri solishtirish uchun esa faol vaqtga nisbatan PnL metrikasidan foydalaning.

Tavsiyalar

Optimallashtirishdan oldin

1. Parametrlar sonini cheklang. Parametr qancha kam bo'lsa — plato shuncha ishonchli. 5-7 ta parametr oqilona maksimum. 12 tada allaqachon yuqori ehtiyotkorlik talab qilinadi.

2. Mazmunli diapazonlar belgilang. Haqiqiy diapazon 0.005 dan 0.05 gacha bo'lsa, htf_entry_sell ni 0.001 dan 1.0 gacha belgilamang. Keraksiz keng diapazonlar plato illyuziyasini yaratadi.

3. Yetarlicha trial-lardan foydalaning. 12 ta parametr uchun kamida 300-500 trial. Ishonchli plato tahlili uchun — 1000+.

Optimallashtirish davomida

4. Konvergensiyani kuzating. Agar Optuna 400 trial-dan keyin ham sezilarli darajada yaxshiroq yechimlarni topishda davom etsa — jarayon yig'ilmagan va plato tahlili ishonchsiz bo'ladi.

5. Pruning-dan ehtiyotkorlik bilan foydalaning. Agressiv pruning (MedianPruner) dastlabki qadamlarda yomon ko'ringan, lekin landshaftning to'liq tasvirini qurish uchun muhim bo'lgan trial-larni kesib tashlashi mumkin.

Optimallashtirishdan keyin

6. Plato hisobotini avtomatik yarating. generate_plateau_report() ni optimallashtirish pipeline-iga kiriting. Vizual baholashga ishonmang — raqamlardan foydalaning.

7. Top-5 parametrlarni tekshiring. Agar fANOVA 3 ta parametr dispersiyaning 80%-ini tushuntirishini ko'rsatsa — qolgan 9 tasini kamroq puxtalik bilan tekshirish mumkin.

8. Baseline strategiya bilan solishtiring. Agar standart parametrli strategiya (optimallashtirishsiz) +30% ko'rsatsa, optimallashtirilgani +55% ko'rsatsa — farq atigi 25 foiz punkt, va plato ehtimol keng. Agar standart 0% ko'rsatsa, optimallashtirilgani +300% ko'rsatsa — butun foydalilik parametrlarning aniq moslashtirilishiga bog'liq.

9. Yakuniy tekshiruv — walk-forward. Plato tahlili barqarorlik uchun zarur, lekin yetarli bo'lmagan shart. Har doim out-of-sample ni tekshiring.

Xulosa

Parametrlarni optimallashtirish kuchli vosita, ammo plato tahlilisiz bu ruletka o'yini. Siz barqaror qonuniyat topganingizni yoki modelni shovqinga moslashtirganingizni bilmaysiz.

Plato tahlilining uch qoidasi:

1. Robustness score-ni hisoblang. Vaznlangan plato kengliklarining ko'paytmasi barcha parametrlarning barqarorligini umumlashtiruvchi yagona raqamni beradi. $R > 0.1$ — yashil chiroq.

2. Asosiy parametrlar uchun sensitivity ratio < 1. Agar parametrning 10%-lik siljishi 10%-dan kam PnL pasayishiga sabab bo'lsa — parametr barqaror. Agar ko'proq bo'lsa — ehtiyot bo'ling.

3. Kontur diagrammalarini vizualizatsiya qiling. Hech qanday metrika landshaft shaklini tushunishning o'rnini bosa olmaydi. Tekis stol tog' — yaxshi. O'tkir igna — yomon.

Plato tahlili optimallashtirishdan keyin 5 daqiqa vaqt oladi va foydasiz jonli treydingning haftalarini tejashi mumkin. Bu study.optimize() va botni ishga tushirish orasidagi majburiy qadam.

Foydali havolalar

1. Optuna Documentation — Visualization
2. Hutter, F., Hoos, H., Leyton-Brown, K. — An Efficient Approach for Assessing Hyperparameter Importance (fANOVA, 2014)
3. Pardo, R. — The Evaluation and Optimization of Trading Strategies
4. Marcos Lopez de Prado — Advances in Financial Machine Learning, Chapter 11: Dangers of Backtesting
5. Bailey, D.H. et al. — The Probability of Backtest Overfitting (2015)
6. Optuna — optuna.visualization.plot_contour
7. Optuna — optuna.importance.FanovaImportanceEvaluator
8. Bergstra, J. & Bengio, Y. — Random Search for Hyper-Parameter Optimization (2012)

Iqtibos

@article{soloviov2026plateauanalysis,
  author = {Soloviov, Eugen},
  title = {Plateau Analysis: How to Distinguish a Robust Optimum from Overfitting},
  year = {2026},
  url = {https://marketmaker.cc/en/blog/post/plateau-analysis-overfitting},
  version = {0.1.0},
  description = {Why finding the best strategy parameters is only half the work. How to visually and quantitatively distinguish a stable plateau from a fragile peak, and why Optuna contour plots are a mandatory step before launching an optimized strategy into production.}
}