Section 01
Overview
A portfolio engine built on Markowitz mean–variance, doing two things that are
usually kept apart: comparing optimisation methods on the same
problem, and running the winner through a realistic walk-forward
backtest on real market data.
Three models, three jobs
Three formulations of one question — which assets, in what proportions, for a
given risk budget? — because each answers something different.
Classical (exact)
MILP · PuLP + CBC
Binary selection with equal weights. Gives a provably optimal answer, so it
is the yardstick everything else is measured against.
Quantum-inspired (QUBO + annealing)
dimod + neal
The same problem rewritten as a quantum-ready QUBO and solved by simulated
annealing. Tests whether the quantum-inspired route is competitive.
Continuous realistic
CVXPY · SCIP
Real-valued weights with ridge and stability regularisation. This is the one
that drives the live backtest you can replay in the app.
How it's built — four layers
Data flows top to bottom, one responsibility per file. The fetcher knows nothing
about optimisation; the benchmarking module builds neither
μ/Σ nor QUBOs, it only measures.
That separation is what makes the comparison auditable.
| Layer | File | In | Out | Note |
| Data |
Engine/data_generater/stock.py |
ticker list, date range, yfinance |
one CSV per symbol in Engine/shares/ |
Falls back to Close when Adj Close is missing; a failed ticker is logged and skipped, never crashing the run. |
| Params |
Engine/data_generater/parms.py |
price CSVs |
mu_daily, sigma_daily, returns_daily, prices_aligned, symbols |
Drops assets with too little history, then keeps only dates where every surviving asset has data — reliability bought with sample size. |
| Solve · exact |
Engine/BenchMarking/clasic.py |
μ, Σ, k, q or α |
selection vector x, risk/return, solver status |
PuLP/CBC. Linearises the quadratic risk term with auxiliary y_ij variables, because CBC is not a QP solver. |
| Solve · approx |
Engine/BenchMarking/quantm.py |
μ, Σ, k, q, penalty A, reads/sweeps/seed |
QUBO dict, feasible candidate pool |
Filters annealer samples for sum(x)=k by hand even after adding the penalty — it does not trust the penalty alone. |
| Compare |
Engine/BenchMarking/benuch.py |
classical + SA outputs |
epsilon curves, quality gap, runtime scaling |
Builds the α grid from real min-risk / max-return solutions instead of arbitrary values — this is what makes the comparison fair. |
| Solve · realistic |
Engine/backtest_generater/markowitz_solver_realistic.py |
μ, Σ, α, optional k, ridge/stability γ |
SolveResult: weights, selected assets, status |
CVXPY. Ridge + stability penalties exist specifically to suppress the corner solutions raw mean–variance is notorious for. |
| Evaluate |
Engine/backtest_generater/backtest_walkforward_realistic.py |
price history, BacktestConfig |
portfolio_daily, weights_history, rebalance_log, summary.json |
Re-estimates μ/Σ from past-only data at every monthly rebalance; applies transaction costs and risk-based cash scaling. |
Highlighted rows are the path that produces the results shown in the live app.
The first five rows are the research/comparison track.
Stack
- Engine — Python:
numpy, pandas, cvxpy, pulp (CBC), dimod + neal, yfinance, matplotlib.
- Server — Node.js + Express, serving replay and PEX APIs over the pre-generated CSV/JSON outputs.
- Frontend — plain HTML/CSS/JS with Chart.js. No build step, no framework.
Section 02
Data Sources & Preprocessing
Two real market universes, neither simulated. There is no synthetic-price
fallback anywhere in the codebase — a failed download is logged and
dropped, never invented.
US universe52
tickers downloaded to Engine/shares/
Aligned rows1,836
2018-09-12 → 2025-12-31 after intersection
PEX rows42,370
57 symbols, 2015-01-04 → 2026-02-23
Corporate actions366
mapped to the next trading day
The pipeline, step by step
1
Download
Daily bars from Yahoo Finance for 52 large-cap US names across nine sectors,
2015-01-01 → 2026-01-01. Prefers Adj Close, falls back to
Close.
in: ticker listout: shares/*.csv
2
Filter on history length
Any symbol with fewer than 451 price rows is dropped — the
threshold for 450 usable daily returns. This removes recent listings whose
short history would poison the covariance estimate.
drops: < 451 rows
3
Align dates by intersection
Rows with any missing value are dropped, so every asset shares one
date index. It refuses to fabricate a return for a day an asset did not trade.
The cost is visible: we asked for 2015-01-01, the intersection starts
2018-09-12. The latest-listing member sets the floor for
everyone.
requested: 2015-01-01actual: 2018-09-12rows: 1,836
4
Estimate μ and Σ
Daily simple returns, then the sample mean vector and covariance matrix
(ddof = 1). Annualised copies are for reporting only — the
optimiser works in daily units.
out: mu_daily.csvout: sigma_daily.csv
5
Shrink before solving
On each rolling 3-year window the estimates are shrunk: μ toward zero by 60%,
Σ toward its diagonal by 20%. Mean–variance optimisers amplify estimation
noise; shrinkage is the standard defence.
μ shrink: 0.60Σ shrink: 0.20
Honest caveat
The parms.py intersection step is applied globally, but the backtest
re-runs the same filter inside each rolling window. That means the asset
set is re-derived at every rebalance and genuinely does differ month to month —
the real logs show 52 assets in 67 of the 73 rebalances and 51 in the other
six, as one name's history moves in and out of the 3-year lookback.
Section 03
Mathematical Models
Notation throughout: n assets,
k selected, w the weight vector,
x ∈ {0,1}n the binary selection vector,
μ the daily expected-return vector,
Σ the daily covariance matrix, and
α the daily variance budget.
3.1 · Returns and parameter estimation
Engine/data_generater/parms.py · backtest_walkforward_realistic.py
$$r_t=\frac{P_t-P_{t-1}}{P_{t-1}}\qquad
\mu_i=\operatorname{mean}(r_i)\qquad
\Sigma=\operatorname{cov}(R)$$
Simple daily returns; sample covariance with ddof = 1.
$$\mu^{\text{ann}}=252\,\mu^{\text{daily}}\qquad
\Sigma^{\text{ann}}=252\,\Sigma^{\text{daily}}$$
Shrinkage
$$\tilde\mu=(1-s)\,\mu \qquad
\tilde\Sigma=(1-\delta)\,\Sigma+\delta\,\operatorname{diag}(\operatorname{diag}\Sigma)$$
s = 0.60 pulls expected returns toward zero; δ = 0.20 pulls the
covariance toward its diagonal, damping unstable off-diagonal correlations.
Both are the values actually used to produce the published results.
Turning a volatility target into a variance cap
$$\sigma_{\text{ann}}=\frac{v\%}{100}\qquad
\alpha_{\text{daily}}=\frac{\sigma_{\text{ann}}^{2}}{252}$$
The app's "Risk 1–5" slider is exactly this: risk 1 sets
v = 20% → α = 1.587×10⁻⁴, risk 5 sets v = 80% → α = 2.540×10⁻³.
3.2 · Continuous Markowitz — the realistic model
Engine/backtest_generater/markowitz_solver_realistic.py
The model behind every portfolio in the live replay: maximise expected return under
a hard variance budget, plus two penalties that exist purely for realism.
$$\max_{w}\;\; \mu^{\!\top} w
\;-\;\gamma_{\text{ridge}}\lVert w\rVert_2^{2}
\;-\;\gamma_{\text{stab}}\lVert w-w_{\text{prev}}\rVert_2^{2}$$
$$\text{s.t.}\quad
w^{\!\top}\Sigma w \le \alpha,\qquad
\textstyle\sum_i w_i = 1,\qquad
w_i \ge 0,\qquad w_i \le u$$
Ridge (γ = 0.002) discourages concentration into a single name.
Stability (γ = 0.01) charges the optimiser for moving away from last
month's portfolio — this is what keeps turnover low instead of churning the book
every rebalance.
Optional cardinality — the mixed-integer form
$$x_i\in\{0,1\},\qquad \textstyle\sum_i x_i=k,\qquad
0\le w_i\le u\,x_i,\qquad w_i\ge \ell\,x_i$$
Enabling k turns a convex QP into an MIQP. Feasibility
requires ℓ ≤ 1/k and u·k ≥ 1; the code raises rather than silently
returning an infeasible portfolio. Published results use
k = 5, ℓ = 0.02, u = 0.25.
Numerical safeguards
$$\Sigma_{\text{sym}}=\tfrac12(\Sigma+\Sigma^{\!\top}),\qquad
\Sigma_{\text{psd}}=\Sigma_{\text{sym}}+\varepsilon I$$
ε = 10⁻⁸, scaled by the mean diagonal magnitude. This does not repair a
bad estimate — it only stops the solver failing on tiny negative eigenvalues
introduced by floating-point noise.
3.3 · Binary equal-weight Markowitz — the exact model
Engine/BenchMarking/clasic.py
Both approaches must decide the same thing, so the decision reduces to pure
selection with weights fixed at 1/k. Any win is then
attributable to the search, not the weighting.
$$w_i=\frac{x_i}{k}\qquad\Longrightarrow\qquad
R_p=\frac{\mu^{\!\top}x}{k},\qquad
\sigma_p^{2}=\frac{x^{\!\top}\Sigma x}{k^{2}}$$
Epsilon-constraint form (risk budget)
$$\max_{x}\;\frac{\mu^{\!\top}x}{k}
\quad\text{s.t.}\quad
\frac{x^{\!\top}\Sigma x}{k^{2}}\le\alpha,\quad
\textstyle\sum_i x_i=k,\quad x_i\in\{0,1\}$$
Implemented in code as the equivalent
xᵀΣx ≤ α·k², which avoids dividing inside the model.
Lagrangian form (scalarised trade-off)
$$\min_{x}\;\; q\,x^{\!\top}\Sigma x \;-\; \mu^{\!\top}x$$
Sweeping q traces the trade-off curve; this is the form the QUBO shares.
Reference bounds for the α grid
$$\alpha_{\min}=\min_x x^{\!\top}\Sigma x,\qquad
\alpha_{\max}=x_{\ast}^{\!\top}\Sigma x_{\ast}\ \text{ where }\
x_{\ast}=\arg\max_x \mu^{\!\top}x$$
Both solved exactly, subject to Σx = k. The comparison
grid is then 12 points spanning min-risk to max-return — real endpoints,
not guessed ones.
Linearising the quadratic term
CBC is linear, so xixj cannot appear
directly. McCormick linearisation adds one auxiliary variable per pair:
$$y_{ij}=x_ix_j:\qquad
y_{ij}\le x_i,\quad y_{ij}\le x_j,\quad y_{ij}\ge x_i+x_j-1$$
$$\text{risk}=\sum_i \Sigma_{ii}x_i \;+\; 2\!\!\sum_{i<j}\!\Sigma_{ij}y_{ij}$$
That is n(n−1)/2 extra variables and 3 constraints each — at
n = 40 it is 780 auxiliary variables and 2,340 constraints. This is the single
biggest driver of the exact method's runtime growth.
3.4 · QUBO formulation
Engine/BenchMarking/quantm.py
A QUBO has no constraints, so cardinality becomes a quadratic penalty inside the
objective:
$$\min_{x\in\{0,1\}^n}\;\;
q\,x^{\!\top}\Sigma x \;-\; \mu^{\!\top}x
\;+\; A\Big(\textstyle\sum_i x_i - k\Big)^{2}$$
Because xi² = xi for binary
variables, the penalty expands into terms that fold cleanly into the QUBO matrix:
$$\Big(\sum_i x_i-k\Big)^{2}
=\sum_i x_i + 2\!\!\sum_{i<j}\! x_ix_j - 2k\sum_i x_i + k^{2}$$
$$Q_{ii}=q\,\Sigma_{ii}-\mu_i+A-2Ak,
\qquad
Q_{ij}=2q\,\Sigma_{ij}+2A$$
Choosing the penalty weight
$$A = 5\Big(\max_i|\mu_i| \;+\; q\,\max_i\textstyle\sum_j|\Sigma_{ij}|\Big)$$
A must dominate any gain the objective could get from violating the constraint.
Scaling it off the largest row-sum of |Σ| makes it adapt to the data instead of
being a magic number.
A design choice worth noticing
Even with the penalty in place, quantm.py re-checks every annealer
sample for Σx = k and discards the ones that fail. A
penalty makes infeasible solutions expensive, not impossible —
the explicit filter is what makes the comparison against the exact solver honest.
Section 04
Solvers
Three solvers, three guarantees. "Which is better" depends entirely on what you are
asking for.
Branch-and-bound over the linearised MILP. Returns a certificate of
optimality, which is exactly what a benchmark needs.
- Optimises
- μᵀx subject to a hard risk budget
- Guarantee
- Global optimum (or proven infeasible)
- Cost
- n(n−1)/2 auxiliary vars; superlinear runtime growth
- Why it exists
- The ground truth every other method is scored against
Simulated annealing over the QUBO. Stochastic, so it is run many times and
reported as a distribution rather than a single number.
- Optimises
- q·xᵀΣx − μᵀx + penalty
- Guarantee
- None — best-found, not proven best
- Budget
- 400 reads × 1,500 sweeps, 12 independent trials
- Why it exists
- The QUBO form is what real quantum annealers consume; SA is its classical stand-in
CVXPY → SCIP
markowitz_solver_realistic.py
Convex QP, or MIQP once cardinality is switched on. Real-valued weights, so it
can actually express a portfolio rather than just a shortlist.
- Optimises
- μᵀw minus ridge and stability penalties
- Guarantee
- Global optimum of a convex problem
- Observed
- SCIP, status optimal, in all 73 rebalances
- Why it exists
- It is the only one of the three that a person could actually trade
Solver selection order
No hard-coded backend — it inspects cvxpy.installed_solvers() and takes
the first available:
# continuous (k = None)
CLARABEL → OSQP → ECOS → SCS
# mixed-integer (k set)
SCIP → GUROBI → CPLEX → MOSEK
# ECOS_BB is deliberately excluded: CVXPY warns it can return
# incorrect solutions for mixed-integer problems.
With no MIP-capable backend installed, the code raises rather than silently
returning a continuous solution that ignores k.
Side by side
| Property | Classical | Quantum-inspired | Portfolio engine |
|---|
| Decision variable | x ∈ {0,1}ⁿ | x ∈ {0,1}ⁿ | w ∈ ℝⁿ (+ optional x) |
| Weights | Fixed 1/k | Fixed 1/k | Optimised |
| Risk handling | Hard constraint | Soft, via q | Hard constraint |
| Cardinality | Hard constraint | Quadratic penalty + filter | Hard constraint (MIQP) |
| Optimality | Proven | Not guaranteed | Proven (convex) |
| Determinism | Deterministic | Seed-dependent | Deterministic |
| Regularisation | None | None | Ridge + stability |
| Used in the live backtest | No | No | Yes |
Section 05
Constraints & Assumptions
Every constraint below is enforced in code. Every assumption is one you should be
able to hold the project to.
| Constraint | Form | Value used | Why |
|---|
| Budget | Σwᵢ = 1 | — | Fully invested; weights are shares of capital |
| Long-only | wᵢ ≥ 0 | — | No shorting — matches what a retail investor can actually do |
| Risk budget | wᵀΣw ≤ α | α from the 20–80% vol targets | Turns "how much risk?" into a single number the user can set |
| Cardinality | Σxᵢ = k | k = 5 | A holdable portfolio, not a 52-name index clone |
| Max weight | wᵢ ≤ u·xᵢ | u = 0.25 | Caps single-name concentration at a quarter of capital |
| Min buy-in | wᵢ ≥ ℓ·xᵢ | ℓ = 0.02 | Stops meaningless 0.1% positions; requires ℓ ≤ 1/k |
| PSD safeguard | Σ + εI | ε = 1e-8 | Floating-point hygiene, not estimation repair |
| μ shrinkage | (1−s)μ | s = 0.60 | Sample means are the noisiest input to mean–variance |
| Σ shrinkage | (1−δ)Σ + δ·diag(Σ) | δ = 0.20 | Damps unstable estimated correlations |
| Transaction cost | bps × Σ|Δw| | 10 bps | Charged on total notional traded, both sides |
When does the risk budget actually bite?
A constraint only matters if it binds. Comparing predicted variance against the cap
across all 73 rebalances gives a clean answer — and an unexpected one.
Figure 1 — Risk-budget saturation
How many of the 73 monthly rebalances hit the variance cap, per risk level.
Bars are counts; the line is the daily variance budget α on a log scale.
Real output
Source: backtest_outs/backtest_out_risk_{1..5}/rebalance_log.csv,
comparing the variance_daily_pred and alpha_daily columns.
A real finding, stated plainly
Above roughly a 50% annual volatility target the constraint stops binding
entirely — at risk levels 3, 4 and 5 the optimiser never once reaches the
cap. The consequence is visible in the results: those three levels select the
same portfolios, and their CAGRs span just
0.004 percentage points (all ≈ 24.94%, max drawdown −67.0% for all three —
the residue is solver tolerance, not a real difference). The risk slider is genuinely doing something
between levels 1 and 2, and nothing above level 3. That is a property of this
universe and this α grid, not a bug — but it is a limitation of the current risk
parameterisation, and it belongs on this page rather than buried in a CSV.
Assumptions the model makes
- Variance is an adequate risk measure. It is symmetric — it penalises upside and downside equally — and it assumes returns are well-described by their first two moments. Real returns have fat tails.
- Past covariance predicts future covariance. Shrinkage softens this; it does not remove it.
- Trades execute at the closing price, in full. No slippage, no market impact, no partial fills.
- Cash earns zero. Scaled-down risk exposure sits in a 0% asset, which understates returns in a high-rate environment.
- No taxes, no dividends beyond what adjusted close already embeds.
Section 06
Backtesting Methodology
Engine/backtest_generater/backtest_walkforward_realistic.py
The easiest way to produce a spectacular backtest is to let the model see the
future. This engine is written primarily to prevent that.
Walk-forward mechanics
1
Rebalance on the last trading day of each month
The last trading day of each month, taken from the real market calendar rather than nominal month-ends. Plus a forced rebalance on day one.
count: 73 rebalancesspan: 2020-01-02 → 2025-12-31
2
Estimate from a past-only window
μ and Σ recomputed from [day − 3 years, day) — end-exclusive, so the decision day's own return is invisible. The universe is re-derived every month.
lookback: 3 yearswindow: end-exclusive
3
Solve, then delay execution by one day
Weights decided today take effect tomorrow — you traded at today's close. Choosing this explicitly is what closes the same-day look-ahead hole.
timing: close
4
Scale toward cash if the risk budget is breached
A second line of defence after the solver's own constraint.
scale = √(α / var_pred)cash = 1 − scale
5
Charge for trading, then compound
Cost is levied on the full traded notional — buys and sells — before the day's return is applied.
avg turnover: 2.84%avg cost: 0.57 bps
The formulas
$$\text{scale}=\operatorname{clamp}\!\left(\sqrt{\tfrac{\alpha}{\widehat{\operatorname{Var}}}},\,0,\,1\right),
\qquad w_{\text{cash}} = 1-\text{scale}$$
$$\Delta=\sum_s\bigl|w_s^{\text{new}}-w_s^{\text{old}}\bigr|,\qquad
\text{turnover}=\tfrac{\Delta}{2},\qquad
c=\tfrac{\text{bps}}{10^4}\,\Delta,\qquad
V \leftarrow V(1-c)$$
Turnover is half the total absolute change because every sale funds a purchase;
the cost, by contrast, is charged on the full Δ, since you pay on both legs.
$$r_{p,t}=\sum_i w_i r_{i,t},\qquad V_t=V_{t-1}(1+r_{p,t}),\qquad
\text{DD}_t=\frac{V_t}{\max_{s\le t}V_s}-1$$
$$\text{CAGR}=V_{\text{final}}^{\,252/n}-1,\qquad
\sigma_{\text{ann}}=\operatorname{std}(r_p)\sqrt{252},\qquad
\text{MaxDD}=\min_t \text{DD}_t$$
$$\text{Sharpe}=\frac{\bar r_p}{\operatorname{std}(r_p)}\sqrt{252},\qquad
\text{Sortino}=\frac{\bar r_p}{\sqrt{\operatorname{mean}(r_-^2)}}\sqrt{252}$$
Risk-free rate is taken as zero, so these are excess-return-free ratios;
the downside deviation uses the root-mean-square of negative returns only.
Computed in public/script.js for the live panel and reproduced here.
Missing-data policy
It refuses to silently fill a missing return with zero. Three explicit modes; the
published results use the first:
cash — an asset with no price that day has its weight treated as cash (0% return) for that day only.drop_day — if any held asset is missing, the whole day is marked NaN and excluded.ffill — forward-fill prices with a hard limit of 2 days. Flagged in the source as potentially optimistic.
Results
Read directly out of backtest_outs/. Benchmarks cover the identical
1,508-day window, normalised to 1.00 on 2020-01-02.
Figure 2 — Growth of 1.00
MarkoScale at risk levels 1 and 2 against both benchmarks, net of transaction costs.
Real output
Source: backtest_out_risk_{1,2}/portfolio_daily.csv and
{sp500,nasdaq100}_daily.csv, sampled every 5th trading day
(302 of 1,508 points) for rendering only — all statistics are computed on the full series.
Figure 3 — Drawdown
Distance below the running peak. The gap between risk 1 and risk 2 is the whole argument for the risk slider.
Real output
Reading it: risk 1 bottoms at −27.2% against the
S&P's −33.7%; risk 2 reaches −66.1%
while earning a higher CAGR. Same engine, same data — only α differs.
| Portfolio | Vol target | Final |
CAGR | Realised vol |
Max DD | Sharpe | Sortino |
CAGR, realised vol, max drawdown and final value are read verbatim from each
summary.json. Sharpe and Sortino are not stored there, so they are
recomputed from portfolio_daily.csv using the same formulas the live
app uses. Risk-free rate = 0.
What the portfolio actually held
The stability penalty is not decorative. At risk 1 the optimiser finds a defensive
core and leaves it alone — three names appear in every one of the 73
rebalances. At risk 3, with the budget slack, it swings to growth and momentum.
Figure 4 — Selection frequency
How many of the 73 rebalances each name survived into, for k = 5.
Real output
Source: the selected_assets column of
rebalance_log.csv. Average turnover across all rebalances is
2.84% (risk 1) — the direct, measurable effect of the
‖w − w_prev‖² penalty.
Section 07
Benchmarking & Comparison
Engine/BenchMarking/benuch.py
One rule: both methods face the identical feasible set. Same k, μ,
Σ and equal-weight accounting — and an α grid whose endpoints are themselves solved
exactly rather than picked by hand.
The protocol
$$\mathcal{F}(\alpha)=\{x : \sigma_p^2(x)\le\alpha\},\qquad
x^{\ast}(\alpha)=\arg\max_{x\in\mathcal{F}(\alpha)} R_p(x)$$
For the exact solver this is solved directly. For SA, a candidate pool is
generated by sweeping q, then filtered to 𝓕(α) and the
best feasible member is taken — the annealer never sees α at all.
$$\text{gap}(\alpha)=
\frac{R^{\text{exact}}(\alpha)-\overline{R^{\text{SA}}}(\alpha)}
{\bigl|R^{\text{exact}}(\alpha)\bigr|+10^{-12}}$$
A relative optimality gap. Because SA is stochastic it is reported as a mean over
12 independent seeds with a standard-deviation error bar; the exact solver needs
no error bar because it is deterministic.
This gap is now measured, not estimated
benuch.py renders its comparison with plt.show() and saves
nothing, so this repository originally held no stored exact-vs-SA numbers.
tools/compare_solvers.js now closes that hole: it ports the same
methodology onto the same real μ and
Σ, and writes
backtest_outs/solver_comparison.json. The two figures below are
measured output from that run. One deliberate upgrade: the exact side is
exhaustive enumeration over all 2,598,960 possible selections
rather than CBC branch-and-bound, so the optimum is proven rather than
certified by a solver — which makes the reported gaps hard lower bounds.
The trade-off is that the runtimes below are enumeration runtimes, not CBC timings,
and are labelled that way.
Figure 5 — Solution quality, epsilon curve
Best achievable return as the risk budget α is relaxed. This is the axis on which "did the quantum-inspired method match the exact one?" is decided.
Measured
Figure 6 — Runtime scaling
Wall-clock time against universe size, log scale, at a fixed annealing budget.
Measured
What this does and does not prove
At MarkoScale's actual problem size the quantum-inspired route was slower
and produced worse portfolios than simply checking every option — the
opposite of the result we expected. But enumeration is only cheap because
k is small and fixed (candidates grow like n⁵/120), so
this is emphatically not a general claim that exact search always wins.
The full breakdown, including how the gap responds to more compute, is on the
Compare page.
The realised frontier — what we can measure
The five risk levels form a real, measured risk/return frontier — realised over six
years of out-of-sample trading, not predicted.
Figure 7 — Realised risk vs realised return
Annualised volatility against CAGR, measured over 2020-01-02 → 2025-12-31. Up and to the left is better.
Real output
Reading it: risk 1 sits above and to the left of the S&P 500 — more
return for less volatility, over the same window. Risk levels 3, 4 and 5 land on
top of each other, which is the saturation effect from Section 05 shown
geometrically. This is realised performance on one historical window and one
52-asset universe; it is evidence, not proof.
Section 08
PEX Integration
pex_data/scripts/build_market_site_data.py · server.js
The second data universe is the Palestine Exchange. Two cleaned
source files become API-ready summaries, rankings, resampled OHLC and liquidity
features, served to the PEX Market module.
Trading rows42,370daily records
Symbols57with a master metadata table
Coverage2015→20262015-01-04 to 2026-02-23
Corporate actions366mapped to next trading day
Why an emerging market changes the problem
A US large-cap trades every session. A PEX listing may not. That single difference
drives everything else here — starting with a coverage ratio per
symbol instead of assuming a complete series.
$$\text{coverage}_s=\frac{\#\{\text{rows for } s\}}{\#\{\text{market calendar dates}\}}$$
The server turns this into plain-language guidance: ≥ 0.70 is described as
reliable for charting and backtesting, 0.20–0.70 as usable with lower model
confidence, and below that as risky without liquidity filters.
Liquidity, measured not assumed
$$\text{Amihud}_t=\frac{|r_{1d,t}|}{\text{trade value}_t},\qquad
\overline{\text{Amihud}}^{60d}=\text{rolling mean}_{60}$$
Amihud illiquidity: how much price movement one unit of traded value produces.
High values mean a thin book — a small trade moves the price a lot.
Risk and return series
$$\text{NAV}_t=\prod_{s\le t}(1+r_s),\qquad
\text{DD}_t=\frac{\text{NAV}_t}{\max_{u\le t}\text{NAV}_u}-1,\qquad
\text{MaxDD}^{252d}=\text{rolling min}_{252}$$
$$\sigma^{(w)}_t=\text{rolling std}_w(r_{1d}),\quad w\in\{20,60,252\}
\qquad
\text{CAGR}=\bigl(\text{total return mult}\bigr)^{1/\text{years}}-1$$
Weekly and monthly bars aggregate properly rather than by naive sampling:
open = first, high = max, low = min, close = last,
volume = sum.
Figure 8 — PEX market activity by year
Annual traded value against the number of symbols active that year.
Real output
Source: pex_data/out/api/market/market_summary_10y.csv.
2026 is a partial year (514 rows to 2026-02-23) and should not be read as a
full-year figure. The 2020 dip and the 2024 contraction are both present in the
source data.
Scope, stated precisely
PEX data currently powers the PEX Market analysis module — symbol
summaries, rankings, liquidity features, corporate-action timelines and the market
overview. The published walk-forward backtest results run on the US
universe. Extending the optimiser itself onto PEX names is a stated
direction, not a completed claim, and the coverage-ratio machinery above is the
groundwork for doing it defensibly.
Section 09
Limitations & Future Work
A result you cannot criticise is a result nobody can check. Everything below is a
genuine weakness of the current build.
Known limitations
-
No formal hyperparameter tuning. The values of q,
γ_ridge, γ_stability and the
two shrinkage intensities were chosen by reasoning and inspection, not by
cross-validation. Until that methodology exists, they should be read as
defensible defaults rather than optimised choices.
-
The risk slider saturates. As Figure 1 shows, levels 3–5 produce
identical portfolios because the variance cap never binds there. The mapping from
a 1–5 dial to α needs rescaling to this universe.
-
Yahoo Finance is a moving dependency. Upstream revisions to
historical adjusted prices can change results without any code change, which
weakens exact reproducibility.
-
The cost model is linear and flat. 10 bps on notional, with no
slippage, no market impact, and no liquidity constraint. For a five-name US
large-cap portfolio this is reasonable; for thin PEX names it would not be.
-
Variance is a symmetric risk measure. It penalises upside
deviation as heavily as downside, and assumes returns are adequately described by
two moments. Market returns are demonstrably fat-tailed.
-
Long-only, no leverage, no shorting. A deliberate choice for the
target user, but it does restrict the reachable frontier.
-
Comparison results are not persisted.
benuch.py
plots without saving, so the exact-vs-QUBO comparison cannot currently be cited
with stored numbers — the reason Figures 5 and 6 are labelled as schematics.
-
Historical performance is not predictive. One universe, one
six-year window, one market regime. It is evidence about a method, not a forecast.
Future work
- Persist the benchmark run — write
classical_curve, sa_curve and the scaling timings to CSV so Figures 5 and 6 become real. This is the smallest change with the largest payoff, and it is the first thing on the list.
- Cross-validated parameter selection for the regularisation and shrinkage terms, on a held-out period.
- Richer trading frictions — slippage, market impact, and tax-aware rebalancing.
- Alternative risk measures, particularly CVaR, which targets the tail directly instead of assuming it away. This also has a natural QUBO analogue in the variational-optimisation literature.
- Run the optimiser on PEX names, using the coverage-ratio and Amihud features already built as the liquidity filter that makes it legitimate.
- A production path — live data feeds and a real execution interface, rather than replay over pre-generated output.
- Behavioural field study — does giving an individual investor this tool actually change their diversification and their outcomes? That is the question the project started from, and it is answerable only with real users.
Section 10
References
Portfolio theory
- Markowitz, H. M. (1952). Portfolio selection. The Journal of Finance, 7(1), 77–91. doi:10.2307/2975974
- Anagnostopoulos, K. P., & Mamanis, G. (2011). The mean–variance cardinality constrained portfolio optimization problem. Expert Systems with Applications, 38(11), 14208–14217.
- Cesarone, F., Scozzari, A., & Tardella, F. (2011). Portfolio selection problems in practice: a comparison between linear and quadratic optimization models. arXiv:1105.3594
- Mansini, R., Ogryczak, W., & Speranza, M. G. (2014). Twenty years of linear programming based portfolio optimization. European Journal of Operational Research, 234(2), 518–535.
Quantum, QUBO and annealing
- Glover, F., Kochenberger, G., & Du, Y. (2019). A Tutorial on Formulating and Using QUBO Models. arXiv:1811.11538
- Lucas, A. (2014). Ising formulations of many NP problems. Frontiers in Physics, 2, 5.
- Grant, E., Humble, T. S., & Stump, B. (2021). Benchmarking quantum annealing controls with portfolio optimization. Physical Review Applied, 15(1), 014012.
- Barkoutsos, P. K., et al. (2019). Improving Variational Quantum Optimization using CVaR. Quantum, 3, 167.
- Sakuler, W., Oberreuter, J. M., Aiolfi, R., et al. (2025). A real-world test of portfolio optimization with quantum annealing. Quantum Machine Intelligence, 7, 43.
- Chen, Y., Koch, T., Peng, H., & Zhang, H. (2025). Benchmarking of Quantum and Classical Computing in Large-Scale Dynamic Portfolio Optimization Under Market Frictions. arXiv:2502.05226
Investor behaviour and financial literacy
- Barber, B., & Odean, T. (2000). Trading Is Hazardous to Your Wealth: The Common Stock Investment Performance of Individual Investors. The Journal of Finance, 55(2), 773–806.
- Calvet, L., Campbell, J., & Sodini, P. (2007). Down or Out: Assessing the Welfare Costs of Household Investment Mistakes. Journal of Political Economy, 115(5), 707–747.
- Kahneman, D., & Tversky, A. (1979). Prospect Theory: An Analysis of Decision under Risk. Econometrica, 47(2), 263–291.
- Klapper, L., Lusardi, A., & van Oudheusden, P. (2015). Financial Literacy Around the World. S&P Global FinLit Survey / GFLEC.
Software
- Diamond, S., & Boyd, S. (2016). CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research, 17(83), 1–5.
- Mitchell, S., et al. (2024). PuLP: an LP modeler written in Python. github.com/coin-or/pulp
- D-Wave Systems Inc. (2024).
dimod and neal — binary quadratic model tools and simulated annealing sampler. github.com/dwavesystems
- Boyd, S., & Vandenberghe, L. (2004). Convex Optimization. Cambridge University Press.