Portfolio Optimization

Markowitz's Mean-Variance Optimization

Harry Markowitz published "Portfolio Selection" in 1952 — a paper short enough to read in one sitting that ended up kicking off modern portfolio theory and, decades later, a Nobel Prize. His core insight sounds almost too simple to have been worth a Nobel: investors care about expected return and risk together, and it's the mix of assets in a portfolio, not any single holding, that determines both.

That leads directly to the efficient frontier — the set of portfolios that squeeze out the most expected return for a given level of risk, or equivalently, the least risk for a given return. Any portfolio sitting below that frontier is "dominated": there's a better portfolio available at the exact same risk level, so there's no reason to hold it.

The Math

Say you have n assets, with expected returns μ = [μ₁, μ₂, ..., μₙ] and a covariance matrix Σ describing how every pair of assets moves together. A portfolio with weights w = [w₁, ..., wₙ] then has:

Expected return: E[R_p] = wᵀ · μ = Σᵢ wᵢ × μᵢ
Variance: σ²_p = wᵀ · Σ · w = Σᵢ Σⱼ wᵢ wⱼ σᵢⱼ

The optimization itself is: pick a target return μ*, then find the weights w that minimize variance subject to wᵀμ = μ* and everything summing to 1 (fully invested, no cash sitting idle). Rule out short-selling and you just add wᵢ ≥ 0 to the constraints.

The Maximum Sharpe Portfolio

Somewhere along that efficient frontier sits the one portfolio with the best Sharpe ratio of all — the most excess return per unit of risk. That's the tangency portfolio, so named because it sits exactly where a line from the risk-free rate touches the frontier tangentially. Under CAPM's assumptions, this tangency portfolio is literally the market portfolio.

Finding it in practice takes numerical optimization. The usual trick is to introduce a risk-aversion parameter λ and solve the unconstrained version of the problem:

max: wᵀμ − λ/2 × wᵀΣw

Sweep λ from high (very risk-averse, favoring a low-vol portfolio) down to low (risk-tolerant, chasing return), and you trace out the entire efficient frontier one point at a time.

The Covariance Matrix: The Hard Part

Here's the part that trips up everyone who tries this for real: Σ has n(n+1)/2 parameters to estimate. For a modest 50-stock portfolio that's 1,275 numbers — and you're probably working with only around 252 daily return observations per year to estimate all of them. More parameters than data points is a recipe for an unstable matrix that's really just fitting historical noise, not any real underlying structure.

Quants lean on a few tricks to keep Σ from falling apart:

• Shrinkage (Ledoit-Wolf): pull the raw sample covariance matrix toward a simpler, more structured target — like the identity matrix or a single-factor model. It sounds like a small tweak, but it noticeably improves how the portfolio performs on data it wasn't fit to.
• Factor models: express covariance through a small set of factor exposures instead — Σ = B·F·Bᵀ + D, where B is factor loadings, F is factor covariance, and D captures each asset's own idiosyncratic variance. Far fewer parameters to estimate, and far more stable as a result.
• Equal-weight heuristic: the famously dumb-sounding "1/N" portfolio — just split evenly across everything — regularly beats fully optimized portfolios out-of-sample, purely because it sidesteps estimation error entirely.

Implementation in Python

For a small enough portfolio, you can build basic mean-variance optimization with nothing more exotic than NumPy. The general recipe:

1. Estimate expected returns (or, if you'd rather not pretend to predict the future, just assume equal returns as a naive baseline)
2. Compute the sample covariance matrix from historical return data
3. Set up the optimization problem with its constraints
4. Solve it numerically — scipy.optimize.minimize handles most general cases
5. Repeat at several target return levels to trace out the efficient frontier and plot it

In the exercise below, we'll build the core portfolio math from scratch using nothing but the Python standard library, so you can see exactly what's happening under the hood before you ever reach for a library that does it for you.