ArticlesCollectionsEnginesInfrastructureStrategiesProjects
Articles/arbitrage

Prerequisite: Average, Spread and the Z-Score

A short from-zero lesson for the arbitrage course. The mean, the standard deviation and the z-score on ten numbers, why one odd value shifts everything, and how a z-score tells you when something is unusual.

2 min readarbitrage course prerequisite

A helper lesson for the course "Build your own arbitrage bot". Lesson 15 uses a z-score to decide when two coins drifted apart. Here it is on ten numbers.

Three questions about a list of numbers

Take the number of minutes your bus was late over ten days: 10, 12, 11, 13, 9, 12, 11, 10, 12, 30.

  1. What is typical? The mean (the average): add all and divide by how many.
  2. How much do they vary? The standard deviation: the typical distance from the mean.
  3. Is today unusual? The z-score: how many standard deviations a value is from the mean.

The steps by hand and in code

import statistics
data = [10, 12, 11, 13, 9, 12, 11, 10, 12, 30]
mean = sum(data) / len(data)
print("mean", mean)
dev = [(x - mean) ** 2 for x in data]
var = sum(dev) / len(data)
print("variance", round(var, 2), "stdev", round(var ** 0.5, 2), "check", round(statistics.pstdev(data), 2))
for x in (11, 30):
    print(x, "z =", round((x - mean) / var ** 0.5, 2))

mean 13.0
variance 33.4 stdev 5.78
11 z = -0.35
30 z = 2.94

Walk through it:

  • The mean is 130 / 10 = 13.
  • For each value, take the distance from the mean and square it (so that minus and plus do not cancel and big distances count more). Average those squares: the variance, 33.4.
  • The square root of the variance is the standard deviation: 5.78. The line statistics.pstdev gives the same number, a check.
  • The z-score is (value − mean) / standard deviation. A day with 11 minutes has z = −0.35, close to normal. The day with 30 minutes has z = 2.94, far from normal.

One odd value shifts everything

Remove the 30 and recompute the other nine days: the mean becomes 11.11 and the standard deviation 1.2. Now the z-score of 30 is 15.8, a huge number. With the odd day inside the list, the mean and the spread grew and 30 looked only mildly strange (2.94). Taking an average over a window that contains the odd value hides it a little. In lesson 15 the window is the last 100 values, and it is a deliberate choice that does not include today.

A rule of thumb

For many things in nature, about 95 values of 100 lie within 2 standard deviations of the mean. A z-score above 2 or below −2 therefore happens on only a few days in 100, which is why a bot treats it as a signal. Real markets have heavier tails than this rule says, so keep the signal and add a stop.

Your exercise

  1. Run the code. Change 30 to 14. What happens to the mean and the standard deviation?
  2. Write your own list of ten numbers (your commute times, your steps per day) and compute the z-score of the last one by hand.
  3. Why do we square the distances instead of adding them as they are?

Back to the course: lesson 15 (pairs trading).

Keep reading