North star / What counts as true
Every look is counted
Each pass over the data spends some credibility.
Test enough ideas on the same history and some will look good by chance. So we record every look, and the bar for what counts as real rises with the number of looks.
We correct for selection too. The best of many is biased upward simply by being the best.
Families that lean on this