Why Leaderboards Should Rank Out of Sample
Search for the best stock pickers in Congress and you will find lists sorted by return. Most of them share one flaw: they rank people on the period that made them look good. This article explains the flaw and what Filed Trades does instead.
The problem
Imagine 300 people who each flip a coin 20 times to decide which stocks to buy. By pure chance a few of them will look brilliant. If you pick the top of the list and call them skilled, you have measured luck.
Real traders are not coin flippers, but the same arithmetic applies. With hundreds of people, some will show large gains over a short record without any skill. Ranking on that record puts them on top, and the ranking says nothing about what they will do next.
The fix: choose first, judge later
Out of sample ranking separates two jobs. First, decide who is eligible using only trades made before a cutoff date. Then rank those people on the trades made after it. Trades after the cutoff were not used to choose anyone, so they cannot flatter the choice.
On this site the cutoff for Congress is chosen so that roughly the newest 35 percent of matured observations fall after it. A person needs a minimum number of scored buys on both sides. That keeps someone with one lucky trade off the list.
What the table shows
Each row shows the average excess return over the S&P 500 before the cutoff, after it, and overall, with the number of trades behind each. It also shows the standard error, a measure of how much the average could move by chance. A big average on a few trades has a big standard error, and the table lets you see that.
What a good rank still does not mean
An out of sample list is honest about the past. It is not a forecast. A person who did well after the cutoff may have been lucky twice, and many people are compared, so some will always sit at the top. Treat a high rank as a reason to read the record, not as a reason to copy.
A quick self check
- Are there at least 10 or 20 trades behind the number?
- Is the standard error smaller than the average?
- Did the person also do well before the cutoff, or only after?
- Would the story survive if the best single trade were removed?
The leaderboards show all of this, and the methodology page gives the exact rules.