ANNIE’s Confidence Numbers Explained

ANNIE gives every signal a confidence number, ranging from -100% sell to +100% buy. I get a lot of questions about what that actually means. The simple answer is that ANNIE uses the number to sort signals. A higher buy confidence moves a stock up the buy list. A stronger sell confidence makes the sell signal look more emphatic. Mechanically, that part is easy.

The more interesting question is whether the confidence number actually affects trade results. Is ANNIE’s top buy really better than the third or fifth stock down the list, or is the confidence number mostly a neat way to organize the output? On the sell side, should I treat a -1% sell differently from a -95% sell, or is that just my human brain trying to invent drama where there may not be any?

I did not have a great answer to that until I thought of a way to test it.

ANNIE normally keeps 10 positions open. When one of those positions receives a sell signal, she sells it and chooses the highest-confidence buy available that day as the replacement. That gave me a clean way to test the buy confidence number. I ran a series of backtests where ANNIE was forced to skip the top buy and choose a lower-ranked buy instead. One test always used the second-best buy of the day. Then the fifth-best, and so on, all the way down to the twenty-sixth-best buy.

The question was simple: if ANNIE cannot use her best buy of the day, does performance degrade as we move down the list?

To my slight surprise, the answer was yes.

There was a definite correlation between buy rank and average return. It was not a perfectly smooth line, because this is stock trading and not pure mathematics. But the pattern was clear enough. The top-confidence buys performed better as a group than the lower-ranked buys.

That means the buy confidence number is doing real work. It is not merely cosmetic.  The rank contains useful information about which candidate should fill the next open portfolio slot.

Then I tested the sell side, and this is where things got more interesting.

The question here was whether a weak sell signal should be treated differently from a strong one. If ANNIE says sell at -1%, should I hold and wait for a stronger signal? Is a -95% sell meaningfully more urgent? That feels like it should be true. A big negative number looks scarier, and humans are wonderfully talented at mistaking scarier for more useful.

So I ran another test. This time ANNIE would only sell if the sell confidence exceeded a certain threshold. I tested 5%, 10%, and continued all the way up to 95%. The idea was to see whether filtering out weaker sell signals improved average returns.

It did not.

To my great surprise, there was no meaningful correlation. A -1% sell was just as useful, on average, as a -95% sell. Waiting for ANNIE to become “more confident” about the sell did not improve the result. It just changed the timing and, in aggregate, did not help.

That is not what I expected. My instinct was that sell confidence would behave like buy confidence. Higher magnitude should mean stronger information. But the test did not support that. I even considered that it would degrade her performance because it might delay a sell longer than it should. The tests told me the reality is possibly both.

The best interpretation I have is that buy confidence and sell confidence represent different kinds of decisions. On the buy side, ANNIE is ranking many possible candidates. She may have dozens of stocks that qualify in some form, but she only needs one replacement for the open slot. In that setting, ranking matters. The top candidate should be better than the twentieth candidate, or the ranking system is not earning its keep.

On the sell side, ANNIE is evaluating a stock already in the portfolio. The question is not “which of these hundred stocks is best?” The question is “does this stock still belong here?” Once ANNIE decides the answer is no, the size of the sell confidence number does not appear to add much useful information. The position has fallen out of favor. Waiting for a more dramatic declaration did not improve the outcome.

There is also a related implication: this helps explain why ANNIE’s sell signals should not be treated as short-selling signals.

A sell signal means ANNIE no longer wants to hold that stock in a long portfolio. That is not the same thing as saying the stock is a good short candidate. Those are different decisions with different training incentives. ANNIE was trained to manage a long portfolio, not to build a ranked list of short opportunities.

The sell-side test reinforces that distinction. If every sell confidence level behaves roughly the same once the sell condition is reached, then ANNIE has not learned to make fine-grained distinctions among sell candidates in the way she does with buy candidates. She had no reason to. For her normal job, once a stock falls out of favor, the useful decision is to remove it and move on. The model does not need to care whether the sell is mildly negative or dramatically negative, because both decisions lead to the same portfolio action.

So the practical conclusion is this: buy confidence matters as a ranking tool, but sell confidence appears to matter mostly as a trigger. If you are choosing from ANNIE’s buy list, the stocks near the top deserve more attention than the stocks farther down. If ANNIE gives a sell signal, even a small one, I do not see evidence from this test that waiting for a stronger sell signal improves the trade.

That does not mean every top buy will work. It does not mean every low-confidence sell will avoid a bad outcome. Individual trades remain noisy, because the market is apparently unwilling to reorganize itself into a tidy spreadsheet for our convenience.

But taken together, the results are clear enough to be useful: ANNIE’s confidence numbers are not a universal measure of “strength,” but a reflection of two different decisions.  One about ranking opportunities, and one about knowing when it’s time to move on.

Based on this study, I have removed the Sell confidence number.  And I'll be rewording the site to talk about Buy Ranking instead of Confidence.