Why You Should be Careful with “Sensitive” Antibody Tests

I’ve just formed a new company, “Dr. Pete’s Reliable Pharmaceuticals Ltd – Healthcare you can trust.” It has a smart looking website and comes top in the Google ads when you search for “covid antibody tests”, so it must be respectable.

I’m selling a new covid-19 antibody test that I guarantee is 100% sensitive: if you’ve got the antibodies then Dr. Pete’s Test will definitely return a positive result. You buy it, you test positive, you celebrate because you have the antibodies, you’re fit and well, you can go out partying again. Two weeks later you catch covid-19 and drop down dead. What went wrong?

To see why my “100% sensitivity” claim is meaningless you have to look at some numbers. Let’s suppose that 10% of the UK population have been infected and have antibodies. As I suggested the other day, this is probably not that far from the truth.

We take 100 people at random from across the UK. It just so happens that 10 of them have antibodies and 90 of them do not.

10 with antibodies
90 without antibodies
100 total

Our ideal test would be SENSITIVE enough to pick up the 10 people with antibodies and return a positive result, but it must also be SPECIFIC enough to only return positive for those 10 people and return negative for the 90 who do not.

As I explain to your grieving family, Dr. Pete’s Test really is 100% sensitive. In fact, it ALWAYS returns positive. You could dip it in a glass of mineral water and it will return positive. I wasn’t lying in the least. I just didn’t tell you what you really needed to know: how specific the test is.

However, even a high specificity may not be as clear as you might think. Dr. Pete’s Test No.2 has been verified as having a 90% sensitivity and a 90% specificity. They sound like pretty good figures. What does testing positive actually tell you though? Less than you might think.

90% sensitivity means it is sensitive to 90% of those who really are positive. So of the 10 people with antibodies, the test will detect 9 of them.

10 with antibodies, 9 test positive
90 without antibodies
100 total

90% specificity means that the test correctly returns negative 90% of the time, but 10% of the time it returns positive for someone who is actually negative. These are false positives.

10 with antibodies, 9 test positive
90 without antibodies, 9 test positive
100 total

So out of the 100 people who take the test, 18 test positive. Those 18 people may think they are safe. In reality, there is only a 50/50 chance that they are safe (and even that assumes they have perfect immunity).

The large number of people who are truly negative, means that even a small percentage of false positives can result in a large overall number of false positives.

I’m not saying that antibody tests are useless. Far from it. A highly sensitive and highly specific test will be invaluable. I would say to be careful buying them online, especially where the test does not tell you how specific it’s results are.

Be careful. Stay safe.

3 thoughts on “Why You Should be Careful with “Sensitive” Antibody Tests

  1. Stop being rational! Coming round here with your fancy Bayesian methods! (sarcasm if it weren’t obvious)

    What tabloids and simple minded politicians want is impressive sounding numbers — never mind whether they have any ‘meaning’ [now doesn’t that sound like a close parallel to another group of speakers??]

    Bayesian statistics and Simpsons paradox should be mandatory parts of the training of anyone interviewing a politician and newspaper editor; then it would be a little easier to hold them to account and more obvious if they were truly convinced or bending numbers to suit them.

    Which makes it all the more ironic that this approach was formulated by Reverend Bayes — although in his day, religious sinecures were handed out to the gentry fairly liberally* and strong religious beliefs not entirely necessary, with many duties delegated allowing the holder to live a relatively carefree life on the backs of his parishioners.

    Usually it went Heir, first spare -> Army commission, 2nd spare -> church

    Like

  2. Thanks for that clear explanation, Peter. Bayes used to live locally, and the Civic Society have put up a red plaque in his honour on the house where he used to live. It would be nice to think that at least some of our current Government could get their heads round some of the sources of uncertainty in science, but I won’t be holding my breath.

    And thanks both for the reference to Simpson’s paradox, which I hadn’t come across before. I am delighted to have learned something new today!

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s