Honestly, I now spend more time battling misinformation and poorly constructed clinical studies than I do actually communicating useful information. Earlier this morning I wrote about flaws in commercially available antibody testing. I deliberately did not write about Stanford’s misleading Santa Clara study as this has been covered nicely elsewhere (see: Feud over Stanford coronavirus study: The author owe us all an apology)

But now in Los Angeles we have our own misleading study to discuss. Yesterday morning, a study out of the University of Southern California on a sampling of residents in Los Angeles reports that hundreds of thousands of adults here may have already been infected. As of Monday, Los Angeles County had recorded fewer than 13,000 cases. From this study in Los Angeles, the estimated infection rate ranged from 2.8% to 5.6%. The Los Angeles Times then immediately reported that this figure is 55 times higher than the reported official case count.

So apparently the new model in COVID-19 science is to release study reports to journalists without publishing the methodology of the study. Or having the study peer reviewed.

So the first methodologic critique is study sampling. According to the press release, USC researchers and Public Health officials conducted drive-through antibody testing April 10th and 11th at six sites. Participants were recruited via a proprietary database that is representative of the county population. The database is maintained by LRW Group, a market research firm. According to their website – LRW is “an analytics-driven marketing services company powered by data, sophisticated analytics, and deep human understanding.” And not a single epidemiologist in senior management. So who precisely was recruited and, of those, how many agreed?

Second, the USC study used a test manufactured by Hangzhou Alltest Biotech which is the same test used in the Stanford study critiqued above. An independent study(Lassauniere et al. from Statens Serum Institut and Copenhagen University Hospital in Denmark) found this to be the least reliable immunoassay of 9 that they evaluated. It is so unreliable that the UK, who had ordered 3.5 million of the tests, will not even use it. The Danish study suggested a a specificity of 87%. Researchers did not test sensitivity further because it performed so poorly. A separate study reported testing sensitivity of 88.7% and specificity of 90.3%. The Alltest Biotech testing kit was shown to cross react with antibodies to Influenza A, Influenza B, adenovirus, and dengue.

So lets work the numbers. The USC/LA County survey included 863 adults in Los Angeles County. They estimate 4.1% of the county’s adult population has antibody to the virus which means that they had 35 positive tests. Using the published sensitivity and specificty numbers of the Alltest Biotech kit, we can then accurately and mathematically critique this study.

Now, the test performs well when it is negative as its negative predictive value is 99.5% (95% confidence interval: 98.6% to 99.8%). This is the probability that subjects with a negative screening test truly don’t have the disease.

This test performs terribly when it is positive as its positive predictive value is 72.1% (95% confidence interval: 62.6% to 79.9%). This is the probability that subjects with a positive screening test truly have the disease.

This potential error in the test can easily dominate the results, especially in such a small sample size.

Statistician John Cherian of D. E. Shaw Research, a computational biochemistry company, made his own calculations given the test’s sensitivity and specificity — and conservatively estimated the proportion of truly positive people in the Stanford study to range from 0.2% to 2.4%. I would suspect that Los Angeles County has similar true seroprevalence rates. But faulty underlying data on 863 people in a County of over 10 million hasn’t stopped wide and sweeping conclusions.