The researchers worked around the clock, in teams of three to five hours, hoping to avoid weariness and keep a sharp mind for the delicate task.
They have set up lines of laboratory volunteers: resident doctors, postdoctoral students, and even experienced science veterans, each carrying out a specific task. They checked and double-checked their data, as if the world depended on it. Because in some ways it is.
In the past few weeks, more than 50 scientists have worked diligently to do something that most of the Food and Drug Administration has not done: verify that 14 anti-coronavirus antibody tests currently on the market do provide precise results.
These tests are crucial to the reopening of the economy, but public health experts have raised pressing concerns about their quality. The new research, completed just a few days ago and published online Friday, confirmed some of these fears: of the 14 tests, only three has always produced reliable results. Even the best had flaws.
The research has not been peer reviewed and is subject to review. But the results already raise difficult questions about the course of the epidemic.
Surveys conducted this week among residents of the Bay Area, Los Angeles and New York found that substantial percentages were positive for antibodies to SARS-CoV-2, the official name for the new coronavirus. In New York, this figure would reach 21%. Elsewhere, it was closer to 3%.
The idea that many people in parts of the country have already been exposed to the virus has broad implications. At the very least, this finding could greatly complicate plans to reopen the economy.
Already, the Americans are scrambling to pass antibody tests to see if they could escape the blockages. Public health experts are wondering if those with positive results could be allowed to return to work.
But these tactics mean nothing if the test results are not reliable.
In new research, researchers found that only one of the tests ever delivered a so-called false positive – that is, it never mistakenly reported antibodies in people who had none.
Two other tests did not give false positive results in 99% of the cases.
But the reverse was not true. Even these three tests only detected antibodies in infected people in 90% of cases, at best.
The false positive measure is particularly important. The result can lead people to believe they are immune to the virus when they are not and to put themselves in danger by giving up social distancing and other protective measures.
It is also the outcome on which scientists are most divided.
“There are several tests that seem reasonable and promising,” said Dr. Alexander Marson, an immunologist at the University of California at San Francisco and one of the project leaders. “This is reason to be optimistic. “
Dr. Marson is also a researcher at Chan Zuckerberg Biohub, which partially funded the study.
Other scientists were less optimistic than Dr. Marson. Four of the tests produced false positive rates ranging from 11% to 16%; many others hovered around 5%.
“These numbers are simply unacceptable,” said Scott Hensley, microbiologist at the University of Pennsylvania. “The tone of the document is,” Look at how good the tests are. “But I’m looking at this data, and I don’t really see that. “
The proportion of people in the United States who have been exposed to the coronavirus is probably 5% or less, said Dr. Hensley. “If your kit contains 3% false positives, how do you interpret that? It’s almost impossible, ”he said. “If your kit contains 14% false positives, it’s useless. “
Dr. Hensley said the study was nevertheless well-designed and the results were pressing, given the sudden proliferation of antibody tests on the market and the desire to use them to lift locks.
“I think this is exactly the type of study we need right now,” he said.
Dr. Marson and his colleagues said they were attracted to the study for the same reason.
As Bay Area universities shut down all research unrelated to the coronavirus, some researchers have begun to focus on ways to improve diagnostic tests for SARS-CoV-2.
Dr. Marson and his collaborator Patrick Hsu, a bioengineer at the University of California at Berkeley, predicted that the antibody tests would also face quality questions.
In mid-March, Dr. Hsu learned that a friend, a venture capitalist with a network of 1,000 community clinics in the New York area, had ordered thousands of rapid antibody tests. Investors and entrepreneurs also appeared to distribute them around San Francisco.
“I realized,‘ My God, this is really the Wild West, “said Dr. Hsu. “We had to figure out which of them would really work.”
The duo recruited Dr. Jeffrey Whitman and Dr. Caryn Bern, who last year published a analysis of antibody tests for Chagas disease. Other graduate students and postdoctoral fellows volunteered to help carry out the assessments.
The team started with a modified version of the method that Dr. Whitman had designed to validate Chagas tests. The researchers created a certified biosecurity space, obtained the necessary approvals and purchased hundreds of blood samples from two hospitals in the Bay Area.
They also bought tests from Chinese manufacturers, removing customs regulations and sometimes accepting Uber deliveries in the middle of the night.
In all, the investigators analyzed 10 rapid tests that provide a yes-no signal for antibodies, and two tests using a laboratory technique known as Elisa that indicate the amount of antibodies present and are generally considered to be more reliable.
Dressed in protective gear, the team worked in shifts of three to five hours in a sort of socially distant factory line.
One researcher located the test with a blood sample and another added the necessary chemical solutions; then two independent readers reviewed the test and a final person recorded the results. Other team members analyzed the results, sometimes working overnight.
In the early hours of recent mornings, they handed over to Dr. Tyler Miller and his colleagues at Massachusetts General Hospital, who performed a slightly different analysis of three tests, including one evaluated in San Francisco.
The Bay Area team completed the evaluation of 12 tests in record time, less than a month. For comparison, the Chagas project required a team of three people working for more than a year just to compare four tests.
Having a study plan in hand helped speed up the work, but there was a key difference. Decades of data have shown that Chagas disease causes lifelong immunity. For this study, the team had no idea how quickly and at what levels anti-SARS-CoV-2 antibodies could appear in the blood.
Each test was evaluated with the same set of blood samples: from 80 people known to be infected with the coronavirus, at different times after infection; 108 samples given before the pandemic; and 52 samples from people who tested positive for other viral infections but tested negative for SARS-CoV-2.
Tests by Sure Biotech and Wondfo Biotech, as well as an internal Elisa test, produced the least number of false positives.
A test by Bioperfectus detected antibodies in 100% of the infected samples, but only after three weeks of infection. None of the tests did better than 80% until that time, which was longer than expected, said Dr. Hsu.
The lesson is that tests are less likely to produce false negatives for a long time since the initial infection occurred, he said.
Tests were particularly variable when looking for a transient antibody that occurs soon after infection, called IgM, and more consistent in identifying a later antibody, called IgG, which could signal immunity to more long term.
“You can see that the antibody levels increase at different points for each patient,” said Dr. Hsu. The tests worked better when the researchers assessed the two types of antibodies together. However, none of the tests could say whether the presence of these antibodies means that a person is protected from reinfection.
Overall, the results are promising, added Dr. Marson. “There are several tests that have specificities greater than 95%.”
Rapid antibody tests are generally used to obtain a simple yes-no result, but the team assigned positive results – which appear as bands on a test strip – a score of zero to six. They trained readers to interpret these results and found that their decisions were often accepted and supported by the more quantitative Elisa tests.
“If you train the readers well, they can start to be reliable,” said Dr. Marson of the rapid tests. “It is essential to understand if these tests could be deployed.”
The Mass General team set a higher bar for specificity; they viewed a score of one for the intensity of a band as a negative result, rather than a score of zero.
Perhaps because they eliminated the weakest bands – the ones most likely to be in error – their estimate of the specificity for BioMedomics, the only test evaluated by the two teams, was more than 99%, against 87% for the San Francisco team.
However, other experts were skeptical about the scoring approach. “It’s not really a method that would give you real quantification,” said Florian Krammer of the Icahn School of Medicine at Mount Sinai in New York.
Dr. Krammer developed a two-step Elisa test that he says has 100% specificity and provides a measure of the amount of IgM and IgG antibodies a person has. Noting the strips for a rapid test could provide data for a scientific study, he said, “but I would not make any decisions based on that. “
Dr. Krammer said that false positives are less of an issue in assessing the extent of the spread of the virus in the population. If a test has a known false positive rate, scientists can factor it into their calculations, he said.
But false positives become dangerous when developing personal policies and decisions about who can return to work. “You don’t want anyone going back to work who has a false positive – it’s the last thing you want to do,” said Dr. Krammer.
Scanwell Health, a Los Angeles-based start-up, ordered millions of test kits from Innovita, a Chinese manufacturer, and asked the Food and Drug Administration to market the tests for home use.
In the new study, the Innovita test detected antibodies in 83% of those infected and gave a false positive rate of 4%.
Dr. Jack Jeng, chief medical officer of health for Scanwell Health, said the study was based on an earlier version of the Innovita test, not the “new improved version” ordered by his company. “It will be interesting to see how it works,” he said.
Dr. Marson and his colleagues have acquired tests from nearly 100 manufacturers and plan to continue to compare them. The scientists also hope to expand their sample to include people who are slightly sick or do not feel sick at all, and stratify their data by age and the presence of chronic diseases.
“This is just the start,” said Dr. Marson. “Our goal would be to continue until we feel there is sufficient supply on the market. “