Vaccine misinformation and bad data

Written by: Rosie Lobley

On 24 June 2021, MDPI Vaccines published an article that made some staggering claims about the safety of COVID-19 vaccines. The study, titled ‘The safety of COVID-19 vaccinations – we should rethink the policy’, was authored by psychologist Dr Harald Walach, medical physicist Dr Rainer J Klement and data analyst Wouter Aukema. It alleged that, for every three deaths from COVID-19 that were prevented by vaccination, it would be necessary to accept two deaths caused by vaccination as an unavoidable factor.

A number of readers pointed out deeply entrenched flaws within the article, which appeared to be drawing the wrong conclusions by taking the wrong data wildly out of context. Multiple members of the journal’s editorial board resigned in protest. Exasperated epidemiologists and mathematicians took to Twitter to outline the study’s faults and query the motivations of its authors. On 28 June, four days after the article’s publication, MDPI Vaccines published an Expression of Concern, highlighting ‘serious concerns’ over the study’s claims and pledging to investigate the issues further. The full article was retracted on 2 July.

Statistical faults

To determine the number needed to vaccinate (NNTV) to prevent one death from COVID-19, the authors used data from an Israeli cohort study with around a million subjects. If their interpretation of these data is to be believed at face value, the NNTV (1 divided by the absolute risk difference between vaccinated and unvaccinated groups) is shown to be 16 667 – that is, to prevent a single death from COVID-19, at least 16 667 people must receive the vaccine.

The provenance of the data used to calculate the NNTV meant that it was functionally impossible to do so with any great accuracy. The Israeli study took place over a period of six weeks, during which only 3% of the country’s population became infected with COVID-19; by comparison, 30% have contracted the virus at some point over the last 12 months. Extrapolation of the figures used to determine the absolute risk reduction – a key factor in identifying the NNTV – over a theoretical 52-week period produces an NNTV closer to 1,960.

The study used data on all deaths occurring after (but not necessarily caused by) COVID-19 vaccination. The data were taken primarily from the Dutch National Register of adverse drug reactions (ADRs), which explicitly states on its website: ‘Death after vaccination does not mean that a side effect of the vaccine is the cause of death.’ ADR reporting programmes are specifically designed to catch potential side effects by encouraging users to submit any adverse events that may occur following treatment, so that medical professionals can investigate further to determine whether they are fully or partially caused by the treatment. The data could have been further skewed by the fact that the vaccine was first offered to patients who were elderly or clinically vulnerable, and who may be more likely to experience medical events unrelated to the vaccine itself.

The authors

It may be considered reasonable to expect research into the risks and benefits of COVID-19 vaccination to feature input from virologists, epidemiologists or biostatisticians. Dr Walach, the lead author of the study, is a psychologist with a documented interest in both complementary medicine and parapsychology. In 2012, he was awarded the Goldene Brett vorm Kopf (‘Golden Blockhead’), a prize for the year’s ‘most astonishing pseudo-scientific nuisance’, by the German Society for the Scientific Investigation of Pseudosciences, in recognition of a master’s thesis attempting to prove the legitimacy of clairvoyance. Aukema and Dr Klement are members of a ‘consortium’, led by molecular biologist and Young Earth creationist Dr Pieter Borger, which appears to be dedicated to debunking the efficacy of polymerase chain reaction (PCR) tests for COVID-19.

Peer review

The study passed through two rounds of peer review with three reviewers with minimal criticism – only one reviewer noted: ‘it is unclear why the study has been performed by comparing NNTV obtained from Israeli study with the side effects reported in the European Medicines Agency and of the Dutch National Register. In my opinion, due to economical and social differences, the data should be calculated using the local register.’

This raises further questions. How assiduously was the paper reviewed? Did the reviewers assume that the figures the authors used were correct, and were they right to do so? Might it be considered good practice to Google the author of a study when considering it for publication? Could the MDPI series’ pay-to-publish model discourage in-depth criticism of a client’s content?

The bigger picture

Misinformation surrounding the COVID-19 pandemic as a whole, and vaccines in particular, is widespread. MDPI Vaccines is now investigating the process that led to the publication of the Walach et al. study. However, its Expression of Concern – and, it seems probable, the retraction of the article – will be unlikely to reach many readers of the original study. The authors’ claims have already gained significant traction in social media spaces geared towards anti-vaccination and COVID-sceptical users. The proliferation of misinformation around the efficacy and purported risks of vaccines may be actively harmful; and any damage caused cannot easily be undone.

At Spirit, we know that the spread of scientific misinformation is a particularly pressing concern in a time when we all need reliable, up-to-date information about COVID-19. We believe in accessible, accurate, evidence-based communication. That’s why we’ve developed a COVID-19 Vaccine Tracker, which is regularly updated with information on the development and rollout of COVID-19 vaccines, as well as a bit about the science that underpins them.

This article was updated on 6 July to reflect MDPI Vaccines‘ retraction of the Walach et al. study.