It's being reported that a Harvard biostatistician is rethinking plans to use Apple Watches as part of a research study after finding inconsistencies in the heart rate variability data collected by the devices. Because Apple tweaks the watch’s algorithms as needed, the data from the same time period can change without warning.

JP Onnela, associate professor of biostatistics at the Harvard T.H. Chan School of Public Health and developer of the open-source data platform Beiwe, told The Verge: "These algorithms are what we would call black boxes — they’re not transparent. So it’s impossible to know what’s in them."

The report by The Verge noted that Onnela doesn’t usually include commercial wearable devices like the Apple Watch in research studies. For the most part, his teams use research-grade devices that are designed to collect data for scientific studies. As part of a collaboration with the department of neurosurgery at Brigham and Women’s Hospital, though, he was interested in the commercially available products. He knew that there were sometimes data issues with those products, and his team wanted to check how severe they were before getting started.

So, they checked in on heart rate data his collaborator Hassan Dawood, a research fellow at Brigham and Women’s Hospital, exported from his Apple Watch. Dawood exported his daily heart rate variability data twice: once on September 5th, 2020 and a second time on April 15th, 2021. For the experiment, they looked at data collected over the same stretch of time — from early December 2018 to September 2020.

Because the two exported datasets included data from the same time period, the data from both sets should theoretically be identical. Onnela says he was expecting some differences. The 'black box' of wearable algorithms is a consistent challenge for researchers. Rather than showing the raw data collected by a device, the products usually only let researchers export information after it has been analyzed and filtered through an algorithm of some kind.

Companies change their algorithms regularly and without warning, so the September 2020 export may have included data analyzed using a different algorithm than the April 2021 export. 'What was surprising was how different they were,' he says. 'This is probably the cleanest example that I have seen of this phenomenon.' He published the data in a blog post last week.

There might be some situations where wearable data could still be useful. The heart rate variability information showed similar trends at both time points — the data went up and down at the same times. Olivia Walch, a sleep researcher who works with wearable and app data at the University of Michigan stated that 'If you’re caring about stuff on that macro scale, then you can make the call that you’d keep using the device.' Walch added: 'But if the specific heart rate variability calculated on each day matters for a study, then the Apple Watch may be riskier to rely on. 'It should give people pause about using certain wearables, if the rug runs the risk of being ripped out underneath their feet.'"

Apple did not respond to a request for comment. For more on this including an audio file, check out the report by The Verge.