Stanford University Medical Center just released a study showing how well fitness trackers work. They tested seven brands of products that are worn on the wrist: the Apple Watch, Fitbit Surge, Basis Peak, Microsoft Band, Mio Alpha 2, PulseOn and the Samsung Gear S2, measuring heart rate and energy (calories) consumed during use.
While most of the devices tracked heart rate accurately, their tracking of calories was significantly off compared to standard medical measurements. The test consisted of having 60 people, both men and women over a range of ages, use these devices while walking, biking and running. They then compared the results using standard medical equipment that measures oxygen and carbon dioxide and by taking electrocardiograph readings.
Six of the devices did an excellent job of measuring heart rate and were accurate to within five percent. However, none of them accurately measured calories consumed, with errors ranging from 27 percent to a whopping 93 percent for the PulseOn. They underestimated the calories burned during less stressful exercises and overestimated the calories consumed when the testers did more strenuous exercises. The authors of the study surmised that it’s very difficult to develop an accurate measurement for calories or energy burned with the variety of users’ weights and heights.
Which devices did the best? The Apple Watch and Fitbit, the best selling devices in this category; each had errors of less than 20 percent for calories burned. Next in line were the Basis Peak and Microsoft Band, but each has been discontinued. The Mio Alpha 2 and Samsung Gear S2 weren’t included in the calorie testing because they didn’t provide measurements every minute.
Fitbit commented on the study saying, “Overall, the success of Fitbit products comes from enabling people to see their overall health and fitness trends over time. It’s these trends that matter most in achieving goals.”
PulseOn responded to their poor results by saying that the researchers may not have properly inputted the user information that includes height, weight and fitness level. “We think the excess error reported in (energy expenditure) is not representative in this study, due to this methodological error,” their CEO said in an email.
Why is accuracy important in counting calories? Dr. Tim Church, professor of preventative medicine at Pennington Biomedical Research Center at Louisiana State University, who wasn’t involved in the study, told NPR, “It’s just human nature. People are checking these inaccurate counts and they think they’ve earned a muffin or earned some ice cream and they’re sabotaging their weight-loss program.”
“For a lay user, in a non-medical setting, we want to keep that error under 10 percent,” Anna Shcherbina, a Stanford graduate student, and study co-author said. One of the key issues, Shcherbina hypothesized, was the difference in users’ body compositions. She noted, “It’s very hard to train an algorithm that would be accurate across a wide variety of people because energy expenditure is variable based on someone’s fitness level, height and weight, etc.” She also pointed out that it’s much easier to measure heart rate because it’s measured directly and not based on using an algorithm that makes assumptions about the user.
What was not measured, and is even more important to some than calories burned, is how accurate the trackers count steps or distance walked. In a review late last year by Lauren Goode of The Verge, she found that the Fitbit Charge2 had “notable discrepancies in distances tracked compared to GPS sports watches. A one-mile walk was recorded as 0.71 miles on the Charge 2. A 3-mile run was recorded as 2.13 miles. A 3.6-mile hike was recorded as 3.3 miles.
There are no industry standards for acceptable accuracy for this popular category of devices, so you’re on your own. Experts and manufacturers say that the best way to use these devices is to measure relative changes, striving to do better over time by monitoring how the numbers improve.
Join the conversation as a VIP Member