WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data

Summary

This paper introduces WearableQA, a novel benchmark designed to assess AI's ability to reason over real-world, longitudinal wearable data, comprising 4,084 questions that challenge models on data interpretation and physiological understanding, revealing significant performance gaps in current LLMs.

Key findings

• WearableQA effectively differentiates the reasoning capabilities of various LLMs.
• Most evaluated LLMs achieve accuracies below 60% on WearableQA, indicating significant room for improvement.
• The benchmark's design captures realistic wearable data distributions, including noise and inter-individual variability.
• The dual-grounding framework enables the creation of meaningful and reliable health-related questions.

Methods

• Constructed a benchmark (WearableQA) with 4,084 questions from 200 users' wearable data, blood biomarkers, and demographics.
• Organized 16 question types into data vs. health and single- vs. cross-signal reasoning axes.
• Employed a dual-grounding framework (literature and statistical) for reliable question generation.
• Evaluated 14 LLMs on WearableQA to assess their reasoning capabilities.

Limitations

• The benchmark is constructed from a specific user cohort, which may not generalize to all populations.
• The evaluation focuses on multiple-choice questions, potentially limiting the assessment of open-ended reasoning.
• Performance of LLMs may be influenced by the specific question generation methodology.
• The benchmark does not include real-time intervention or feedback loops.

Applications

• Evaluating AI models for health monitoring and diagnostics using wearable data.
• Developing more robust AI systems capable of understanding complex physiological signals.
• Benchmarking progress in AI reasoning for personalized health applications.
• Guiding future research in wearable data analysis and interpretation.