The useful question is not “Where can we add AI?” It is “Which defined decision or workflow problem is important enough to test, and what evidence would make the new approach safer or more useful than the current one?”
1. What is the intended use?
Name the user, setting, input, output and decision. A tool that summarises internal documents is different from one that predicts injury risk or recommends patient-specific action. Vague intended use makes meaningful evaluation impossible.
2. Is AI necessary?
Compare the proposal with a checklist, a rules-based system, better data collection and no change. A simpler method may be cheaper, clearer and easier to govern.
3. Do we have lawful, representative data?
Data access, purpose limitation and minimisation come before model performance. Club datasets may be small, selected and shaped by changes in staff, competition, equipment and recording practice.
4. Could information leak from the future?
Randomly splitting repeated observations from the same athlete can place near-duplicate information in development and evaluation sets. Time-aware partitions, athlete-level separation where appropriate and explicit leakage checks are essential.
5. What is the baseline?
Compare against the existing workflow and simple alternatives. A high headline metric does not show that the tool improves a real decision, reduces workload or transfers to another squad.
6. Which errors matter?
Overall accuracy can hide clinically or operationally important failures. Define false-positive and false-negative consequences, calibration, uncertainty and relevant subgroup performance before evaluation.
7. Who remains accountable?
Human oversight must be an operating process, not a sentence in a policy. Define who reviews output, when it can be overridden, how disagreement is recorded and when the tool must stop.
8. How will the system change?
Data pipelines, populations and software versions drift. Version models and datasets, monitor agreed indicators and require review before retraining or expanding intended use.
9. What regulatory and privacy route applies?
The UK's Medicines and Healthcare products Regulatory Agency notes that many health-software and AI applications will be regulated as medical devices. The Information Commissioner's Office provides guidance on applying UK GDPR principles and explaining AI-assisted decisions. Classification depends on the specific intended purpose and cannot be inferred from the word “AI” alone.
10. What evidence would make us stop?
Predefine failure criteria: inadequate data quality, unstable performance, poor calibration, unacceptable workflow burden, security weaknesses or benefits that disappear under realistic evaluation. A responsible project can conclude that deployment is not justified.
Sources and further guidance
- Information Commissioner's Office. Artificial intelligence and data protection guidance.
- Medicines and Healthcare products Regulatory Agency. Software and AI as a Medical Device Change Programme.
- National Institute for Health and Care Excellence. Evidence standards framework for digital health technologies.
Sources checked 8 September 2026. Requirements and guidance can change; organisations should verify the current position with appropriate specialists before deployment.
Defining a medical AI or software project?
Start with the intended workflow, evidence requirements and failure boundaries.