Evaluating the Effects of Telemedicine on Quality,
Access and Cost
From ‘Telemedicine’, Institute of Medicine, 1996
Table: Categories of Evaluation Questions for Comparing
Telemedicine to Alternative Health Services |
1.
What are the effects of the
application on the clinical process of care compared to the alternative(s)? 2. What were the effects of the application on patient status or health outcomes compared to the alternative(s)? 3. What are the effects of the application on access compared to the alternative(s)? 4. What are the costs of the application for patients, private or public payers, providers and other affected parties compared to the alternative(s)? 5.
How did patients, clinicians
and other relevant parties view the application and were they satisfied with
the application compared to the alternative(s) |
Note: Each question assumes that an
analysis of results will control for or take into account severity of
illness, comorbidities, demographic characteristics and other relevant
factors |
Table: Evaluating Quality of Care and Health Outcomes |
What are the effects of the
Teleemdicine application on the clinical process of care compared to the
alternative(s) |
1.
Was the application
associated with differences in the use of the health services (e.g. office
visits, emergency transfers, diagnostic tests, length of hospital stay)? 2. Was the application associated with differences in appropriateness of services (e.g. underuse of clearly beneficial care)? 3. Was the application associated with differences in the quality, amount or type of information available to clinicians or patients? 4. Was the application associated with differences in patients’ knowledge of their health status, their understanding of the care options or their compliance with care regiments? 5. Was the application associated with differences in diagnostic accuracy or timeliness, patient management decisions or technical performance? 6.
Was the application
associated with differences in the interpersonal aspects of care? |
What were the effects of the
telemedicine application on immediate, intermediate or long term health
outcomes compared to the alternative(s)? |
1.
Was the application
associated with differences in physical signs or symptoms? 2. Was the application associated with differences in morbidity or mortality? 3. Was the application associated with differences in physical, mental or social and role functioning? 4. Was the application associated with differences in health-related behaviours (e.g. substance abuse)? 5.
Was the application
associated with differences in patient satisfaction with their care or
patient perceptions about the quality or acceptability of the care they
received? |
Note: Each question assumes that an
analysis of results will control for or take into account severity of
illness, comorbidities, demographic characteristics and other relevant
factors |
Table: Evaluating Access to Care |
Did Telemedicine affect the use of services
or the level or appropriateness of care to the alternative(s) |
1.
What was the utilisation of
telemedicine services before, during and after the study period for target
population and clinical problem(s)? 2. When offered the opinion of a telemedicine service, how often did patients · accept or refuse an initial service or fail to keep an appointment · accept or refuse a subsequent service or fail to keep an appointment? 3. What was the utilisation of specified alternative services before, during and after the study period for the target population and clinical problem(s)? · consultants travelling to distant sites · patients travelling to distant consultants · consultation by mail or courier · transfer to other facilities · self care 4. Was the telemedicine application associated with a difference in overall utilisation (e.g. number of services or rate) or indicators of appropriateness of care for · speciality care · primary care · transport services ·
services associated with lack
of timely care? |
Did the application affect the
timeliness of care or the burden of obtaining care compared to the
alternative(s)? |
1.
Was the difference in the · timing of care · appointment waiting times for referrals 2. What were patient attitudes about the · timeliness of care · burden of obtaining care · appropriateness of care
3. What were the attitudes of attending and consulting physicians and other personnel about the · timeliness of care · burden of providing care · appropriateness of care? |
Note: Each question assumes that an analysis
of results will control for or take into account severity of illness,
comorbidities, demographic characteristics and other relevant factors |
Table: Evaluating Health Care Costs and Cost-Effectiveness |
What were the costs of the
telemedicine application for participating health care providers or health
plans compared to the alternative(s) |
1.
Was an application associated
with differences in attending clinicians’ costs for personnel, equipment,
supplies, administrative services, travel or other items? Was an application
associated with differences in revenues or productivity? What was the net
effect? 2. Was an application associated with differences in consulting clinicians’ or consulting organisations’ costs for personnel, equipment, supplies, space, administrative services, travel or other items? Was an application associated with differences in revenues or productivity? What was the net effect? 3. Was an application associated with differences in the cost per service, per episode of illness or per member (health plan enrolee, capitated lives) per month? |
What were the costs of the
telemedicine application for patient and families compared to the
alternative(s)? |
1.
What was the application
associated with differences in direct medical costs for patients of families? 2. What was the application associated with differences for patients or families in other direct costs (e.g. travel, child care) or indirect costs (e.g. lost work days)? |
What were the costs for the society
overall compared to the alternative(s)? |
1.
Was an application associated
with differences in total health care cost, the cost per service, per episode
of illness or per capita? |
How did the costs of the application relate
to the benefits of the telemedicine application compared to the
alternative(s)? |
Note: Each question assumes that an
analysis of results will control for or take into account severity of
illness, comorbidities, demographic characteristics and other relevant
factors |
Table: Evaluating Patient Perceptions |
Were patients satisfied with the
telemedicine service compared to the alternative(s) |
1.
How did patients rate their
physical and psychological comfort with the application? 2. How did patients rate the convenience of the encounter, its duration, its timeliness and its cost? 3. How did patients (and family members) rate the skills and personnel manner of the consultant and the attending personnel (e.g. primary care physician, nurse practitioner)? 4. Was the lack of direct physical contact with the distant clinician acceptable? 5. How did patients rate the explanations provided to them of what their problem was and what was being recommended? 6. Did patients have concerns about whether the privacy of personal medical information was protected? 7. Would patients be willing to use the telemedicine service again? 8. Overall, how satisfied were patients with the telemedicine service they received? |
Note: Each question assumes that analysis
of results will control for or take into account prior experiences with the
health care system, severity of illness, comorbidities, demographic
characteristics and other relevant factors |
Table: Desirable Attributes of Evaluation Criteria |
Reliability / Reproducibility An evaluation instrument or criterion is reliable if repeated use
under identical circumstances by the same or different users produces the
same results Validity An evaluation instrument or criterion is valid if it measures the properties, qualities or characteristics it is intended to measure Responsiveness An evaluation instrument or criterion is responsive if it can detect important differences in outcomes across evaluation groups or time periods Interpretability An evaluation instrument or criterion is interpretable if users find the results of its application understandable Feasibility An evaluation instrument or criterion is feasible if users can accomplish the required activities, collect the necessary information and analyse the resulting data within available evaluation resources and without imposing excessive burdens on those whose co-operation is required for the evaluation Flexibility An evaluation instrument or criterion is flexible if it is adaptable to a variety of evaluation problems and circumstances Documentation An evaluation instrument or criterion is documented if the protocols for applying and interpreting it are specified and if evidence of its successful use is summarised or cited |
Elements of an Evaluation Plan |
Project description and research
question(s): the application or programme to be
evaluated and the basic questions to be answered by the evaluation Strategic objectives: how the project is intended to serve the sponsor or parent organisation’s purposes Clinical objectives: how the telemedicine project is intended to affect individual or population health by changing the quality, accessibility or cost of care Business plan or project management plan: a formal statement of how the evaluation will help decision makers judge whether and when the application will be a financially and otherwise sustainable enterprise or less formally, what the project’s management, work plan, schedule and budget will be Level and perspective of evaluation: whether the focus of the research question(s) and objectives id clinical, institutional, societal or some combination Research design and analysis plan: the strategy and steps for developing valid comparative information and analysing it Experimental and comparison groups: characteristics of (a) the group or groups that will be involved in testing the target telemedicine application and (b) the group or group that will receive alternative services for purposes of comparison Technical, clinical and administrative process: as planned and actually implemented, the communications and information systems, the methods for providing medical care and the supportive organisational processes Measurable outcomes: the variables and the data to be collected to determine whether the project is meeting its clinical and strategic objectives Sensitivity analysis: the inclusion of techniques to assess to what extent conclusions may change if assumptions or values of key variables changed Documentation: the explicit reporting of the methods employed in the evaluation and the findings so that others can determine how the results were established |
Categories of Evaluation Questions for Comparing Telemedicine to
Alternative Health Services |
1.
What were the effects of the
application on the clinical process of care compared to the alternative(s)? 2. What were the effects of the application on patient status or health outcomes compared to the alternative(s)? 3. What were the effects of the application on access compared to the alternative(s)? 4. What were the costs of the application for patients, private or public payers, providers and other affected parities compared to the alternative(s)? 5. How did patients, clinicians and other relevant parties view the application and were they satisfied with the application compared to the alternative(s) |
Note: Each question assumes an analysis
of results will control for severity of illness, comorbidities, demographic
characteristics and other relevant factors |