|Excerpt of the presentation|
Maged N. Kamel Boulos
Educational objective: At the conclusion of this presentation, participants should be able to appreciate that different evaluation methods for Web-based medical/ health information services have different strengths and weaknesses and so are not completely effective when used alone, and that only a good multi-method evaluation approach can help put together an adequate, more complete picture of how a service is being used and received by users. (Evaluation methods include questionnaires, semi-structured interviews, video tracking of a sample of users in a usability lab, analysis of Web server transaction logs, and expert panel critique using a standard checklist of usability heuristics. The choice of the specific methodologies used for a particular evaluation exercise depends on the resources (human, financial, time, etc.) available for the evaluation, among other things.)
Study design : A two-method formative evaluation approach was adopted using an online user questionnaire in addition to HCM server log analysis, guided by NIH Web Site Evaluation and Performance Measures Toolkit methodologies among others. The two chosen methods act in a complementary synergistic way, providing a more complete picture about HCM and crosschecking each other (e.g., to verify that the sample of questionnaire respondents is a good representative of the larger set of users logged in server logs).
Methods: HCM evaluation questionnaire covers service usability, usefulness, and user satisfaction issues. It includes a usability task, and a comparative task, comparing HCM to another approach. Both tasks are assessed qualitatively. Thirty-five subjects responded to the questionnaire during the 45-day period from 18 April-1 June 2002. Sawmill 6.3.8 was selected to analyse HCM server logs covering the same 45-day period of this study. Some service usage details derived from server log analysis, e.g., information about top referring search engines/ sites, the geographical provenance of service visitors and the most common browser types/ versions used to access the service, might be more accurate than similar information obtained from a user questionnaire, because of the much larger number of users covered in server logs. Server transaction logs can also provide valuable quantitative information about server traffic and performance, and visitor retention or "stickiness" among other things.
Conclusions: Using more than one method for the evaluation of a Web-based
health information service can help putting together an adequate, more
complete picture of how the service is being used and received by users
(compared to results obtained using only a single evaluation method).