• Users Online: 235
  • Home
  • Print this page
  • Email this page
Home About us Editorial board Ahead of print Browse Articles Search Archives Submit article Instructions Subscribe Contacts Login 


 
 Table of Contents  
ORIGINAL ARTICLE
Year : 2017  |  Volume : 8  |  Issue : 1  |  Page : 87

Promoting the quality of health research-based news: Introduction of a tool


1 Knowledge Utilization Research Center, Department of Health Education and Promotion, School of Public Health, Tehran University of Medical Sciences, Tehran, Iran
2 Knowledge Utilization Research Center, Department of Epidemiology and Biostatistics, School of Public Health, Tehran University of Medical Sciences, Tehran, Iran
3 Knowledge Utilization Research Center, Tehran University of Medical Sciences, Tehran, Iran

Date of Submission08-Jan-2017
Date of Acceptance24-Jun-2017
Date of Web Publication01-Nov-2017

Correspondence Address:
Mahnaz Ashoorkhani
Knowledge Utilization Research Center, PhD in Health Education and Promotion Unit 7, 7th Floor, No. 1547, North Karegar Ave., Enghelab Sq., Tehran
Iran
Login to access the Email id

Source of Support: None, Conflict of Interest: None


DOI: 10.4103/ijpvm.IJPVM_16_17

Rights and Permissions
  Abstract 


Introduction: While disseminating health research findings to the public, it is very important to present appropriate and accurate information to give the target audience a correct understanding of the subject matter. The objective of this study was to design and psychometrically evaluate a checklist for health journalists to help them prepare news of appropriate accuracy and authenticity. Methods: The study consisted of two phases, checklist design and psychometrics. Literature review and expert opinion were used to extract the items of the checklist in the first phase. In the second phase, to assess content and face validity, the judgment of 38 persons (epidemiologists with a tool production history, editors-in-chief, and health journalists) was used to check the items' understandability, nonambiguity, relevancy, and clarity. Reliability was assessed by the test–retest method using intra-cluster correlation (ICC) indices in the two phases. Cronbach's alpha was used to assess internal validity of the checklist. Results: Based on the participants' opinions, the items were reduced from 20 to 14 in number. The items were categorized into the following three domains: (a) items assessing the source of news and its validity, (b) items addressing the presentation of complete and accurate information on research findings, and (c) items which if adhered to lead to the target audiences' better understanding. The checklist was approved for content and face validity. The reliability of the checklist was assessed in the last stage; the ICC was 1 for 12 items and above 0.8 for the other two. Internal consistency (Cronbach's alpha) was 0.98. Discussion and Conclusions: The resultant indices of the study indicate that the checklist has appropriate validity and reliability. Hence, it can be used by health journalists to develop health research-based news.

Keywords: Checklist, health news, medical journalism, reliability, validity


How to cite this article:
Ashoorkhani M, Majdzadeh R, Nedjat S, Gholami J. Promoting the quality of health research-based news: Introduction of a tool. Int J Prev Med 2017;8:87

How to cite this URL:
Ashoorkhani M, Majdzadeh R, Nedjat S, Gholami J. Promoting the quality of health research-based news: Introduction of a tool. Int J Prev Med [serial online] 2017 [cited 2022 May 24];8:87. Available from: https://www.ijpvmjournal.net/text.asp?2017/8/1/87/217594




  Introduction Top


The delivery of health research knowledge to the public through mass media is particularly an important and sensitive issue. Multiple evidences suggest the impact mass media has on changing health behavior, going so far as altering the methods of treatment.[1]

Several studies have underscored the inappropriate quality of health news. This problem exists in developed countries too,[2],[3],[4],[5] and shortcomings and exaggerations have been reported in medical and health news reports.[5],[6],[7] Many factors at multiple levels affect the quality of the news during its production process. Factors at individual level, media and organizational level, extra-media factors (such as economic, social, and political factors) have been recognized.[2],[8],[9]

At individual level, health journalists constitute the most important ring in the news production chain and greatly affect the quality of the news produced. Dearth of health knowledge, lack of familiarity with scientific knowledge and research methods, lack of familiarity with scientific terminology, and difficulties in finding and using valid sources are some of the barriers at individual level.[4],[7],[8],[10]

So far, academic educational courses have been held to train health journalists. Furthermore, short-term educational courses have been held to promote the journalists' level of knowledge.[11] In some countries, measures have been taken to control the quality of health news published, such as establishment of websites that assess disseminated news using certain indicators.[12] Such measures can encourage journalists to adhere to the items proposed for health news production.

Based on the available evidence, like many other countries, the quality of the news disseminated in Iran is undesirable.[2],[3],[4],[5] Bearing in mind the dearth of health and research knowledge among journalists and media personnel, the need to find solutions for improving and promoting their level of knowledge is felt. The objective of this study was to develop an appropriate checklist for health journalists to help them in producing health research news. In this way, the target audience will receive the necessary information, and the news will have a high level of accuracy and authenticity. The existent tools had certain limitations, so we tried to develop a practical tool for health journalists, in addition to assessing its reliability and validity. Items of this checklist are the items journalists need to consider in the stages of selection, production, and writing news to improve the quality of health news.


  Methods Top


Production of checklist

To identify domains and the required questions for each domain and for checklist design to produce news on the result of medical and health research, (1) literature review and (2) experts' views were used.

Google Scholar and PubMed were searched for similar checklists and instruments with the following keywords: ((health news OR medical news) AND (tools OR checklist OR guideline)). Henceforth, relevant articles were extracted and their items were used in the first draft [Appendix 1]. Since other studies have been carried out by the research team on the quality of disseminated news, the needs provided by health news producers (editors, editors-in-chief, and health journalists) were used in this checklist. Two appropriate tools were found and used. Oxman checklist consists of eight items on the evaluation of research studies that have been decided to produce news from. Another applied tool was Media Doctor website. Media Doctor uses this tool to evaluate disseminated medical innovation news (medication, equipment, etc.), scores each piece of released news on a scale ranging from half to 5 stars and put it on their website.

Initially, a seminar session was held with three physician epidemiologists with a tool production history as research experts, and a specialist in health education and promotion was used to evaluate the comprehensibility of concepts and sentences – three news producers, two editors-in-chief, and one health journalist – as media experts, wherein their opinions and judgments were used. The primary draft of the checklist was then prepared which consisted of twenty items. All the participants in the seminar were asked to mention any items they believe are necessary to be included in the checklist in addition to the existing twenty items.

Validity

The content and face validities of the primary draft were qualitatively examined. Here, we used the opinions of 38 experts in the field of health news and checklist production (13 editors-in-chief, 19 health journalists, 1 health news desk translator, and 5 methodologists [with a tool production history]). A total of 26 sessions (23 interviews and 3 focus group discussions) were held to collect their opinions. Participants were handed out a checklist of twenty items. Each item was presented and discussed. Participants examined each question in terms of wording, proportion, logical sequence, appropriateness and transparency of selected items, and declared their views. After applying the recommended changes, the number of items was reduced from 20 to 14.

Then, the checklist was again sent to six experts to acquire their opinions about the relevancy and clarity of the items quantitatively. Regarding the appropriateness of the question, the following questions were asked: “In your opinion, to what extent this question is relevant to the subject of improving the quality of health news (results of health studies)?” and “in other words, how much the content of the question is relevant and appropriate?” To assess the clarity of the question, they were asked: “How much is the question clear?” In addition, comprehensiveness of the questionnaire was evaluated by a question. The relevancy and clarity of each item were assessed using a 4-item Likert scale that ranged from 1 (not relevant/clear) to 4 (completely relevant/clear). The inter-rater agreement (IRA) and item-content validity index (I-CVI) were calculated to evaluate relevancy and clarity.

First, the IRA index was examined. The items that were approved by all experts as appropriate and necessary for the “news quality improvement index” were preserved and the remaining were removed. Both options “3” (relevant/clear) and “4” (completely relevant/clear) were considered as agreements. Items that scored >80% were accepted as relevant/clear items. The items that scored lower were reviewed and modified. Finally, the checklist was assessed and approved by two epidemiologists who have a history of tool production, and one expert in health promotion and education and another health editor-in-chief assessed the checklist and applied participants' feedbacks and comments in it.

The comprehensiveness of the checklist was evaluated in the form of a 4-item Likert scale question. The scale-CVI of the entire checklist's comprehensiveness was approved by 80% of the respondents. If >80% chose the number 3 or 4 regarding the question's comprehensiveness, that question was accepted in this respect.

Reliability

The test–retest method was used to determine the checklist's reliability. In the first round, 12 Persian medical articles were chosen and a health journalist was asked to produce 12 news stories from those articles. These news stories, the relevant articles, and the checklist were handed out to eight individuals. They were asked to assume that this primary news draft had been written by them and then compare it with the checklist to find if the items were included and finally mark the answer to each question based on the checklist and its guide. This was done twice at a 2-week interval. After data collection, SPSS Statistics for Windows,(Statistical Package ver. 17.0; SPSS Inc, Chicago, IL, USA), was used to assess the IRA, applying the intra-cluster correlation (ICC) index. Cronbach's alpha was used to assess the internal consistency of the checklist. After data analysis, Kappa scores which were >0.7 were accepted. The items scored <0.7 were modified using the individuals' comments and after the advisory meeting of the research team. Moreover, the option of “No Applicable” was added to the response section in some of the questions. Having done these modifications, the retest was performed on another group of 8 individuals after 2 weeks using the 12 previous news stories. At this stage, each individual's ICC acquired in two steps of the test was calculated. Among the eight individuals, the two who had the highest IRA score in both rounds were chosen. The ICC was then calculated for each item between these two individuals.


  Results Top


This checklist has been developed for health news journalists. Its application will result in controlling certain items of health research news production that can improve the quality of the news. Fourteen items were categorized into the following three domains: (a) items assessing the source of news and its validity (screening question), (b) items addressing the presentation of complete and accurate information on research findings (items 2–10), and (c) items which if adhered to lead to the target audiences' better understanding (items 11–14). The items are as follows: (1) validity, (2) precision, (3) magnitude, (4) prominence of scientific findings, (5) applicability, (6) consequences, (7) consistency, (8) conflict of interests, (9) treatment options, (10) availability, (11) news headline, (12) level of complexity of the news report content, (13) providing the specifications of the news source, and (14) avoiding creation of fear in the community. Each item has a “hint” that helps the journalist better understand the item. The response options include “yes,” “somewhat,” “no,” and in some questions, “not applicable.” Wherever a question's response is one other than “yes,” instructions are given on how to make corrections [Appendix 2].

Validity

In the qualitative phase, 14 of the 20 items remained in the final draft. The measures taken such as omission, addition, and conceptual modification in some items led to the development of a checklist with 14 items [Appendix 2].

In the quantitative phase, the 14 items were assessed for clarity and relevancy by 6 persons. Four items and seven items scored <0.8 for relevancy and clarity, respectively. Therefore, after corrections were made, clarity and relevancy were evaluated once more by another 9 persons in the second round. The second round's results are summarized in [Table 1]. The comprehensiveness of the checklist too was assessed by a single question. All the respondents (100%) rated the checklist as comprehensive “3” and completely comprehensive “4.”
Table 1: Clarity and relevancy for each question (item-content validity index)

Click here to view


Reliability

In the first round, the items that had an ICC <0.7 were modified and clarified according to the respondents' comments. Given the applied changes, the reliability of the checklist was re-assessed in the second round.

During the second round, the same 12 news stories and articles from which the news had been extracted were assessed by 8 other health journalists 2 weeks later. At this stage, ICC was calculated for each person by test–retest method. Two persons were selected among the eight, and between-instrument ICC was calculated. The resultant ICC was 100% for most of the questions. These results have been summarized in [Table 2]. To assess internal consistency, Cronbach's alpha was calculated, which was 0.98. The effect of removing each question on the value of Cronbach's alpha showed that none of the items needed to be removed. These values have been summarized in [Table 2].
Table 2: Reliability indicator for each item in the second phase

Click here to view



  Discussion Top


In this study, we developed a checklist to guide health journalists in producing health research news. This checklist comprises 14 items in three sections. The first section assesses the source of the news and its validity (1 item). The second section includes items that assess the presentation of accurate and complete information on the research findings (9 items). The third section covers items that, if adhered to, can lead to the target audiences' better understanding and lessen the possibility of being misled (4 items). The checklist's validity was assessed both qualitatively and quantitatively. Its reliability was assessed through test–retest method.

The studies conducted by our research team indicate that, after taking the two criteria of accuracy and authenticity into account, 18% of health news stories were found inappropriate for dissemination in mass media. The qualitative study also concluded that health journalists face difficulties in creating health news as a result of inadequate knowledge of health issues and health research.[2] Therefore, it seems that preparing a checklist for production of health research-based news can help guide health news producers in producing health news, and eventually improve the quality of the news.

Similar tools have been developed elsewhere, but they are very few in number. Examples are those currently used in Canada, Australia, and Hong Kong, which assess the disseminated news stories on the basis of certain items in the news content. An example is “Media Doctor .” This tool can be applied to medical news (medicine, drugs, medical equipment, and diagnostic and therapeutic techniques). The items include: (1) discussion on costs, (2) evaluation of benefits, (3) evaluation of harms, (4) addressing other treatment options, (5) search for sources and keeping conflict of interests in mind, (6) avoiding magnification of certain natural conditions to encourage the purchase of a product, (7) investigating the quality of evidence, 8) novelty of the topic, (9) investigating the availability of the new method, and (10) “Appears not to rely solely or largely on a news release.”[13] This tool examines news after it is disseminated and in case if there are problems in it they cannot be edited. Moreover, there is no control over the evidence and source of news.

Another tool has been developed by Oxman et al. The seven criteria enlisted in this checklist are: (1) application, (2) validity of the document, (3) presentation of main findings and facts, not the researcher's personal opinions, (4) magnitude of effect, (5) statistical accuracy, (6) consistency with other study findings, and (7) statement of side effects. This checklist can help an individual who has adequate knowledge of research methodology to assess the research from the standpoint of the aforementioned criteria.[14] Using this tool is not easy for health journalists and demands a high level of research knowledge. In addition to the items presented in this checklist, other items were added to that as well. The strength of this checklist is its comprehensiveness. Different aspects affecting the correctness, accuracy, and quality of the news item have been paid attention to including news validity in the form of news source evaluation. On the one hand, factors affecting the quality and comprehensiveness of the news item have been regarded and on the other hand the aspect of writing the text of the item has been considered in order for target audience to get a proper understanding of the news and also for its successful transfer. Moreover, this checklist is not merely allocated to the news of medical and pharmaceutical innovations and includes all the news concerning research results and health and medical innovations and has a wide range of functions although it has merely been used for medical interventions, drugs, diagnostic tests, surgery, etc. in Media Doctor indices. In addition, each item has a hint that helps the journalist use the checklist properly. The response options include “No Applicable” in some questions which can create more flexibility whenever the checklist is used for different health and medical subjects. This tool can also be used during writing and production process and evaluates news items before they are published. Hence, it can contribute to producing high-quality news stories.

The current checklist has 14 items (in 3 sections) that help news producers develop news content before and during the process of news production. Given the instructions, the journalist can describe each item in the best possible way. Moreover, s/he can assess the news story, score it, and correct it before dissemination if need be. The content and indices used in the aforementioned checklists were used. However, they had certain limitations such as “difficulty of use among journalists with low level of research knowledge” and “overlooking news evidence and merely assessing the news after publication.” Hence, certain measures were taken to remove these limitations during the development of the current checklist. It guides journalists during the production process and helps them edit the draft before it is published. Furthermore, since most journalists do not have a high level of research knowledge, statistical and research concepts are explained in a simpler manner. To better explain the items to those using the checklist, a “hint” and an “example” have been added to each item. The latter two have been added to help users better answer the questions. Based on the score given in each item, the news story is edited and improved if need be. However, there are some restrictions on the use of the checklist of which it can be pointed out that it is often used to cover health and medical research news, and for the news resulted from sources other than research studies, such as news about health advice and educational recommendations, its use will be more limited and less effective.

The validity and authenticity of the news source is the most important point that cannot be overlooked and compensated by any other item, and it can alone determine if the news is appropriate for dissemination or not. The first section of the checklist addresses this issue. In this section, instructions have been given on how to develop news stories on the basis of different sources (articles and research documentations, interviews, and websites). Certain criteria have been set to examine each of the aforementioned sources. If this item is taken into account, it may prevent the dissemination of many news stories of poor quality and evidence. The type of research study and the quality with which it is performed have been approved by others as well.[7],[10]

The second section of the checklist explains research points that can be best addressed while reporting research findings. Information on the accuracy of the study conducted, quantitative statement of findings, prominence of findings, the main target audience, statement of negative aspects of the research study, consistency of findings with other studies, possibility of conflict of interests, and other treatment options should be mentioned in the news body. The significance of these items has been described in other studies as well. These items are usually not adhered to by journalists, owing to their dearth of knowledge of health research methodology and statistics.[7],[10],[15] Other factors such as persons who financially and intellectually benefit from the dissemination of results can affect the authenticity and quality of health news. One study showed that, among the 306 drug-related news stories disseminated, 130 had not mentioned the sponsors.[16] This matter is important because its statement can rule out conflict of interests. Many studies have indicated that many times news stories have exaggerated the health findings.[17],[18] One study showed that news stories describe the effects of treatment in an excessively positive or utterly discouraging manner.[19] Hence, it seems that, by clearly describing these items in the news content, the status of health research-based news will improve.

The third section of the checklist addresses items that can help target audiences better understand the news published. Here, items such as “level of complexity of the news content” and “access to the main source of news” have been described. Since the ability of individuals to access the sources of health knowledge is a component of health literacy, describing the route of access to the news source has been included as an item in the checklist to achieve health literacy. James examined 1700 health news reports and observed that only a small number had mentioned the source of information used.[6],[20]

In our setting, we do not have specially trained health journalists. In the recent years, a small number of students have been enrolled in the “media and health discipline” at master's level to give specialized training to journalists. The low acceptance rate of students at this degree level and journalists' low level of health knowledge can lead to the applicability of this checklist in our context and many developing and even developed countries that have contexts similar to ours.


  Conclusions Top


The checklist prepared can prove helpful to health journalists in preparing health and medical research-based news. The application of this tool can prevent the intentional or unintentional overlooking of items that matter as health news quality indices. Therefore, it seems that health journalists can affect and promote the scientific quality of a major portion of news items.

[Additional file 1]

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.



 
  References Top

1.
Haas JS, Kaplan CP, Gerstenberger EP, Kerlikowske K. Changes in the use of postmenopausal hormone therapy after the publication of clinical trial results. Ann Intern Med 2004;140:184-8.  Back to cited text no. 1
    
2.
Ashorkhani M, Gholami J, Maleki K, Nedjat S, Mortazavi J, Majdzadeh R, et al. Quality of health news disseminated in the print media in developing countries: A case study in Iran. BMC Public Health 2012;12:627.  Back to cited text no. 2
    
3.
Cassels A, Lexchin J. How well do Canadian media outlets convey medical treatment information?: Initial findings from a year and a half of media monitoring by Media Doctor Canada. Open Med 2008;2:e45-8.  Back to cited text no. 3
    
4.
Voss M. Checking the pulse: Midwestern reporters' opinions on their ability to report health care news. J Inf 2002;92:1158-60.  Back to cited text no. 4
    
5.
Wilson A, Bonevski B, Jones A, Henry D. Media reporting of health interventions: Signs of improvement, but major problems persist. PLoS One 2009;4:e4831.  Back to cited text no. 5
    
6.
Molnar FJ, Man-Son-Hing M, Dalziel WB, Mitchell SL, Power BE, Byszewski AM, et al. Assessing the quality of newspaper medical advice columns for elderly readers. CMAJ 1999;161:393-5.  Back to cited text no. 6
    
7.
Miranda GF, Vercellesi L, Bruno F. Information sources in biomedical science and medical journalism: Methodological approaches and assessment. Pharmacol Res 2004;50:267-72.  Back to cited text no. 7
    
8.
Larsson A, Oxman AD, Carling C, Herrin J. Medical messages in the media – Barriers and solutions to improving medical journalism. Health Expect 2003;6:323-31.  Back to cited text no. 8
    
9.
Shoemaker PJ, Reese SD. Mediating the Message. New York: Longman White Plains; 1996.  Back to cited text no. 9
    
10.
Wilson A, Robertson J, McElduff P, Jones A, Henry D. Does it matter who writes medical news stories? PLoS Med 2010;7. pii: e1000323.  Back to cited text no. 10
    
11.
Association of Health Care Journalists. Available from: http://www.healthjournalism.org/training-jump.php. [Last accessed on 2015 Dec 15].  Back to cited text no. 11
    
12.
Media Doctor Australia. Available from: http://www.mediadoctor.org.au/. [Last accessed on 2008 Dec 12].  Back to cited text no. 12
    
13.
Schwitzer G. HealthNewsReview.org: Criteria for excellence in health and medical journalism. Virtual Mentor 2007;9:225-8.  Back to cited text no. 13
    
14.
Oxman AD, Guyatt GH, Cook DJ, Jaeschke R, Heddle N, Keller J, et al. An index of scientific quality for health reports in the lay press. J Clin Epidemiol 1993;46:987-1001.  Back to cited text no. 14
    
15.
Woloshin S, Schwartz LM. Media reporting on research presented at scientific meetings: More caution needed. Med J Aust 2006;184:576-80.  Back to cited text no. 15
    
16.
Hochman M, Hochman S, Bor D, McCormick D. News media coverage of medication research: Reporting pharmaceutical company funding and use of generic medication names. JAMA 2008;300:1544-50.  Back to cited text no. 16
    
17.
Schwitzer G, Mudur G, Henry D, Wilson A, Goozner M, Simbra M, et al. What are the roles and responsibilities of the media in disseminating health information? PLoS Med 2005;2:e215.  Back to cited text no. 17
    
18.
Moynihan R. Making medical journalism healthier. Lancet 2003;361:2097-8.  Back to cited text no. 18
    
19.
Therapeutics Initiative Evaluating the Media as a Source of drug Therapy Information Therapy Letter; 2007. Available from: http://www.ti.ubc.ca/PDF/67.pdf. [Last accessed on 2015 Jun 18].  Back to cited text no. 19
    
20.
Pribble JM, Goldstein KM, Fowler EF, Greenberg MJ, Noel SK, Howell JD, et al. Medical news for the public to use? What's on local TV news. Am J Manag Care 2006;12:170-6.  Back to cited text no. 20
    



 
 
    Tables

  [Table 1], [Table 2]



 

Top
 
 
  Search
 
Similar in PUBMED
   Search Pubmed for
   Search in Google Scholar for
 Related articles
Access Statistics
Email Alert *
Add to My List *
* Registration required (free)

 
  In this article
Abstract
Introduction
Methods
Results
Discussion
Conclusions
References
Article Tables

 Article Access Statistics
    Viewed1618    
    Printed29    
    Emailed0    
    PDF Downloaded79    
    Comments [Add]    

Recommend this journal


[TAG2]
[TAG3]
[TAG4]