Quality and readability of online information resources on insomnia
Yan Ma1(), Albert C. Yang1, Ying Duan2, Ming Dong3, Albert S. Yeung4
1. Division of Interdisciplinary Medicine and Biotechnology, Department of Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA 02215, USA 2. Sleep Medicine Center, Airforce General Hospital, Beijing 100142, China 3. IBM, Software Development Lab, Littleton, MA 01460, USA 4. Depression Clinical and Research Program, Massachusetts General Hospital, Harvard Medical School, Boston, MA 02114, USA
The internet is a major source for health information. An increasing number of people, including patients with insomnia, search for remedies online; however, little is known about the quality of such information. This study aimed to evaluate the quality and readability of insomnia-related online information. Google was used as the search engine, and the top websites on insomnia that met the inclusion criteria were evaluated for quality and readability. The analyzed websites belonged to nonprofit, commercial, or academic organizations and institutions such as hospitals and universities. Insomnia-related websites typically included definitions (85%), causes and risk factors (100%), symptoms (95%), and treatment options (90%). Cognitive behavioral therapy for insomnia (CBT-I) was the most commonly recommended approach for insomnia treatment, and sleep drugs are frequently mentioned. The overall quality of the websites on insomnia is moderate, but all the content exceeded the recommended reading ease levels. Concerns that must be addressed to increase the quality and trustworthiness of online health information include sharing metadata, such as authorship, time of creation and last update, and conflicts of interest; providing evidence for reliability; and increasing the readability for a layman audience.
. [J]. Frontiers of Medicine, 2017, 11(3): 423-431.
Yan Ma, Albert C. Yang, Ying Duan, Ming Dong, Albert S. Yeung. Quality and readability of online information resources on insomnia. Front. Med., 2017, 11(3): 423-431.
4. Is it clear what sources of information were used to compile the publication (other than the author or producer)?
3.1
(2.6, 3.6)
5. Is it clear when the information used or reported in the publication was produced?
2.9
(2.4, 3.4)
6. Is it balanced and unbiased?
3.8
(3.5, 4)
7. Does it provide details of additional sources of support and information?
3.4
(2.9, 3.8)
8. Does it refer to areas of uncertainty?
2.1
(1.8, 2.4)
SECTION 2: How good is the quality of information on treatment choices?
9. Does it describe how each treatment works?
3.4
(3, 3.8)
10. Does it describe the benefits of each treatment?
3.5
(3.1, 3.8)
11. Does it describe the risks of each treatment?
2.8
(2.4, 3.2)
12. Does it describe what would happen if no treatment is used?
3.1
(2.7, 3.4)
13. Does it describe how the treatment choices affect overall quality of life?
2.8
(2.4, 3.1)
14. Is it clear that there may be more than one possible treatment choice?
3.8
(3.4, 4.1)
15. Does it provide support for shared decision-making?
3.1
(2.6, 3.5)
SECTION 3: Overall rating of the publication
16. Based on the answers to all of the above questions, rate the overall quality of the publication as a source of information about treatment choices
3.3
(2.9, 3.7)
Tab.1
Readability
Equivalent grade level in the US
British school age
FRES
FKGL
GFOG or SMOG
Extremely easy
4th or below
Over 100
0
0?6
Very easy
5th
9?12 years
90?100
1?5
Easy
6th
80?89
Fairly easy
7th
12?15 years
70?79
Standard
8th?9th
60?69
6?8
7?8
Fairly difficult
10th?12th
15?17 years
50?59
9?14
9?12
Difficult
College
30?49
13?16
Very confusing
Above college
0?29
≥15
17?19
Tab.2
The studied websites
Quality
Readability
Evaluated items
DISCERN
Checklist
FRES
FKGL
GFOG
SMOG
Reference score/recommended level
0?80
Yes (%)
60?69
6?8
<12
<10
WebMD
55.5
80%
50.1
11.1
13.4
10.4
National Sleep Foundation
62.8
70%
50.6
11.4
14.4
10.6
Mayo Clinic
62.5
90%
57.8
8.9
11.7
8.7
Helpguide.org
64.5
80%
58.3
9.5
12.6
9.2
Wikipedia
67.5
80%
30.2
13.8
17.2
12.8
MNT Knowledge Center
48.0
90%
28.5
13.6
16.7
12.3
eMedicineHealth
62.5
90%
37.2
12.2
15.2
11.3
American Academy of Sleep Medicine
50.0
70%
46.7
10.9
14.0
10.4
Dr. Weil
40.3
50%
51.3
9.7
12.3
9.4
NIH
56.8
70%
57.9
8.9
11.5
8.7
MedlinePlus
44.0
80%
56.1
8.7
10.4
8.7
MedicineNet
63.0
90%
30.7
13.8
17.6
12.8
Life Extension
62.0
60%
33.7
13.2
15.6
12.1
Medscape Reference
68.8
90%
30.0
14.5
18.3
13.5
insomnia.net
37.0
20%
39.9
14.7
17.3
12.0
LiveScience
47.5
70%
40.3
13.9
16.3
12.4
PyschCentral
37.8
50%
33.1
11.8
15.4
11.1
Everyday Health
45.0
30%
40.0
11.9
15.1
11.4
University of Cambridge
45.3
50%
57.1
11.0
13.8
9.7
American Sleep Association
45.8
40%
52.1
10.3
13.4
10.0
Overall quality and readability
53.3
68%
44.1
11.7
14.6
10.9
Tab.3
Fig.1
Fig.2
Fig.3
1
Statistics Canada. Canadian Internet Use Survey-2009. 2010. Available from:
2
Atkinson NL, Saperstein SL, Pleis J. Using the internet for health-related activities: findings from a national probability sample. J Med Internet Res 2009; 11(1): e4 https://doi.org/10.2196/jmir.1035
pmid: 19275980
Fallowfield LJ, Hall A, Maguire GP, Baum M. Psychological outcomes of different treatment policies in women with early breast cancer outside a clinical trial. BMJ 1990; 301(6752): 575–580 https://doi.org/10.1136/bmj.301.6752.575
pmid: 2242455
Kumar VS, Subramani S, Veerapan S, Khan SA. Evaluation of online health information on clubfoot using the DISCERN tool. J Pediatr Orthop B 2014; 23(2): 135–138
pmid: 24048196
7
Ma Y, Dong M, Mita C, Sun S, Peng CK, Yang AC. Publication analysis on insomnia: how much has been done in the past two decades? Sleep Med 2015; 16(7): 820–826 https://doi.org/10.1016/j.sleep.2014.12.028
pmid: 25979182
8
Pletneva N, Vargas A, Kalogianni K, Boyer C. Online health information search: what struggles and empowers the users? Results of an online survey. Stud Health Technol Inform 2012; 180: 843–847
pmid: 22874311
9
NETMARKETSHARE. Desktop Search Engine Market Share. Available from: (Accessed April 13, 2017)
10
Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA 2002; 287(20): 2691–2700 https://doi.org/10.1001/jama.287.20.2691
pmid: 12020305
11
Feufel MA, Stahl SF. What do web-use skill differences imply for online health information searches? J Med Internet Res 2012; 14(3): e87 https://doi.org/10.2196/jmir.2051
pmid: 22695686
12
Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health 1999; 53(2): 105–111 https://doi.org/10.1136/jech.53.2.105
pmid: 10396471
13
Ademiluyi G, Rees CE, Sheard CE. Evaluating the reliability and validity of three tools to assess the quality of health information on the Internet. Patient Educ Couns 2003; 50(2): 151–155 https://doi.org/10.1016/S0738-3991(02)00124-6
pmid: 12781930
14
Wilson P. How to find the good and avoid the bad or ugly: a short guide to tools for rating quality of health information on the internet. BMJ 2002; 324(7337): 598–602 https://doi.org/10.1136/bmj.324.7337.598
pmid: 11884329
15
Aldairy T, Laverick S, McIntyre GT. Orthognathic surgery: is patient information on the Internet valid? Eur J Orthod 2012; 34(4): 466–469 https://doi.org/10.1093/ejo/cjr046
pmid: 21459834
16
Batchelor JM, Ohya Y. Use of the DISCERN instrument by patients and health professionals to assess information resources on treatments for asthma and atopic dermatitis. Allergol Int 2009; 58(1): 141–145
17
Fast AM, Deibert CM, Hruby GW, Glassberg KI. Evaluating the quality of Internet health resources in pediatric urology. J Pediatr Urol 2013; 9(2): 151–156 https://doi.org/10.1016/j.jpurol.2012.01.004
pmid: 22281281
18
Kaicker J, Debono VB, Dang W, Buckley N, Thabane L. Assessment of the quality and variability of health information on chronic pain websites using the DISCERN instrument. BMC Med 2010; 8(1): 59 https://doi.org/10.1186/1741-7015-8-59
pmid: 20939875
19
Pusz MD, Brietzke SE. How good is Google? The quality of otolaryngology information on the internet. Otolaryngol Head Neck Surg 2012; 147(3): 462–465 https://doi.org/10.1177/0194599812447733
20
Zahedi R, Taheri B, Shahrzadi L, Tazhibi M, Ashrafi-rizi H. Quality of Persian addiction websites: a survey based on silberg, discern and wqet instruments (2011). Acta Inform Med 2013; 21(1): 46–50 https://doi.org/10.5455/aim.2012.21.46-50
pmid: 23572862
21
Cerminara C, Santarone ME, Casarelli L, Curatolo P, El Malhany N. Use of the DISCERN tool for evaluating web searches in childhood epilepsy. Epilepsy Behav 2014; 41: 119–121
22
Daraz L, Macdermid JC, Wilkins S, Gibson J, Shaw L. The quality of websites addressing fibromyalgia: an assessment of quality and readability using standardised tools. BMJ Open 2011; 1(1): e000152 https://doi.org/10.1136/bmjopen-2011-000152
pmid: 22021777
Daraz L, MacDermid JC, Wilkins S, Shaw L. Tools to evaluate the quality of web health information: a structured review of content and usability. Int J Technol Knowl Soc 2009; 5: 3
25
Walsh TM, Volsko TA. Readability assessment of internet-based consumer health information. Respir Care 2008; 53(10): 1310–1315
pmid: 18811992
26
US National Library of Medicine. How to Write Easy-to-Read Health Materials 2013. Available from: . Page last updated on December 29, 2016
27
Weiss BD. Health Literacy: A Manual for Clinicians. Chicago, IL: American Medical Association, American Medical Foundation. 2003
28
Kincaid JP, Fishburne RP Jr, Rogers RL, Chissom BS. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. DTIC Document, 1975
29
Davis TC, Mayeaux EJ, Fredrickson D, Bocchini JA Jr, Jackson RH, Murphy PW. Reading ability of parents compared with reading level of pediatric patient education materials. Pediatrics 1994; 93(3): 460–468
pmid: 8115206
30
McLaughlin GH. SMOG grading: a new readability formula. J Read 1969; 12(8): 639–646
31
Fitzsimmons PR, Michael BD, Hulley JL, Scott GO. A readability assessment of online Parkinson’s disease information. J R Coll Physicians Edinb 2010; 40(4): 292–296 https://doi.org/10.4997/JRCPE.2010.401
pmid: 21132132
32
Hesse BW, Nelson DE, Kreps GL, Croyle RT, Arora NK, Rimer BK, Viswanath K. Trust and sources of health information: the impact of the Internet and its implications for health care providers: findings from the first Health Information National Trends Survey. Arch Intern Med 2005; 165(22): 2618–2624 https://doi.org/10.1001/archinte.165.22.2618
pmid: 16344419
33
Hu X, Bell RA, Kravitz RL, Orrange S. The prepared patient: information seeking of online support group members before their medical appointments. J Health Commun 2012; 17(8): 960–978 https://doi.org/10.1080/10810730.2011.650828
pmid: 22574697
34
Lv X, Ma Y, Feng F, Hong L, Wang J, Wang W. Personality development since childhood associated with adult chronic insomnia: a study by Wang’s Memory-Tracing Personality Development Inventory (WMPI). Glob J Health Sci 2016; 9(3): 80 https://doi.org/10.5539/gjhs.v9n3p80
35
Ma Y, Yeung A, Yang AC, Peng CK, Clain A, Alpert J, Fava M, Yeung AS. The effects of tai chi on sleep quality in Chinese American patients with major depressive disorder: a pilot study. Behav Sleep Med2016 Sep 27. [Epub ahead of print] doi: 10.1080/15402002.2016.1228643 https://doi.org/10.1080/15402002.2016.1228643
pmid: 27676270
36
Sun S, Hong L, Ma Y, Zhao Y, Qiao J. WO-109 A Chinese questionnaire-based survey: sleep medicine awareness in primary care. Sleep Med 2011; 12: S122 https://doi.org/10.1016/S1389-9457(11)70456-7
Ma Y, Zhou K, Fan J, Sun S. Traditional Chinese medicine: potential approaches from modern dynamical complexity theories. Front Med 2016; 10(1): 28–32 https://doi.org/10.1007/s11684-016-0434-2
pmid: 26809465
Shi W, Shang P, Ma Y, Sun S, Yeh CH. A comparison study on stages of sleep: quantifying multiscale complexity using higher moments on coarse-graining. Commun Nonlinear Sci Numer Simul 2017; 44: 292–303 https://doi.org/10.1016/j.cnsns.2016.08.019
41
Ma Y, Sun S, Peng CK, Fang Y, Thomas RJ. Ambulatory blood pressure monitoring in Chinese obstructive sleep apnea patients. J Clin Sleep Med 2017;13(3):433–439 https://doi.org/10.5664/jcsm.6498
pmid: 27855748
42
Lu Y, Sun Y, Ma Y, Li H, Chang Y, Tong Y, Sun S. Investigation on the awareness of sleep medicine in a university of traditional Chinese medicine. Sleep Med 2015; 16(S1): S61 https://doi.org/10.1016/j.sleep.2015.02.153
43
Ma Y, Sun S. Knowledge update on complex sleep apnea. Chin J Tuberc Respir Dis (Zhonghua Jie He He Hu Xi Za Zhi) 2013; 36(9): 687–691 (in Chinese)
44
Tseng PH, Lee PL, Hsu WC, Ma Y, Lee YC, Chiu HM, Ho YL, Chen MF, Wu MS, Peng CK. A higher proportion of metabolic syndrome in Chinese subjects with sleep-disordered breathing: a case-control study based on electrocardiogram-derived sleep analysis. PLoS One 2017; 12(1): e0169394 https://doi.org/10.1371/journal.pone.0169394
pmid: 28081171
45
Hong L, Ma Y, Lin Y, Wang F, Tong C, Zhao X, Wang W. Correlation study of depression, anxiety and sleep disturbance. Int J Tradit Chin Med (Guo Ji Zhong Yi Zhong Yao Za Zhi) 2009; 31(2): 152–154 (in Chinese)
46
Hong L, Wei H, Ma Y, Wang W. Manifestation study on depression in comprehensive hospitals. Int J Tradit Chin Med (Guo Ji Zhong Yi Zhong Yao Za Zhi) 2010; 32(3): 259–261 (in Chinese)
47
Ma Y. Review on sleep belief and attitude. World J Sleep Med (Shi Jie Shui Mian Yi Xue Za Zhi) 2015; 2(6): 356–361 (in Chinese)
48
Ma Y, Dong M, Zhou K, Mita C, Liu J, Wayne PM. Publication trends in acupuncture research: a 20-year bibliometric analysis based on PubMed. PLoS One 2016; 11(12): e0168123 https://doi.org/10.1371/journal.pone.0168123
pmid: 27973611
49
Zhou K, Ma Y, Brogan MS. Dry needling versus acupuncture: the ongoing debate. Acupunct Med 2015; 33(6): 485–490
50
Xu H, Ma Y, Wang W. Patients’ independent choices to resistance. Med Philos (Humanist Soc Med Ed) (Yi Xue Yu Zhe Xue: Ren Wen She Hui Yi Xue Ban). 2010; 31(2): 44–46 (in Chinese)
51
Ma Y, Sun SC. Sleep medicine and blood pressure variation: significance to clinical application of anti-hypertensive medicines. World J Sleep Med (Shi Jie Shui Mian Yi Xue Za Zhi) 2014; (4): 235–238 (in Chinese)
52
McPherson AC, Gofine ML, Stinson J. Seeing is believing? A mixed-methods study exploring the quality and perceived trustworthiness of online information about chronic conditions aimed at children and young people. Health Commun 2014; 29(5): 473–482 https://doi.org/10.1080/10410236.2013.768325
pmid: 24099647
53
Tan BH, Kostapanagiotou K, Jilaihawi AN. A review of mesothelioma information on the World Wide Web. J Thorac Oncol 2009; 4(1): 102–104
54
Kaicker J, Dang W, Mondal T. Assessing the quality and reliability of health information on ERCP using the DISCERN instrument. Health Care Curr Rev 2013;1: 104. doi:10.4172/hccr.1000104
55
Ye Y. Correlates of consumer trust in online health information: findings from the health information national trends survey. J Health Commun 2011; 16(1): 34–49 https://doi.org/10.1080/10810730.2010.529491
pmid: 21086209
56
Hargrave DR, Hargrave UA, Bouffet E. Quality of health information on the Internet in pediatric neuro-oncology. Neuro-oncol 2006; 8(2): 175–182 https://doi.org/10.1215/15228517-2005-008
pmid: 16533758
57
Berland GK, Elliott MN, Morales LS, Algazy JI, Kravitz RL, Broder MS, Kanouse DE, Muñoz JA, Puyol JA, Lara M, Watkins KE, Yang H, McGlynn EA. Health information on the Internet: accessibility, quality, and readability in English and Spanish. JAMA 2001; 285(20): 2612–2621 https://doi.org/10.1001/jama.285.20.2612
pmid: 11368735
58
D’Alessandro DM, Kingsley P, Johnson-West J. The readability of pediatric patient education materials on the World Wide Web. Arch Pediatr Adolesc Med 2001; 155(7): 807–812 https://doi.org/10.1001/archpedi.155.7.807
pmid: 11434848
59
Graber MA, Roller CM, Kaeble B. Readability levels of patient education material on the World Wide Web. J Fam Pract 1999; 48(1): 58–61
pmid: 9934385