Fluoride Action Network

Water fluoridation and the quality of information available online

Source: Clinical Oral Investigations 11:189–193. | February 27th, 2007

Background: The Internet has transformed the way in which people approach their health care, with online resources becoming a primary source of health information. Little work has assessed the quality of online information regarding community water fluoridation. This study sought to assess the information available to individuals searching online for information, with emphasis on the credibility and quality of websites.

Methods: We identified the top 10 web pages returned from different search engines, using common fluoridation search terms (identified in Google Trends). Web pages were scored using a credibility, quality and health literacy tool based on Global Advisory Committee on Vaccine Safety (GAVCS) and Center for Disease Control and Prevention (CDC) criteria. Scores were compared according to their fluoridation stance and domain type, then ranked by quality. The functionality of the scoring tool was analysed via a Bland-Altman plot of inter-rater reliability.

Results: Five-hundred web pages were returned, of which 55 were scored following removal of duplicates and irrelevant pages. Of these, 28 (51%) were pro-fluoridation, 16 (29%) were neutral and 11 (20%) were anti-fluoridation. Pro, neutral and anti-fluoridation pages scored well against health literacy standards (0.91, 0.90 and 0.81/1 respectively). Neutral and pro-fluoridation web pages showed strong credibility, with mean scores of 0.80 and 0.85 respectively, while anti-fluoridation scored 0.62/1. Most pages scored poorly for content quality, providing a moderate amount of superficial information.

Conclusion: Those seeking online information regarding water fluoridation are faced with comprehensible, yet poorly referenced, superficial information. Sites were credible and user friendly; however, our results suggest that online resources need to focus on providing more transparent information with appropriate figures to consolidate the information.

Key words:

Dental public health
oral health promotion
online health information
health literacy
website credibility

Introduction

Water fluoridation was first introduced into local drinking water supplies after it was found that those living in Colorado Springs, an area with naturally high water fluoride concentrations, were extremely resistant to dental caries1. The dramatic reduction in the prevalence and severity of dental caries following the introduction of water fluoridation led the Centers for Disease Control and Prevention (CDC) to proclaim it as one of the major public health initiatives of the 20th century2. Many studies have been carried out demonstrating the effectiveness of community water fluoridation (CWF), with both children and adults living in fluoridated areas exhibiting a lower prevalence of dental caries compared with their counterparts living in non-fluoridated or low-level fluoridation areas3., 4.. Like many public health initiatives, CWF has attracted controversy from a seemingly small, yet vocal, section of the population who claim, among other unsubstantiated health risks, that CWF is an unethical socialised medicine5., 6.. This is particularly concerning as, for the most part, decisions regarding the implementation of CWF are made by local governments who may be more likely to be influenced by these vocal opponents, often local residents. This is evident from the cessation of CWF in communities where it had been previously established, as well as the hesitancy to introduce CWF in Europe1., 5.. Therefore, it is important that when the topic of CWF is being debated in local communities, the population is able to access accurate, unbiased information, enabling them to make an informed decision. In today’s digital age, many people seek out this information on the Internet.

With over 3 billion people worldwide now online7, the Internet has transformed the way people approach their health care. While physicians remain the most trusted source of information, they have been superseded by the Internet as the primary source of information8. This has two major drawbacks: first, anyone can post material on the Internet without review, which raises concerns regarding credibility9. Second, discrepancies can arise between what the author has written and the ability of the intended audience to comprehend and interpret it correctly. The large volume of people able to access the Internet for health information highlights the importance of having both credible and effectively presented online information. While previous research has focussed on the quality of online information as a whole, or on particular conditions10., 11., little work has been carried out on specific public health issues, such as CWF. This study attempts to address this gap in the literature by assessing the information available to individuals who search online for water-fluoridation information, with emphasis on credibility and quality of websites.

Methods

A broad search was simulated by first establishing general search terms for water-fluoridation information. Google Trends was used to convey the relative frequency with which generic search terms ‘fluoride’, ‘water fluoridation’ and ‘fluoridation’ were used globally from 2004 to 2015. These terms were then entered into Google Trends ‘related search’ function, revealing terms often searched in relation to water fluoridation. This generated a list of the most commonly used search terms regarding water fluoridation globally. As the primary focus of this study was to evaluate global online information regarding water-fluoridation, terms such as ‘sodium’, ‘fluoride toothpaste’ and ‘water fluoridation UK’ were removed from the search terms. Specifically these terms, as well as others, were excluded because they did not present sufficient information regarding global water fluoridation, focussing on alternatives such as fluoride toothpaste or the state of fluoridation in specific regions. For example, the term ‘sodium’ was excluded because the predominant results generated related to the chemical properties of sodium as well as to safety information. Conversely, the term ‘sodium fluoride’ was retained as this is one chemical used to fluoridate water supplies and thus the search results returned mainly pertained to water fluoridation as well as to the perceived controversy surrounding it. Table 1 specifies the complete list of search terms used in this study.

Table 1. List of water-fluoridation search terms used in the search strategy

Generic search term Related search terms returned by Google Final list of search terms examined
Fluoride Fluoride water
Fluoride in water
Sodium fluoride
Sodium
Fluoride toothpaste
Toothpaste
What is fluoride
Fluoride water
Fluoride in water
Sodium fluoride
What is fluoride
Fluoridation of water
Fluoridation in water
Fluoridated water
What is fluoridation
Fluoridation water
Fluoridating of water
Fluoride in water
Water fluoridation Fluoridation of water
Fluoridation in water
Fluoride water
Fluoride in water
Water fluoridation UK
Fluoridated water
What is fluoridation
Fluoridation Fluoridation waterFluoridating of water
Fluoridation in water
Fluoride water
Fluoride in water
What is fluoridation
UK fluoridation

The most commonly used search engines were identified via an online Search Engine List that ranks sites via an average of their Alexa Global Traffic Rank and their US Traffic Rank12. The search engines used were: Google.com.au; bing.com; yahoo.com; Ask.com; and aol.com. Additional search criteria, such as region restrictions, were not implemented as this study was attempting to simulate a global search. Before starting the searches, and between entries of each search term, the browser history of the computer being used was cleared. Additionally, search engines with login capabilities, such as Google, were not logged into in an attempt to minimise ‘filter bubble’13 – the effect of algorithms based on individual user search histories.

All searches were performed on 30 January, 2016. The final list of terms (see Table 1), were entered consecutively into each of the search engines. The search results were limited to the top 10 results returned for each search term as research has shown that people rarely search further than the top 10 results14. The results of each search were transferred into a Microsoft Excel spreadsheet in the order that they were returned by the search engines. News articles were excluded, as this was not an analysis of media portrayal of water fluoridation. Furthermore, sponsored sites (paid advertisements) were not included as people performing focussed searches avoid advertised material15.

Web pages were categorised as pro-fluoridation, neutral and anti-fluoridation. Anti-fluoridation pages were defined as those pages that built an argument against water fluoridation. Similarly, pro-fluoridation pages were defined as those that built an argument in favour of water fluoridation. Alternatively, pages that presented all views evenly, or without an overt position, were deemed neutral.

The web pages were assessed under the domains of credibility, quality and health literacy. In the case of the credibility criteria the site as a whole was explored to find the relevant details. Conversely, the quality and health-literacy criteria were limited to the page generated from the search. This was because of the fact that searchers rarely look past the first page16; however, the decision was made to explore the whole site for credibility as often these criteria are found on dedicated pages within the site and ignoring these may have resulted in an underestimation of the credibility of the websites.

The credibility, quality and health-literacy criteria were developed from the World Health Organization Global Advisory Committee on Vaccine Safety (GAVCS) criteria for assessing vaccine safety sites and the CDC Health Literacy Online Guide17., 18.. Information from these two sources was combined to produce a 27-item checklist that fell under three domains (see Table 2). Each item was applied to each website and given a grade of present, absent or not applicable. Not-applicable criteria were subtracted from 27 and a standardised score was produced via the total number of present items and remaining applicable items. To assess accuracy, we used a number of critical reviews regarding CWF1., 5., 19., 20.. These reviews were used as they provided comprehensive scientific information regarding the significant effect of CWF on reducing the prevalence of dental caries. Furthermore, the reviews also highlighted associated risks, such as dental fluorosis, and also demonstrated no causal link between CWF and cancer, which is often used by anti-fluoride groups in their campaigns.

Table 2. Criteria for assessing web-page credibility and the content quality and health literacy of fluoridation web pages were developed from the World Health Organization Global Advisory Committee on Vaccine Safety (GAVCS) criteria for assessing vaccine safety sites and the Centers for Disease control and Prevention (CDC) Health Literacy Online Guide

Criteria Item
Credibility (WHO domain)
Mission of site Purpose of website stated
Intended audience defined
Disclosure of ownership/source Organisation name, contact information and links to contact page or home page present on every page
Qualifications of organisation/individual should be clear, including type of organization
Disclosure of individual’s affiliations and alliances as well as relationships that may influence content of the site
Transparency of sponsorship Disclosure of all sources of funding for organisation/website
Accountability to users Ability to contact site owner to report technical issues, such as broken links, or to provide feedback, is provided
Information regarding contact methods should be easily accessible from any page of the site
Responsible partnering Websites should indicate when leaving the home site by using an external link icon, a disclaimer statement or by opening a new window
Content quality (WHO domain)
Accuracy Authority of sources: Websites should provide a clear statement of the source for scientific, medical and health information, including the author’s name and affiliations
Quality of information: Information supported by citations to scientific data or references
Currency The date the content was last updated or reviewed must clearly be displayed on each page
Quantity Balance: Both benefits and risk of health intervention are included in a fair and unbiased way
Provision of links to other resources: Most hypertext links to reputable primary sources, such as peer-reviewed journal articles or government pages
Health literacy (CDC item)
Design Logical organisation: Coherent structure allowing logical progression through the information
Ease of navigation: Aids to finding information, such as a site map, index, help function, frequently asked questions page and/or internal search engine
Consistent plan: The site should be internally consistent in terms of design, including the use of logos and icons, colour, fonts, page layout, etc
Professional presentation: Overall look of the site should be professional, including sponsor logo, linear information paths and strong visual components
Actionable content Written expression: Written in familiar language that is easy to understand with any relevant terminology fully defined in short, simple sentences and paragraphs
Formatting: The most pertinent information presented first and above the fold
Display Headings: Headings should be meaningful and specific with sufficient space between the text above and the heading to differentiate the information
Font: Font size should be large and in a style that is easily read. Although not essential, the use of a relative text option is advisable
Spacing: Sufficient use of white space, allowing visual separation of design and text elements, thus improving readability
Appropriate line spacing between texts to aid readability
Figures: Supporting and realistic images that aid in understanding of the content
Figures should have a caption that clearly highlights their meaning
Contrast: Appropriate contrast (light and dark colours) used to improve readability

To examine the suitability and clarity of the scoring criteria, an independent rater was enlisted to score a random selection of 20 web pages and a measure of inter-rater reliability was used. The scores given by the independent rater and primary rater of this study were compared via a Bland-Altman plot in which the mean score of both raters is plotted against the difference in the two scores to demonstrate reliability of the scoring system.

Results

Quantity

Overall, 500 web pages were returned via the search strategy, with 407 duplicates removed. A further 38 web pages were excluded from the scoring process for the following reasons: contained insufficient information pertaining to water fluoridation; focussed on the chemical nature of the compounds used in fluoridation; addressed bottled water; or presented information as personal opinion (for example, blogs about water fluoridation in the user’s local area; see Figure 1 for a flow diagram). A number of websites returned multiple pages in the search process, resulting in a greater representation in the search results (see Table 3). The most commonly occurring websites were: Wikipedia; Fluoride Alert; About (now Verywell); the U.S. Center for Disease Control and Prevention; and Queensland Health.

Figure 1.

Figure 1.. Search strategy flow diagram.

Table 3. List of websites that returned multiple web pages about water fluoridation following the search strategy

Website title Website URL No. of pages returned by search
Wikipedia https://en.wikipedia.org 4
Fluoride Action Network http://fluoridealert.org 3
About http://www.about.com/ 3
Centre for Disease Control and Prevention http://www.cdc.gov 3
Queensland Health https://www.health.qld.gov.au/ 3

Water-fluoridation stance

Of the 55 web pages included in the scoring process, 28 (51%) were pro-fluoridation, 16 (29%) were neutral and 11 (20%) were anti-fluoridation. Analysis of the top 10 results returned by the most-used search term (‘fluoridation of water’) showed that of the top 43 results (seven were excluded from scoring), 20 (46.5%) were pro-fluoridation and 13 (30.2%) were neutral. The remaining 10 (23.2%) were anti-fluoridation, with the majority coming from fluoridealert.org, run by the Fluoride Action Network (a vocal anti-fluoride group in the USA).

Web-page origins

Of the 55 web pages scored, 35 originated from the USA, 14 from Australia, three from the UK and one from Canada. Two were of unclear origin.

Quality of scoring system

Bland-Altman plot analysis revealed that when scores differed between the independent rater and primary rater they only differed by a maximum of 4.5 out of a possible difference of 100 (see Figure 2). A difference of 4.5 can be attributed to a difference between the marker’s scores in a single criterion. Across the web pages analysed, the criterion in which differences occurred varied; however, the most common was in the presentation of the most pertinent information in the top half of the page.

Figure 2.

Figure 2.. Bland-Altman plot showing the distribution of agreement between two raters of 55 web pages about water fluoridation.

Quality of information

The quality of information assessed using the GACVS and CDC guidelines varied greatly between websites. The median score across all web pages assessed was 80/100, with a maximum score of 96 (www.cancer.org/cancer/cancercauses/othercarcinogens/athome/water-fluoridation-and-cancer-risk) and a minimum score of 22.2 (www.angelfire.com/az/sthurston/fluoride.html).

Grouping web pages according to their domain (.com, .gov and .org) and ranking them according to score revealed a clear trend (see Figure 3). Overall, .gov websites scored the highest; however, their score fell below the median in the final two sites, ranking with scores ranging from 95 to 69. The score of two .gov sites fell below the median because they failed to indicate to the user when they were moving to pages not maintained by the original website (responsible partnering) and to provide insightful figures with clear explanations. Additionally, .org websites outscored .com sites, with scores ranging from 96 to 39 and 95 to 22, respectively.

Figure 3.

Figure 3.. Fifty-five web pages about water fluoridation, grouped according to domain and ranked by score.

Furthermore, when grouped according to stance and examined at the levels of credibility, content quality and health literacy, further trends appeared (see Figure 4). Overall, all three stances corresponded well to standards required to support health literacy, with average scores of 0.91, 0.90 and 0.81 for pro, neutral and anti-fluoridation web pages, respectively. The most common shortcomings regarding health-literacy standards were failing to provide the most pertinent information in the upper half of the web page, as well as irrelevant, or not clearly relevant, figures. Similarly, neutral and pro-fluoridation web pages showed strong credibility with mean scores of 0.80 and 0.85, although these web pages most commonly did not disclose any potential conflicts of interest or sources of funding for the site. By contrast, anti-fluoridation web pages had a lower mean credibility score of 0.62. These pages often did not have clear qualifications, provide contact details or exhibit responsible partnering (informing the user they were moving to pages not maintained by the original website). Regardless of their stance on fluoridation, most pages performed poorly in terms of quality of content. The majority of sites provided a moderate amount of superficial information, with little information regarding authors and lacked references to scientific articles. In cases where author’s names were provided, their qualifications were unclear. Similarly, when references to scientific articles were provided they were not accompanied by hypertext links, making even freely available articles harder to access for the general public.

Figure 4.

Figure 4.. Mean credibility, content quality and health-literacy scores of 55 web pages about water fluoridation, grouped according to fluoridation stance (anti, neutral or pro).

Google rank

Table 4 demonstrates that there was no clear correlation between the order in which web pages appeared in the Google search and their quality, as determined by the criteria used in this study. It is also evident that the top sites returned by each search were predominantly Australian sites, despite the browser history being cleared before each search and between search terms.

Discussion

To the best of our knowledge this is the first published assessment of the quality of online information regarding water fluoridation. We found that .com and anti-fluoridation websites scored lower in credibility, quality and health literacy.

It is a long-standing view that anti-fluoridation lobbyists make up a relatively small, yet vocal, component of the fluoridation debate5. In this study, anti-fluoridation web pages were a minority (20%) and yet produce considerable numbers of attention-grabbing claims (albeit unsubstantiated, through failing to reference reputable scientific articles to support their assertions).

There was a reasonable prevalence of neutral web pages (29%) returned by the search, which presented pros and cons of water fluoridation supported in the literature, such as its dental-health benefit and the possibility of fluorosis. Critical consumers of health information frequently seek information they perceive as unbiased, allowing them to come to their own conclusions21. We also found that the majority of pro-fluoridation web pages came from sources such as government websites, which were more likely to present a balanced view of water fluoridation.

Geolocation, that is the location of the person conducting the search, is an issue regarding web searches as it has a significant impact on the results generated. The effect of geolocation can clearly be seen through this study as, despite clearing the browser history before each search and between search terms, a large proportion of sites returned in this study originated within Australia. One study has found that geolocation generates the greatest variability in terms of search results compared with any other factor22. Specifically, it was shown that the majority of web pages returned tended to reflect the pre-existing beliefs of the particular region based on previous Internet traffic and omit conflicting information that the algorithm deems unnecessary, generating a ‘filter bubble’. This enhances human confirmation bias in which people tend only to read information that conforms to their already established beliefs23. In this case, the search engine is influencing the information provided via location and previous search terms. This is particularly important when considering potentially divisive public health issues, such as water fluoridation, as it would result in searchers in regions where the community is strongly opposed to water fluoridation being more exposed to anti-fluoridation sites, preventing them from being able to make an unbiased and well-informed decision. We recommend that this information forms the basis of a set of guidelines for health-related web searches to assist the general public to perform these searches. These guidelines would need to be fully evaluated to contribute to a limited evidence base on how to enhance online health literacy24. More broadly, public health and pro-fluoridation communication also needs to provide compelling information via a range of sources to counterbalance anti-fluoride information in communities where this activity is prominent.

A common misconception held by individuals performing health-related searches is that the order in which search engine results appear is an indication of their quality or relevance14. However, the order of web page appearance is, in fact, generated by an algorithm that takes into account the number of times a page is linked to the search term, with pages that have more links to them appearing higher in the search results. While this algorithm is highly effective when performing consumer searches in which the most popular pages are desirable, it produces a false perception of quality in health-related searches.

In terms of quality, while the majority of pro-fluoridation pages presented accurate information, it was often not referenced or credited to a qualified author. Additionally, they did not disclose affiliations that could produce a conflict of interest or sources of funding for the website. All of these factors are key to establishing a sense of transparency and credibility of the health information and are likely to increase levels of trust. We recommend that those developing online information about water fluoridation credit information to appropriate authors, disclose conflict of interests and state sources of funding, as this will improve their overall quality according to international criteria.

This study has some limitations. First, the dynamic nature of the Internet means that there is constant addition, removal or updating of information. Our sample captured a snapshot from January 2016 using a search conducted with a clear browser history. The degree to which any individual’s search results would differ has been shown, in other research, to account for an 11.7% variation, with one study showing less variation in higher-ranked hits22., 25.. Additionally, the two compounds most commonly used to fluoridate water supplies – hydrofluorosilicic acid and silicofluoride – were not used as search terms. As neither of these terms appeared in the Google Trends ‘related search’ function when searching for commonly associated terms, they are unlikely to have greatly impacted the results of this study. Furthermore, the search captured a predominance of Australian servers as a result of the search being performed in Australia. Nonetheless, this study does provide a diverse snapshot of the stance and quality of information generally available regarding water fluoridation and includes international sites. Finally, our quality scoring system was based on a World Health Organization (WHO) system for rating vaccine safety rather than a pre-established and validated scoring system. Our tool was fit-for-purpose as it was a global tool that had been applied to vaccination, a thematically similar topic with respect to its public health importance, safety claims and controversy. Furthermore, the tool included many of the same domains described in the review of Zhang et al. 26, across currency, credibility and readability. However, our design elements were viewed through a health literacy framework.

In conclusion, members of the public performing a search regarding water fluoridation are presented with a moderate amount of superficial information that is easy to understand but is poorly referenced. The sites available were, for the most part, credible and presented information that was easy to comprehend and remember. Our findings suggest that when developing resources regarding water fluoridation, greater emphasis on providing well-accredited, transparent web pages that are open in terms of potential conflicts of interest as well as funding, is required. Furthermore, we recommend that communicators give clearly explained, relevant figures that aid in consolidating, or further explaining, the information presented in the text. Finally, regarding future use, the criteria used to assess the web pages in this study were by no means specific to water fluoridation and therefore provide a basis for further studies where they can be used directly, or as a scaffold to be tailored to specific research questions.

Acknowledgements

The authors would also like to thank Petra Macaskill for her advice regarding statistical analysis. Zachary Frangos was supported by a University of Sydney Medical School Summer Research Scholarship to undertake this research. J Leask was supported by an NHMRC Career Development Fellowship.

Conflict of interest

J Leask’s institution has received funding from New South Wales Health for work related to water fluoridation.

References

1. Pizzo G, Piscopo MR, Pizzo I, et al. Community water fluoridation and caries prevention: a critical review. Clin Oral Invest, 11 (2007), pp. 189-193
2. Horowitz HS. The 2001 CDC recommendations for using fluoride to prevent and control dental caries in the United States. J Public Health Dent, 63 (2003), pp. 3-8.

3. Armfield JM, Akers HF. Risk perception and water fluoridation support and opposition in Australia. J Public Health Dent, 70 (2010), pp. 58-66.

4. Griffin SO, Regnier E, Griffin PM, et al. Effectiveness of fluoride in preventing caries in adults. J Dent Res, 86 (2007), pp. 410-415.

5. Howat P, Binns C, Jancey J. New international review supports community water fluoridation as an effective and safe dental health promotion measure. Health Promot J Austr, 26 (2015), pp. 1-3.

6. Olson R. The Politics of Water Fluoridation from a Problem Definition Perspective. Northeastern University: ProQuest Dissertations Publishing, Boston, MA (2008), p. 25. Ph.D. thesis.

7. Union IT. ICT Facts & Figures. Secondary ICT Facts & Figures; 2015. Available from: https://www.itu.int/en/ITU-D/Statistics/Documents/facts/ICTFactsFigures2015.pdf. Accessed 15 January 2016.

8. Hesse BW, Nelson DE, Kreps GL, et al. Trust and sources of health information: the impact of the internet and its implications for health care providers: findings from the first health information national trends survey. Arch Intern Med, 165 (2005), pp. 2618-2624

9. Metzger MJ. Making sense of credibility on the Web: models for evaluating online information and recommendations for future research. J Am Soc Inform Sci Technol, 58 (2007), pp. 2078-2091

10. Purcell GP, Wilson P, Delamothe T. The quality of health information on the internet. BMJ (Clinical research ed.), 324 (2002), pp. 557-558.

11. Meric F, Bernstam EV, Mirza NQ, et al. Breast cancer on the world wide web: cross sectional survey of quality of information and popularity of websites. BMJ, 324 (2002), pp. 577-581.

12.eBIZMBA. Top 15 Most Popular Secondary Top 15 Most Popular 2016. Available from: http://www.ebizmba.com/articles/search-engines. Accessed 10 January 2016.

13. Bozdag E, van den Hoven J. Breaking the filter bubble: democracy and design. Ethics Inf Technol, 17 (2015), pp. 249-265.

14. Mager A. Mediated health: sociotechnical practices of providing and using online health information. New Media Soc, 11 (2009), pp. 1123-1142.

15. Cho C, Cheon HJ. Why do people avoid advertising on the internet? J Advert, 33 (2004), pp. 89-97.

16. Lorigo L, Haridasan M, Brynjarsdóttir H, et al. Eye tracking and online search: lessons learned and challenges ahead. J Am Soc Inform Sci Technol, 59 (2008), pp. 1041-1052.

17. Global Advisory Committee on Vaccine Safety Safety. Good information practices for vaccine safety web sites Geneva. Secondary Good information practices for vaccine safety web sites Geneva. Available from: http://www.who.int/vaccine_safety/good_vs_sites/en/. Accessed 20 November 2015.

18. Office of Disease Prevention and Health Promotion. Secondary Health Literacy Online 2016. Available from: https://health.gov/healthliteracyonline/. Accessed 7 January 2016.

19. Carey CM. Focus on fluorides: update on the use of fluoride for the prevention of dental caries. J Evid Based Dent Pract, 14 (Suppl) (2014), pp. 95-102

20. Lewis DW, Banting DW. Water fluoridation: current effectiveness and dental fluorosis. Community Dent Oral Epidemiol, 22 (1994), pp. 153-158.

21. Sillence E, Briggs P, Fishwick L, et al. Trust and mistrust of online health sites. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 985776: ACM, Vienna, Austria (2004), pp. 663-670

22. Location, location, location: the impact of geolocation on web search personalization 2015.
23. Nickerson RS. Confirmation bias: a ubiquitous phenomenon in many guises. Rev Gen Psychol, 2 (1998), pp. 175-220
24. Car J, Lang B, Colledge A, et al. Interventions for enhancing consumers’ online health literacy. Cochrane Database Syst Rev (Online) (6) (2011), p. CD007092.
25. Hannak A, Sapiezynski P, Kakhki AM, et al. Measuring personalization of web search. Proceedings of the 22nd international conference on World Wide Web. 2488435: ACM, Rio de Janeiro, Brazil (2013), pp. 527-538.
26.  Zhang Y, Sun Y, Xie B. Quality of health information for consumers on the web: a systematic review of indicators, criteria, tools, and evaluation results: quality of Health Information for Consumers on the Web. J Assoc Inf Sci Technol, 66 (2015), pp. 2071-2084.

*Full-text article online at https://link.springer.com/article/10.1007%2Fs00784-007-0111-6