Evaluation of drinking water quality test kits for home use in the United States.

Our results showed highly variable performance in the test kits’ ability to detect potential contaminants in the water. In the measurements, results showed that results were largely consistent; therefore, test kit results were generally highly precise but not always accurate. Accuracy was often affected by the water matrix: in general, there were few noticeable differences in performance between tests performed in tap water compared to river water, while differences were frequently observed when comparing results in DI water as compared to the tap or river water. Notably, no one brand or kit performed consistently well across multiple water quality parameters; the same brand in some cases performed well for one parameter and poorly for another, or the same multiparameter kit was able to adequately measure one parameter but not another. In general, the test kits that measured only single parameters performed better than the multiparameter kits when measuring iron or copper. Two of the test kits measuring only iron that performed well included a reducing agent, which would reduce any ferrous iron to ferric iron. It is important to consider the intervals at which each kit measures: a concentration measured by a test kit that was markedly different from the laboratory-obtained concentration may have been limited not by the intensity of color it produced but by the design of the OTS kit as having only the ability to measure in measurement bins (e.g. 0–5 mg/L rather than the ability to differentiate 1 vs. 4 mg/L).

Our results are consistent with those found by studies focused on the measurement of other contaminants, which have also identified challenges that home users had or would have with obtaining accurate results, such as for nitrate or lead, showing that false negatives or user-measured results as lower were common (Nielsen et al. 2008; Kriss et al. 2021). The observed differences in performance are likely due to test kit chemistry and water matrix interference. While the test strip kit chemistries are not known (information was not disclosed by manufacturers), our results demonstrated that the iron measurement test kits involving reducing agents performed better as they could then likely detect both ferrous and ferric iron (similar challenges with metal speciation detection via home test kits have been previously observed with the measurement of lead (Kriss et al. 2021)). Multi-parameter test kits often performed worse than single-parameter kits, potentially as they tried to optimize for multiple analytes at once, many of which may need different pH for optimum performance. Also, as noted in the literature on colorimetric methods, interference from the water matrices likely affected kit performance; in our experiments, performance in tap and river water was poorer than in DI water, suggesting interference from organic matter and other ions likely occurred. However, notably, our laboratory-determined values also relied on colorimetric methods. Additionally, we selected sulfate salts as the source compounds for iron, copper, and manganese; however, to facilitate future standardization for evaluating commercial test kit performance, future work could compare kit performance with the addition of different anions to the ability to mimic those found in drinking water, particularly where water matrices may influence results, as well as better understand speciation and its effect on results.

Some of the multiparameter kits outperformed the parameter-only kits if the goal was a binary outcome of above or below the SMCL; while the copper-only kits may be more accurate, the sensitivity analysis revealed that some of the multiparameter kits outperformed the copper-only kits when assessing their ability to compare to the SMCL. Therefore, the selection of test kits should consider its application: for a water consumer, the ability of a test kit to meaningfully provide information about a water sample’s compliance with an MCL may be more important than accuracy or the ability to detect >0 mg/L.

This study had limitations that also suggest future directions. Our ‘true value’ test was with another colorimetric method; while a more accurate method such as ICP-MS would have yielded more accurate results for our true value, we selected the single-parameter colorimetric method as our base of comparison as many water utilities and scientific studies are conducted with this method, and this is what the test strips are trying to emulate. For example, the kit testing manganese was a direct modification of the Hach PAN LR spec method, with the steps and reagents closely matching each other. Future work could also compare to results from an ICP-MS. For these experiments, tap and river water were sometimes collected on different days due to logistical constraints, which would result in different background concentrations of parameters of interest or constituents that could potentially interfere with results; however, these sources were not expected to vary by much over the few days in which water was collected, and we measured the background concentration of the parameter of interest on each day water was collected. Future studies should also incorporate testing in different waters, particularly groundwater, as those are the sources that many US homeowners use, as well as with varying pH and temperature, which would be expected to significantly affect results. Future work should investigate the constituents in source waters that could be causing interference with the test kits, such as through the use of synthetic water and testing potential interference (e.g. organic matter, alkalinity, hardness, and pH) and conducting more detailed analysis of water matrix chemistry. While consistent lab personnel were used in this study, factors such as time of day, lighting conditions, colorblindness, and eyesight may affect results, although the kit instructions did not mention these factors. A procedure for consistently reading color charts could help eliminate this large source of error, such as other efforts to use a mobile phone to read the results of colorimetric methods. Future work should also focus on varying some of the key method procedures to reproduce mistakes likely made by potential users, such as waiting more or less time than instructed to dip the strip in the water or matching the color. Finally, we tested only a very small fraction of the test kits that are available on the market, and the selection was nonrandom; we sought to replicate the procedure a home user may use when selecting test kits and acknowledged that manufacturer and kit variability changes daily. Future studies could focus on more holistically evaluating kits on the market and other procedures for selecting test kits, such as by kit chemistry or aligned result bins, as well as evaluating the many other parameters that users may want to measure.

Overall, these test kits are being sold and used by consumers across the United States. However, our results suggest that there are concerns about their accuracy and usefulness and that whether the results can be relied on depends on which parameter is being measured in which water with which kit and for which purpose. While many of these studies have focused on the assessment of commercially available test kits, recent efforts have highlighted opportunities to co-design test kits with users such as middle and high school teachers (Haynes et al. 2019). Previous studies focused on preparing and validating field test kits, particularly for use in low- and middle-income countries to measure chlorine or fecal indicator bacteria, have shown that, with method development and evaluation of user experiences, such kits can be reliable and accurate; however, many of these are still focused on use by those with some training in water quality rather than home users (Bain et al. 2012; Khush et al. 2013; Murray & Lantagne 2015; Bain et al. 2021). The results of our study suggest a need to develop standardized protocols for evaluating the appropriateness and validity of these home test kits given different goals and water sources. There are significant gaps in our knowledge of the performance of these test kits and how to engage with potential users in selecting appropriate parameters to measure and in interpreting the results into actionable steps. Considering both the reliability and accuracy of the results, as well as how potential users may interact with the kits, is important for home users, community groups, or citizen science initiatives that may be relying on these results.

We acknowledge funding from the University of Massachusetts Amherst Civil and Environmental Engineering.