Operator error appears to have set off a series of events that left approximately 80,000 residents in southern Orange County without water last week.
An “unintentional” key stroke from an Orange Water and Sewer Authority water treatment plant operator, according to an investigative report released Friday, was the trigger to what OWASA executive director Ed Kerwin called a “worst-case scenario.”
The plant, located in Carrboro, was shut down after higher-than-normal levels of fluoride were detected in the plant last Thursday afternoon. Officials have been quick to confirm that no water with increased fluoride levels made it into the drinking water for OWASA customers. The fluoride overfeed led OWASA to bring water in from the City of Durham through an emergency interconnect. Then on Friday, February 3, there was a break in the water main bringing water into southern Orange County. That caused a Do Not Drink/Do Not Use order to be issued for OWASA customers that was in effect for more than 24 hours.
The report from CH2M says the “unintentional WTP operator key stroke at 11:43 AM sent a command to the fluoride feed pump to increase its chemical feed rate to 80% of its total design speed (normal operating range is 8-12% of design speed). A subsequent command was sent to the pump approximately 12 seconds after the initial key stroke to readjust the pumping rate back down to the normal range, but the pump did not respond to the command.”
The pump then appears to have operated at the increased rate for more than three hours. The report says the “length of time the pump remained in an overspeed condition is the result of distracted staff.”
While the treatment plant is typically staffed with two operators per shift, only one was on site when the issues began, according to the report, because of “training and a dental apportionment” for the second operator.
The investigation also found that “at least five outside contractors” were at the plant on the day in question, with “two of them requiring direct interaction with the operator.”
“Given the reduction of operators on the site on this day, having additional duties contributed to a lack of concentrated focus when the event occurred,” the report continued.
One piece of evidence from the investigation points to the operator noticing “a higher than normal calculated fluoride dose” around 12:25 Thursday afternoon. The operator then wanted to recheck the calculations but “became otherwise engaged in additional discussions” with an on-site outside contractor and told investigators they “forgot to get back to reviewing the data.”
The investigation also says that in “at least two instances” the plant’s standard operating procedures were not “strictly followed.” The report also criticized the review, updating and training process surrounding the SOPs.
“Most SOPs provided for this review were at least four years old and included inaccurate or out of date discussions on expected operational practices,” and the report shows the last recorded training on SOPs “coincided with the SOP initial roll out.”
A secondary independent investigation into the water main break was also commissioned by OWASA.
That report described the 44-year-old 12” main that broke as “on the shorter end, but within the reasonable service life range.” The investigation did report the pipe was buried with “shallow cover” from the pavement above while also having “only six to eight inches of separation” from a storm sewer pipe. Additionally, while the 12” main had been activated to bring water from Durham to OWASA, an initial effort to activate a second interconnection “ran into difficulties and was not successful until later in the day.” Once the secondary main was activated, it produced “significantly less water than the rated capacity.”
“The reports from these consultants are a critical step in our reaching an objective, fact-based understanding of the causes of the disruptive service outage,” said John Young, Chair of the OWASA Board of Directors.
“Last night, the Board agreed that OWASA would develop solutions to prevent the particular failures of last week, assess outage risks more broadly and take action to improve the water system resiliency, and review communication practices of OWASA and its partners during critical events. We may turn again to outside parties for both expertise and objectivity.
“OWASA apologizes for the significant disruptions and impact experienced by the community and thanks its many partners for their support.”
The Do Not Drink/Do Not Use order was lifted for OWASA customers around 2:30 PM Saturday, February 4.
The order had also forced the Orange County Health Department to close all hotels and restaurants served by OWASA for more than 24 hours. The UNC men’s basketball game against Notre Dame was also postponed from Saturday to Sunday and moved from Chapel Hill to Greensboro as a result of the outage.