Washington County website restored

For Immediate Release: Monday, May 23, 2022

All Washington County internet services have been restored

en español

Following an unplanned outage affecting Washington County’s website and web-based services on Tuesday evening, all systems are now back online. The outage was the result of hardware failure and not malicious software or some other type of cyberattack. 

As stated by County Administrator Tanya Ange, “We are grateful for our amazing team in Information Technology Services who worked literally night and day to restore these vital systems. As you can imagine, this situation created several challenges to our ability to deliver services and I’m proud of our entire organization for adapting on the fly to meet public need as best we could. I thank each and every one of our employees, as well as members of the public for their patience.” 

The timing of the event was coincidental with the 2022 May Primary Election but did not hinder ballot processing or reporting of results to the Secretary of State’s Office for posting on their website. 

The following questions and answers are provided to help explain the cause of the outage, as well as the recovery process.

What specific hardware failed?
This was a storage hardware failure. A normally redundant component failed in such a way that it caused onsite storage-based systems to be temporarily inaccessible.

Why does the county disaster recovery plan not account for this with cold spare hardware?
The County has a secondary site with hardware and replication in place as well as tape backup for disaster recovery scenarios. In this particular situation, after consulting with our hardware vendor, we determined that it would be more efficient and faster to restore our primary site to service rather than retrieve data from alternate sources within the backup plan. The secondary site is intended for a "worse-case scenario" and recovery time would have been significantly longer to bring all services up on that site. Thanks to a strategic move by Information Technology Services (ITS) some time ago, many of the County’s core services are stored in the cloud and were never impacted by this hardware failure.

What was the total outage time, and for which services?
Critical services such as Office 365 and several other cloud-based services were not impacted by the outage. The County website and other services were impacted for less than 72 hours. 95% of all services were available within 96 hours.

How will this be prevented in the future?
County ITS engages in frequent lessons-learned discussions to determine improvements and enhancements to performance, security, resiliency, and redundancy. Our teams are working through this process now to determine better approaches to ensure recovery in similar unusual scenarios. Initiatives underway include replication of numerous County web services to cloud hosted and more resilient providers.

-

Todos los servicios de Internet del Condado de Washington han sido restaurados

Después de una interrupción no planificada que afectó el sitio web del Condado de Washington y los servicios basados en la web el martes por la noche, todos los sistemas ahora están de nuevo en línea. La interrupción fue el resultado de una falla de hardware y no de software malicioso o algún otro tipo de ataque cibernético. 

Como declaró la Administradora del Condado Tanya Ange, "Estamos agradecidos por nuestro increíble equipo en Servicios de Tecnología de la Información que trabajó literalmente noche y día para restaurar estos sistemas vitales. Como pueden imaginar, esta situación creó varios desafíos a nuestra capacidad de brindar servicios y estoy orgulloso de toda nuestra organización por adaptarse sobre la marcha para satisfacer las necesidades del público lo mejor que pudimos. Agradezco a todos y cada uno de nuestros empleados, así como a los miembros del público por su paciencia." 

El momento del evento fue una coincidencia con la elección primaria de mayo de 2022, pero no obstaculizó el procesamiento de la boletas o la presentación de informes de los resultados a la Oficina del Secretario de Estado para ser publicados en su sitio web. 

Media Contact: