06:46 AM - Deployment happened at EST 09:35 AM - Service Delivery team received the bug report 12:53 PM - RebelMouse tech team started a rollback procedure involving CTO and Director of IT Operations 01:10 PM - RebelMouse received the report about multiple ongoing issues on some pages like missing styles or text 03:47 PM - release was reverted from the production
The impact of the incident:
Page rendering issues in the case of usage sections intersection feature.
The underlying cause: The unexpected problem was caused by an application release to a website's rendering and routing service where we introduced the change for our routing system. This change was needed to give the possibility to implement new routing features like wildcard redirects from the redirects dashboard which in current implementation was impossible.
Actions taken: - Initiated a meeting between the Development and QA teams to thoroughly review the incident. The goal was to classify the incident, define and identify the necessary tests to prevent similar issues in the future. - Incident was classified as not Major. - Updated the regression test suite by incorporating new tests specifically designed to cover the sections intersection functionality. This ensures comprehensive testing moving forward. - Addressed and resolved the bug introduced in the initial release by implementing a fix. The fix is aimed at restoring the intended behavior of the section's intersection functionality.
Preventive Measures: By FEB 9 - Conducting a comprehensive review of our custom functionalities to identify potential points of vulnerability. By FEB 16 - Implementing additional checks in the QA phase to catch nuanced issues in custom functionalities. Strengthening collaboration between development and QA teams to improve test coverage for less commonly used features.