tag:status.rebelmouse.com,2005:/historyRebelMouse Status - Incident History2024-03-27T22:49:48-04:00RebelMousetag:status.rebelmouse.com,2005:Incident/203801532024-03-27T15:51:53-04:002024-03-27T15:51:53-04:00Performance degradation<p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>15:51</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>14:08</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>13:50</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>13:31</var> EDT</small><br><strong>Monitoring</strong> - There was an isolated surge of highly suspicious traffic. While we aren't certain of its origin or source, we have isolated it away from the production clusters to its own environment. This means all productions systems should have returned to normal and we believe the problem is under control. We don't fully understand the why behind this yet though so we will be updating this with more details soon.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>13:02</var> EDT</small><br><strong>Investigating</strong> - We've are experiencing the performance degradation for the logged in users.</p>tag:status.rebelmouse.com,2005:Incident/203023072024-03-24T07:00:56-04:002024-03-24T07:00:56-04:00Talaria DB Maintenance<p><small>Mar <var data-var='date'>24</var>, <var data-var='time'>07:00</var> EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>24</var>, <var data-var='time'>02:00</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>11:40</var> EDT</small><br><strong>Scheduled</strong> - We are planning to implement security updates for the Talaria DB.</p>tag:status.rebelmouse.com,2005:Incident/203022892024-03-21T06:00:56-04:002024-03-21T06:00:56-04:00RabbitMQ Maintenance<p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>06:00</var> EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>02:00</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>11:37</var> EDT</small><br><strong>Scheduled</strong> - We will be performing maintenance on the RabbitMQ instance to update and enhance its capabilities. While we anticipate no issues, there might be slight delays in cache flushing during the process.</p>tag:status.rebelmouse.com,2005:Incident/201737992024-03-07T04:00:56-05:002024-03-07T04:00:56-05:00Stats DB Maintenance<p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>04:00</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>02:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>03:24</var> EST</small><br><strong>Scheduled</strong> - We are going to upgrade the MySQL database used for statistics. Please anticipate a delay of up to 30 minutes in the display of new statistics during the update.</p>tag:status.rebelmouse.com,2005:Incident/200372302024-02-22T06:00:56-05:002024-02-22T06:00:56-05:00Pharos DB Maintenance<p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>06:00</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>03:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>05:38</var> EST</small><br><strong>Scheduled</strong> - We would like to inform you that we are planning to implement security updates for the DocumentDB used by Pharos. During this maintenance period, we anticipate a delay in the display of real-time data on the Pharos dashboard.</p>tag:status.rebelmouse.com,2005:Incident/199749212024-02-18T03:00:56-05:002024-02-18T03:00:56-05:00MySQL DB Maintenance<p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>03:00</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>01:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>08:17</var> EST</small><br><strong>Scheduled</strong> - We are planning to implement security and performance updates for the main MySQL database. We'll turn off crons and celeries during this migration, as well as lock the writing capabilities (such as posts creation) to our MySQL production up to 30 minutes.</p>tag:status.rebelmouse.com,2005:Incident/199431582024-02-08T22:10:05-05:002024-02-12T14:55:10-05:00Performance issues<p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>22:10</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>21:12</var> EST</small><br><strong>Monitoring</strong> - We identified the root cause and deployed a fix for it and now we are monitoring application performance</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>18:47</var> EST</small><br><strong>Update</strong> - We have replaced the last servers and expecting performance to get back to normal in a couple of minutes</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>18:05</var> EST</small><br><strong>Update</strong> - Newly added servers are functioning correctly and we see an improvement in performance. We are now keep adding new servers and manually removing old one that have issues</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>17:52</var> EST</small><br><strong>Update</strong> - We are adding new servers manually to increase a capacity to resolve performance degradation</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>17:30</var> EST</small><br><strong>Identified</strong> - We identified that the issue is caused by Kubernetes cluster not being able to launch new instances. We are working on a fix for that right now</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>17:16</var> EST</small><br><strong>Investigating</strong> - We are experiencing a performance degradation. We are investigating what is a root cause of it right now.</p>tag:status.rebelmouse.com,2005:Incident/199007362024-02-05T07:01:13-05:002024-02-05T07:01:13-05:00Stats DB Maintenance<p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>07:01</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>02:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>08:49</var> EST</small><br><strong>Scheduled</strong> - We are planning to implement security and performance updates for the MySQL database used for statistics. Please anticipate a delay of up to 30 minutes in the display of new statistics during the update.</p>tag:status.rebelmouse.com,2005:Incident/198973392024-02-01T06:30:00-05:002024-02-02T18:57:43-05:00Page rendering issues<p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>06:30</var> EST</small><br><strong>Resolved</strong> - Chronology of the incident (EDT timezone):<br /><br />06:46 AM - Deployment happened at EST<br />09:35 AM - Service Delivery team received the bug report<br />12:53 PM - RebelMouse tech team started a rollback procedure involving CTO and Director of IT Operations<br />01:10 PM - RebelMouse received the report about multiple ongoing issues on some pages like missing styles or text<br />03:47 PM - release was reverted from the production<br /><br /><br />The impact of the incident:<br /><br />Page rendering issues in the case of usage sections intersection feature.<br /><br /><br />The underlying cause:<br />The unexpected problem was caused by an application release to a website's rendering and routing service where we introduced the change for our routing system. This change was needed to give the possibility to implement new routing features like wildcard redirects from the redirects dashboard which in current implementation was impossible.<br /><br /><br />Actions taken:<br />- Initiated a meeting between the Development and QA teams to thoroughly review the incident. The goal was to classify the incident, define and identify the necessary tests to prevent similar issues in the future.<br />- Incident was classified as not Major.<br />- Updated the regression test suite by incorporating new tests specifically designed to cover the sections intersection functionality. This ensures comprehensive testing moving forward.<br />- Addressed and resolved the bug introduced in the initial release by implementing a fix. The fix is aimed at restoring the intended behavior of the section's intersection functionality.<br /><br /><br />Preventive Measures:<br />By FEB 9 - Conducting a comprehensive review of our custom functionalities to identify potential points of vulnerability.<br />By FEB 16 - Implementing additional checks in the QA phase to catch nuanced issues in custom functionalities.<br />Strengthening collaboration between development and QA teams to improve test coverage for less commonly used features.</p>tag:status.rebelmouse.com,2005:Incident/190564472023-11-08T15:00:00-05:002023-11-08T07:31:31-05:00Editorial Tools Outage<p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>15:00</var> EST</small><br><strong>Resolved</strong> - The issue is fully resolved</p>tag:status.rebelmouse.com,2005:Incident/190025132023-11-05T07:00:13-05:002023-11-05T07:00:13-05:00MongoDB & Redis Maintenance<p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>07:00</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>02:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>14:00</var> EDT</small><br><strong>Scheduled</strong> - During this maintenance window we don't expect unavailability of any our services, however slight performance degradation is possible for a short periods of time for logged in user experience.</p>tag:status.rebelmouse.com,2005:Incident/187540962023-10-11T09:44:25-04:002023-10-11T09:44:25-04:00Editorial Tools Performance Degradation<p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>09:44</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>09:11</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.rebelmouse.com,2005:Incident/187529862023-10-11T07:09:04-04:002023-10-11T10:57:22-04:00Editorial Tools Issues<p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>07:09</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>07:07</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>06:45</var> EDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>06:44</var> EDT</small><br><strong>Investigating</strong> - Editors may currently experience unavailability of publishing tools. Our team is already working on the recovery.</p>tag:status.rebelmouse.com,2005:Incident/186670882023-10-04T06:00:22-04:002023-10-04T06:00:22-04:00Network Infrastructure Maintenance<p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>06:00</var> EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>02:00</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>10:34</var> EDT</small><br><strong>Scheduled</strong> - We are performing an update to our network infrastructure aimed at improving our service quality. This upgrade involves optimizing network routes by minimizing the number of hops each data packet needs to take. The goal of this optimization process is to reduce round-trip time (RTT) and potentially enhance the speed and fluidity of data flow within the network.</p>tag:status.rebelmouse.com,2005:Incident/185652242023-09-21T07:54:15-04:002023-09-21T09:16:05-04:00Posts Loading Issue<p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>07:54</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>07:33</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>07:12</var> EDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>07:08</var> EDT</small><br><strong>Investigating</strong> - We've faced a problem during the deploy which caused problems with posts loading. We are already solving the problem.</p>tag:status.rebelmouse.com,2005:Incident/184572232023-09-10T08:49:11-04:002023-09-11T07:51:36-04:00Users Unable to Publish Content<p><small>Sep <var data-var='date'>10</var>, <var data-var='time'>08:49</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>10</var>, <var data-var='time'>08:25</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'>10</var>, <var data-var='time'>07:50</var> EDT</small><br><strong>Identified</strong> - We are experiencing unexpected issue with publishing workflow during our scheduled maintenance to a new version of Redis.<br /><br />Only logged in users are affected - regular logout users are not experiencing any issues<br /><br />The team is working on a deploy and we should have things back on track in next 15 mins</p>tag:status.rebelmouse.com,2005:Incident/184382862023-09-10T07:00:01-04:002023-09-10T07:00:01-04:00Redis Maintenance<p><small>Sep <var data-var='date'>10</var>, <var data-var='time'>07:00</var> EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>10</var>, <var data-var='time'>02:01</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>13:57</var> EDT</small><br><strong>Scheduled</strong> - We are going to upgrade our Redis to a new version. This Redis upgrade is necessary to:<br /><br />1. Improve Performance: The new Redis version introduces optimizations that will enhance the speed and responsiveness of our applications.<br />2. Enhance Security: Security patches and updates are included in this release to fortify our Redis infrastructure against potential vulnerabilities.<br />3. Strengthen Reliability: This upgrade aims to increase the overall stability and reliability of our services.<br /><br />During this maintenance window we don't expect unavailability of any our services, however slight performance degradation is possible for a short periods of time.</p>tag:status.rebelmouse.com,2005:Incident/183999392023-09-05T16:30:00-04:002023-09-05T16:57:29-04:00Performance degradation<p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>16:30</var> EDT</small><br><strong>Resolved</strong> - On September 5, 2023, between 4:17 PM EST and 4:22 PM EST, our services experienced a performance degradation. Our team is actively investigating the root causes, and we will furnish further information once the investigation is complete.</p>tag:status.rebelmouse.com,2005:Incident/183510822023-09-01T13:26:44-04:002023-09-01T13:51:51-04:00Periodic Tasks Delay<p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>13:26</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>13:21</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>13:08</var> EDT</small><br><strong>Update</strong> - We have encountered unexpected challenges during the instance launch process, and we regret to inform you that we require additional time to resolve the issue and fully restore the Celery Beat service. Our technical team is actively working to address the underlying problems and expedite the recovery process.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>12:48</var> EDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>12:48</var> EDT</small><br><strong>Investigating</strong> - We are experiencing problems with processing of periodic tasks like sending newsletters, posts scheduling, social scheduling, feeds processing and others. <br /><br />Celery beat instance responsible for managing these tasks was taken out of service by AWS due to unhealthy state. We are replacing the instance right now. The process will take up to 15 min. After we launch the instance, all the collected tasks will be processed.</p>tag:status.rebelmouse.com,2005:Incident/182474502023-08-24T12:00:09-04:002023-08-25T04:42:08-04:00Performance Degradation<p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>12:00</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>11:38</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>11:32</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.rebelmouse.com,2005:Incident/181836562023-08-18T07:34:36-04:002023-08-18T07:36:30-04:00Images Load Issue<p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>07:34</var> EDT</small><br><strong>Resolved</strong> - During a recent maintenance period dedicated to enhancing page speed load performance, an incident occurred that led to certain images responding with a 404 status. <br /><br />The incident was triggered by modifications to the nginx headers for images as part of our efforts to optimize page speed load times. The change inadvertently caused certain images to respond with a 404 status, rendering them inaccessible to users. Some scenario had not been accounted for during testing, leading to the oversight.<br /><br />For future nginx changes we will add a test step to verify image loading.</p>tag:status.rebelmouse.com,2005:Incident/181536782023-08-17T07:00:24-04:002023-08-17T07:00:24-04:00Caching Layer Maintenance<p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>07:00</var> EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>02:00</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>11:43</var> EDT</small><br><strong>Scheduled</strong> - In order to improve performance of the caching service we're going to upgrade underlying hardware for all of the deployments, we don't expect downtime or performance degradation during the event.</p>tag:status.rebelmouse.com,2005:Incident/181677432023-08-16T21:19:14-04:002023-08-16T23:40:12-04:00Performance degradation for logged in experience<p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>21:19</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>20:51</var> EDT</small><br><strong>Monitoring</strong> - The issue was resolved, we are monitoring the solution and preparing postmortem</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>20:19</var> EDT</small><br><strong>Identified</strong> - We identified a source of issue and working on the fix</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>19:45</var> EDT</small><br><strong>Investigating</strong> - We are experiencing degradation for editorial experience. We are investigating what is a cause and fixing it ASAP</p>tag:status.rebelmouse.com,2005:Incident/179466772023-07-24T11:49:34-04:002023-07-24T11:49:34-04:00Problems during publishing<p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>11:49</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>11:47</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>11:38</var> EDT</small><br><strong>Identified</strong> - We are investigating issues reported while publishing articles.</p>tag:status.rebelmouse.com,2005:Incident/178534782023-07-16T06:01:43-04:002023-07-16T06:01:43-04:00MongoDB Cluster Maintenance<p><small>Jul <var data-var='date'>16</var>, <var data-var='time'>06:01</var> EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>16</var>, <var data-var='time'>02:00</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>07:41</var> EDT</small><br><strong>Scheduled</strong> - During this maintenance window we don't expect unavailability of any our services, however slight performance degradation is possible for a short periods of time for logged in user experience.</p>