tag:status-oregon.sendbird.com,2005:/historySendbird Oregon Status - Incident History2024-03-29T04:54:42-07:00Sendbird Oregontag:status-oregon.sendbird.com,2005:Incident/203903962024-04-02T15:30:00-07:002024-03-28T11:29:11-07:00Major version upgrade of the database (Dashboard)<p><strong>THIS IS A SCHEDULED EVENT Apr <var data-var='date'>2</var>, <var data-var='time'>15:30</var> - <var data-var='time'>15:35</var> PDT</strong></p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>11:29</var> PDT</small><br><strong>Scheduled</strong> - Date and Time<br />* Tue Apr 2nd 22:30 - 22:35 UTC<br />* Tue Apr 2nd 15:30 - 15:35 PDT<br />* Wed Apr 3rd 07:30 - 07:35 KST<br /><br />Duration of downtime: Up to 5 minutes during the maintenance window<br /><br />Purpose: Major version upgrade of the database (Dashboard)<br /><br />Impact:<br />* The Dashboard will be unavailable<br />* The Desk service will e impacted during the maintenance window, as Desk Agents will be unable to log into Dashboard to handle tickets.<br /><br />What this means:<br />* Customers will not be able to access the Sendbird Dashboard. Calls dashboard, Moderation dashboard, and Desk agent dashboard<br />* Specifically for Desk customers’ users will still be able to create tickets, however agents will not be able to answer tickets during the actual maintenance time. We recommend our customers set working hours within the Dashboard to send auto messages, during the maintenance window<br />* NOTE: Sendbird Calls, Chat, and Desk services themselves will not be interrupted during the maintenance activity</p>tag:status-oregon.sendbird.com,2005:Incident/199896632024-02-26T03:00:57-08:002024-02-26T03:00:58-08:00All services impacted during bi-yearly maintenance<p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>03:00</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>02:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>23:06</var> PST</small><br><strong>Scheduled</strong> - * Date and Time<br /> * Feb 26 10:00 - 11:00 UTC<br /> * Feb 26 02:00 - 03:00 PST<br /> * Feb 26 19:00 - 20:00 KST<br />* Duration of downtime: 20 minutes during the maintenance window<br />* Purpose: major version upgrade of the database (bi-yearly)<br />* Service impact<br /> * Full outage (APIs will get 5xx errors) during the downtime<br /> * Announcements, GDPR will be stopped for an hour beforehand<br />* Affected product: Chat, Desk, and Calls</p>tag:status-oregon.sendbird.com,2005:Incident/199457842024-02-08T23:56:20-08:002024-02-08T23:56:20-08:00Message Threading partial failure<p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>23:56</var> PST</small><br><strong>Resolved</strong> - Hotfix has been deployed and we have confirmed that threading is back to operational</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>23:10</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>23:09</var> PST</small><br><strong>Investigating</strong> - We have identified that the message threading functionality is partially broken. <br />Creating the first threaded message works, but adding thread messages to existing threads have been broken since Feb 7 9 pm PST<br />We are rolling out the hotfix immediately (Feb 8 11 pm PST)</p>tag:status-oregon.sendbird.com,2005:Incident/196753542024-01-11T00:05:39-08:002024-01-11T00:05:39-08:00Elevated API Errors<p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>00:05</var> PST</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>00:04</var> PST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-oregon.sendbird.com,2005:Incident/196629602024-01-09T18:53:41-08:002024-01-09T18:53:41-08:00Elevated API Errors<p><small>Jan <var data-var='date'> 9</var>, <var data-var='time'>18:53</var> PST</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Jan <var data-var='date'> 9</var>, <var data-var='time'>18:51</var> PST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-oregon.sendbird.com,2005:Incident/188917262023-10-22T15:02:33-07:002023-10-22T15:02:33-07:00Elevated API Errors<p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>15:02</var> PDT</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>15:02</var> PDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-oregon.sendbird.com,2005:Incident/188418302023-10-18T09:41:44-07:002023-10-18T12:14:52-07:00Elevated API Errors<p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>09:41</var> PDT</small><br><strong>Resolved</strong> - The servers have returned operational. <br /><br />08:53 AM- 09:10 AM. Major Outage (error rate above 50%)<br />9:10 AM, traffic recovered with minimal failures <1% .<br />9:41 AM PDT, services operational</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>08:47</var> PDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-oregon.sendbird.com,2005:Incident/188094722023-10-15T22:16:42-07:002023-10-15T22:16:42-07:00Elevated API Errors<p><small>Oct <var data-var='date'>15</var>, <var data-var='time'>22:16</var> PDT</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Oct <var data-var='date'>15</var>, <var data-var='time'>22:08</var> PDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-oregon.sendbird.com,2005:Incident/186384842023-09-28T19:55:39-07:002023-09-28T19:55:39-07:00Elevated API Errors<p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>19:55</var> PDT</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>19:54</var> PDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-oregon.sendbird.com,2005:Incident/185650572023-09-21T03:48:54-07:002023-09-21T03:48:54-07:00Elevated API Errors<p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>03:48</var> PDT</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>03:42</var> PDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-oregon.sendbird.com,2005:Incident/184935442023-09-13T16:24:42-07:002023-09-14T15:31:34-07:00Full outage<p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>16:24</var> PDT</small><br><strong>Resolved</strong> - The servers have returned operational.<br /><br />When: 12:54 - 16:24 PDT Sept 13 (19:54 - 23:24 GMT Sept 13; 04:54 - 08:24 KST Sept 14)<br />Impact: Full outage - all services unavailable<br />Root cause: Ongoing investigation into Redis 100% CPU utilization which caused broader infrastructure resource issues.<br />Remediation: Restarted instances while actively managing instance allocation and incoming traffic.</p><p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>12:54</var> PDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-oregon.sendbird.com,2005:Incident/184838222023-09-12T01:34:00-07:002023-09-12T14:12:44-07:00Full outage<p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>01:34</var> PDT</small><br><strong>Resolved</strong> - When: 01:08 - 01:34 PDT September 12 (08:08 - 08:34 GMT; 17:08 - 17:34 KST)<br />Impact: Full outage - all services unavailable<br />Root cause: Ongoing investigation with AWS regarding high engine CPU utilization and ElastiCache replication lag that caused auto-scaling policies to take effect.<br />Remediation: Terminated instances, and scaled in auto-scaled resources.</p>tag:status-oregon.sendbird.com,2005:Incident/181787722023-08-17T04:20:00-07:002023-08-17T16:25:42-07:00Intermittent web socket request failures<p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>04:20</var> PDT</small><br><strong>Resolved</strong> - When: 03:50 - 04:20 PDT August 17 (10:50 - 11:20 GMT, 19:50 - 20:20 KST)<br />Impact: Partial outage, approximately 30% of web socket requests failed<br />Root cause: Investigation ongoing - unexpected AWS load balancer issue when pushing a Sendbird platform update; actively investigation with AWS.<br />Remediation: Restarted impacted services - normal service restored</p>tag:status-oregon.sendbird.com,2005:Incident/180485252023-08-03T22:55:37-07:002023-08-03T22:55:37-07:00Elevated API Errors<p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>22:55</var> PDT</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>22:54</var> PDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-oregon.sendbird.com,2005:Incident/177959922023-07-07T20:26:40-07:002023-07-07T20:26:42-07:00Elevated API Errors<p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>20:26</var> PDT</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>20:24</var> PDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-oregon.sendbird.com,2005:Incident/177915242023-07-07T08:56:41-07:002023-07-07T08:56:41-07:00Elevated API Errors<p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>08:56</var> PDT</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>08:54</var> PDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-oregon.sendbird.com,2005:Incident/177087082023-06-28T05:35:57-07:002023-06-28T05:35:59-07:00Elevated API Errors<p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>05:35</var> PDT</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>05:33</var> PDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-oregon.sendbird.com,2005:Incident/176846012023-06-25T22:47:37-07:002023-06-25T22:47:37-07:00Elevated API Errors<p><small>Jun <var data-var='date'>25</var>, <var data-var='time'>22:47</var> PDT</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Jun <var data-var='date'>25</var>, <var data-var='time'>22:41</var> PDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-oregon.sendbird.com,2005:Incident/171891112023-05-08T23:01:41-07:002023-05-08T23:01:41-07:00Elevated API Errors<p><small>May <var data-var='date'> 8</var>, <var data-var='time'>23:01</var> PDT</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>May <var data-var='date'> 8</var>, <var data-var='time'>23:00</var> PDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-oregon.sendbird.com,2005:Incident/168529532023-04-13T09:12:41-07:002023-04-13T09:12:41-07:00Elevated API Errors<p><small>Apr <var data-var='date'>13</var>, <var data-var='time'>09:12</var> PDT</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Apr <var data-var='date'>13</var>, <var data-var='time'>09:09</var> PDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-oregon.sendbird.com,2005:Incident/165785932023-03-20T12:00:32-07:002023-03-20T12:00:32-07:00Internal issue only - no impact to product/platform functionality<p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>12:00</var> PDT</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>11:18</var> PDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>11:17</var> PDT</small><br><strong>Identified</strong> - This incident has been raised due to an internal issue with Sendbird's monitoring tools. Product and platform functionality is not impacted.<br /><br />This incident will remain open as the necessary components are updated.</p><p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>11:00</var> PDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-oregon.sendbird.com,2005:Incident/165778282023-03-20T10:59:37-07:002023-03-20T11:19:30-07:00Internal issue only - no impact to product/platform functionality<p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>10:59</var> PDT</small><br><strong>Resolved</strong> - The servers have returned operational.<br /><br />This incident was raised due to an internal issue with Sendbird's monitoring tools. Product and platform functionality is not impacted.</p><p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>09:59</var> PDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-oregon.sendbird.com,2005:Incident/163435422023-03-04T14:54:42-08:002023-03-04T14:54:43-08:00Elevated API Errors<p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>14:54</var> PST</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>14:53</var> PST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-oregon.sendbird.com,2005:Incident/163435012023-03-04T14:46:45-08:002023-03-04T14:46:46-08:00Elevated API Errors<p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>14:46</var> PST</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>14:44</var> PST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-oregon.sendbird.com,2005:Incident/163431152023-03-04T12:59:46-08:002023-03-04T12:59:46-08:00Elevated API Errors<p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>12:59</var> PST</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>12:55</var> PST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>