Fw: Update from Canvas: Some Canvas users were running into page errors when accessing Canvas on 4/28/2019

​FYI.

---

From: Canvas Support <support@instructure-contact.com>
Sent: Tuesday, April 30, 2019 5:52 PM
To: Scott Vigallon
Subject: Update from Canvas: Some Canvas users were running into page errors when accessing Canvas on 4/28/2019

 

Some Canvas users were running into page errors when accessing Canvas on 4/28/2019 OVERVIEW Between 6:40 PM and 7:05 PM MDT on April 28 2019, some Canvas users experienced slow loading times or page errors. This was caused by a server that became unresponsive, and needed to be restarted.   DETAILS At 6:40 PM MDT, users began seeing 504 page errors when trying to access courses, and pages within Canvas. This was caused by a job server that became unresponsive to requests. Job servers are responsible for processing server requests, including course migration files, imports and exports, and gradebook downloads. These requests had started to queue when the jobs server became unresponsive, resulting in the errors users were seeing. Our DevOps team was alerted to the issue by automated monitoring we have in place, and they quickly identified an unresponsive jobs server as the culprit. They restarted the job server at 6:53 PM. After the job server finished restarting, we verified that the timeout errors were no longer occurring. The queued job requests still needed to be processed, and resumed after the server restart. This resulted in some additional slow page load times until the queued jobs had been processed. The incident was resolved and Canvas was functioning as normal, at 7:03 PM MDT.   MITIGATION The root cause of the slow load times were due to the underlying database server being unresponsive. Such unresponsiveness isn’t something that can be foreseen, which is why we put measures in place to monitor and alert our team. Our goal is to resume server activity as soon as possible after such an event like this occurs. CONCLUSION We apologize for the inconvenience to you and your users. We understand any service interruptions while using Canvas directly impact your ability to serve your students and teachers. We appreciate the trust you have placed in us, and we will continue to apply the best technology and procedures to minimize future issues.   Copyright © 2019 Instructure, All rights reserved. You are receiving this message because you're listed as a Canvas Admin for your institution and either you or your Customer Success Manager requested that you be added to this mailing list. Want to change how you receive these emails? You can update your preferences or unsubscribe from this list.