All AWS servers turned OFF

Felipe Rojas Palma's Avatar

Felipe Rojas Palma

18 Jul, 2016 03:11 PM

Hi,
today at 13:57 UTC all servers managed by Scalr in AWS was turned off. This is devastating for our clients.
We are investigating causes, and trying to restore the lost instances, and databases.

There's 2 kind of logs in Scalr related with this catastrophe:
1 - Message: Unable to read LoadAverage value from server #1: 54.227.104.242: Wrong crypto key!
2 - Message: Server d423fd2e-53dd-4ef4-a889-18d2a0bc26e8 (ec2) found in Scalr but not found in the cloud. Terminating.

All instances was terminated at the same time, with the same log.

Please, can you give us some ideas to figure out what triggered this issue?

  1. 1 Posted by marc on 18 Jul, 2016 03:30 PM

    marc's Avatar

    Hi Felipe,

    We're investigating this issue and will provide details shortly. Right now all our team is focused on service recovery.

    Many thanks,
    Wm. Marc O'Brien
    Scalr Technical Support

  2. Support Staff 2 Posted by Michael Lochead on 18 Jul, 2016 11:06 PM

    Michael Lochead's Avatar

    Hi Felipe,

    As communicated, Scalr experienced a critical error this morning as detailed here: http://support.scalr.net/discussions/problems/26464-hosted-scalr-se...

    At this point, we have completed automated recovery of all impacted Servers. If you are experiencing any remaining impact to your service, please re-open this ticket, or create a new one and we will put you in direct contact with our engineering team to resolve the outstanding issues.

    Thank you for your patience and collaboration in helping to address this issue.  We will send a final report on the problem soon.

    Best,
    Michael

  3. Michael Lochead closed this discussion on 18 Jul, 2016 11:06 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac