Urgent: Our farm is not booting

Mark Whelan's Avatar

Mark Whelan

18 Jul, 2016 01:30 PM

Hi,

We are having issues with Farm 13241 booting, all of the servers are failing to connect to scalarizer and some servers are stuck in initialising, could you advise the problem (everything was working this morning)

https://my.scalr.net/#/servers?farmId=13241

Thanks,

Thomas

  1. 1 Posted by Alex Kira on 18 Jul, 2016 01:37 PM

    Alex Kira's Avatar

    This has just happened to all farms under my account as well. All servers have been rebooted. A few are stuck on "initializing". Mysql instance is now constantly restarting - this is causing site to be down. Farms 5017 and 15246.

    Looking at AWS console, seems like multiple instances of the same server role are up and running but Scalr has lost connection to them.

    After updates:
    Please relink the old servers to Scalr. We have had issues with EBS, and we have not snapshotted latest state so the new servers will not be current.

  2. 2 Posted by Mark Whelan on 18 Jul, 2016 01:37 PM

    Mark Whelan's Avatar

    I'm on annual leave and will pick up your email up on Monday 25th July on my return.

    Many thanks,
    Mark

  3. 3 Posted by Kevin Sample on 18 Jul, 2016 01:41 PM

    Kevin Sample's Avatar

    Ditto here--every farm in my env is being rebuilt. Not cool, man.

    22645
    22649
    22650
    22668
    22868

  4. 4 Posted by Chris Cooper-Jo... on 18 Jul, 2016 01:41 PM

    Chris Cooper-Jones's Avatar

    Same here

  5. 5 Posted by Arie Fishler on 18 Jul, 2016 02:21 PM

    Arie Fishler's Avatar

    We see this too. But check your amazon consoles if you are on AWS. It seems this was some flase trigger on Scalr and the original servers, at least for us, are still alive.

    Scalr removed them from display and started new ones.

  6. 6 Posted by Kevin Sample on 18 Jul, 2016 02:34 PM

    Kevin Sample's Avatar

    Thanks Arie. That's absolutely correct. All I had to do was reassign the EIPs that got ganked when Scalr tried to rebuild and we're back up. Still not cool though.

  7. 7 Posted by Nir Ben-Dor on 18 Jul, 2016 02:39 PM

    Nir Ben-Dor's Avatar

    Hi Kevin,

    Do you mean elasic IP ?
    For us it tries to start servers but the volumes are not free due to the duplicate instances so new servers fail to start and Scalr doesnt see the old ones so its stuck in this state.

    How can EIP help?
    Please elaborate... do you get the errors that servers cant start due to volume attached to other servers?

  8. 8 Posted by Kevin Sample on 18 Jul, 2016 02:44 PM

    Kevin Sample's Avatar

    I terminated all my farms and manually reassigned the EIPs to the still-running instances through the AWS console. No Scalr-based orchestration is available, but at least we're up & running. Of course, YMMV.

  9. Support Staff 9 Posted by Marat Komarov on 18 Jul, 2016 03:05 PM

    Marat Komarov's Avatar

    We're investigating this issue and will provide details later. Right now all our team is focused on service recovery.

  10. 10 Posted by Kevin Sample on 18 Jul, 2016 03:13 PM

    Kevin Sample's Avatar

    Nir:

    I got the volume attachment errors too, but what killed us was Scalr automatically re-assigning all our Elastic IPs to the new instances it was trying to build. Once I killed all my farms (to stop them from rebuilding and stealing the EIPs) I could then manually re-assign the EIPs to the correct instances and access was restored to my customers.

    Now Scalr has to help me figure out how get my instances back into the UI w/out having to trigger another rebuild. Womp womp.

  11. Support Staff 11 Posted by Igor Savchenko on 18 Jul, 2016 10:46 PM

    Igor Savchenko's Avatar

    Guys, I see a lot of customers in one thread. It's hard to track. Please create a separate ticket (if you don't have one).

    Thanks,
    Igor

  12. marc closed this discussion on 18 Jul, 2016 11:11 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions

02 Jul, 2019 07:54 PM
07 Jun, 2019 07:12 PM
02 May, 2019 04:04 PM
28 Mar, 2019 05:24 PM
22 Feb, 2019 08:11 PM