My database seems to be down - master AND slave - can anyone point to a problem?

Arie Fishler's Avatar

Arie Fishler

17 Jun, 2017 03:46 AM

Any help would be greatly appreciated - farm 923

  1. 1 Posted by Arie Fishler on 17 Jun, 2017 03:49 AM

    Arie Fishler's Avatar

    It seems to be something is wrong with the EBS volumes. What's going on?

  2. Support Staff 2 Posted by Marat Komarov on 17 Jun, 2017 03:55 AM

    Marat Komarov's Avatar

    We're checking, and will get back to you shortly

    Marat.

  3. 3 Posted by Arie Fishler on 17 Jun, 2017 04:00 AM

    Arie Fishler's Avatar

    Thanks

  4. 4 Posted by Arie Fishler on 17 Jun, 2017 04:54 AM

    Arie Fishler's Avatar

    Any update?

  5. Support Staff 5 Posted by Marat Komarov on 17 Jun, 2017 05:11 AM

    Marat Komarov's Avatar

    Both Master/Slave data volumes are detached from instances. Master volume is "available" on AWS. I have no data about the Slave one. The OS still knows both volumes like mounted. Lots of I/O errors in dmesg. This points that detach was done abnormally.

    The Slave server lost his volume no later than May 10. Master hangs in rebooting, and we have no access to it.

    I suggest replacing both servers.

    Regards,
    Marat

  6. 6 Posted by Arie Fishler on 17 Jun, 2017 05:15 AM

    Arie Fishler's Avatar

    Terminate both?

  7. 7 Posted by Arie Fishler on 17 Jun, 2017 05:19 AM

    Arie Fishler's Avatar

    Scalr will handle reattach?

  8. Support Staff 8 Posted by Marat Komarov on 17 Jun, 2017 05:24 AM

    Marat Komarov's Avatar

    Right. Terminate the whole farm, then launch it.

  9. 9 Posted by Arie Fishler on 17 Jun, 2017 05:30 AM

    Arie Fishler's Avatar

    If i terminate thr farm will it attach the same volume?

  10. Support Staff 10 Posted by Marat Komarov on 17 Jun, 2017 05:37 AM

    Marat Komarov's Avatar

    Please make sure that the volume Scalr holds information about is the right one (Percona status page). Manually re-attached volumes are not reflected in farm configuration.

  11. 11 Posted by Arie Fishler on 18 Jun, 2017 11:19 AM

    Arie Fishler's Avatar

    Marat, thanks for your help. It was really important for us to get the fast response and you pointed the correct way so all went pretty smooth. Much appreciated.

  12. Support Staff 12 Posted by Marat Komarov on 19 Jun, 2017 10:50 PM

    Marat Komarov's Avatar

    Arie, really happy being helpful. Can we close this?

  13. 13 Posted by Arie Fishler on 20 Jun, 2017 07:09 AM

    Arie Fishler's Avatar

    Sure. can close.

  14. Marat Komarov closed this discussion on 20 Jun, 2017 07:22 AM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac