ELB Termination before Deregistration

Nate Policar's Avatar

Nate Policar

19 Sep, 2015 07:03 AM

When a role is set to scale up/down and set to use an AWS ELB, when scaling down, Scalr terminates the server before it deregisters the instance from the load balancer.

Usually the gap in the log file between the events is only a few seconds, but we have noticed that during that period the terminated instance is still registered on the load balancer, traffice gets routed to it and returns an error to the end user. In some cases, the delay between the two events is significant enough that it fails a 20 second health check on the ELB.

Is there any way to change the behavior so that when an ELB is enabled on a role, Scalr will not terminate the server until it has received confirmation that the instance has been deregistered from the load balancer?

Example from system log:
Sep 18, 2015 15:37:06
c2c46281-001c-47e5-8acc-99634dc15639/FarmLog
Terminating server 'c2c46281-001c-47e5-8acc-99634dc15639' (Platform: ec2) (ServerTerminate).

Sep 18, 2015 15:37:09
c2c46281-001c-47e5-8acc-99634dc15639/FarmLog
Instance 'i-71b5dbaa' deregistered from 'mysql-lb' load balancer

  1. Support Staff 1 Posted by Vitaliy Demidov on 22 Sep, 2015 01:23 PM

    Vitaliy Demidov's Avatar

    Hi Nate,
    We are looking into this problem. Need some time to investigate.
    Thank you for the report.

    Vitaliy

  2. Support Staff 2 Posted by Vitaliy Demidov on 02 Oct, 2015 03:19 PM

    Vitaliy Demidov's Avatar

    Hi Nate,
    The problem is not confirmed. Deregister is triggered twice: the first time it is triggered on BeforeHostTerminate event, the second - on HostDown event. If two events happened during the same hour it stacks only one record in the log with the little difference. It updates the timestamp of the record so that the time of the final event is shown.

    Vitaliy.

  3. 3 Posted by Nate Policar on 02 Oct, 2015 05:00 PM

    Nate Policar's Avatar

    Interesting, because we were receiving Failed Health Check alerts from
    the AWS ELB after Scalr terminated an instance. Not sure how that
    happens if you're saying that Scalr removes the instance from the ELB
    prior to terminating it.

  4. marc closed this discussion on 22 Jul, 2016 07:18 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac