servers stuck in initializing

dan's Avatar

dan

18 Dec, 2014 05:14 PM

this has been happening on servers for at least the last 30 minutes, possibly longer, on all the new instances. they start, run all the scripts, but stay in initializing forever. i grabbed some information from the scalarizr log below.

farm 13308
role 46201
server 53415126-942e-4896-9686-bf1fb2e5bace

2014-12-18 17:09:00,786+00:00 - INFO - scalarizr.app - [pid: 2422] Starting scalarizr 2.12.5
2014-12-18 17:09:01,014+00:00 - INFO - scalarizr.app - UpdateClient state: completed
2014-12-18 17:09:11,022+00:00 - INFO - scalarizr.messaging.p2p.consumer - Building message consumer server on 0.0.0.0:8013
2014-12-18 17:09:11,023+00:00 - INFO - scalarizr.app - Starting API server on http://0.0.0.0:8010
2014-12-18 17:09:11,048+00:00 - INFO - scalarizr.handlers.lifecycle - Normal start
2014-12-18 17:09:11,049+00:00 - ERROR - scalarizr.handlers - Unrecoverable! Can't find operation 'system.init'. This may caused by Scalarizr restart during initialization. If you're a developer, digging into this problem, see _start_init() method

  1. 1 Posted by dan on 18 Dec, 2014 10:58 PM

    dan's Avatar

    still having this issue. this is a big problem in terms of being able to scale up to meet increasing demand.

  2. Support Staff 2 Posted by Marat Komarov on 18 Dec, 2014 11:12 PM

    Marat Komarov's Avatar

    One of BeforeHostUp scripts lead to Scalarizr restart

    2014-12-18 22:58:21,641+00:00 - INFO - scalarizr.handlers - Executing 6 BeforeHostUp script(s)
    2014-12-18 22:58:33,355+00:00 - INFO - scalarizr.app - [pid: 2447] Starting scalarizr 2.12.5
    

    It's not supported. Scalarizr couldn't be safely restarted before Server will reach Running state.

  3. 3 Posted by dan on 18 Dec, 2014 11:16 PM

    dan's Avatar

    Its getting the unrecoverable error before any of the scripts are executed. Also, the same scripts are used in two other farms and not generating the same errors.  The scripts do actually execute correctly and the server is in a good condition otherwise, but it's useless unless it gets out of "initializing"

  4. Support Staff 4 Posted by Marat Komarov on 18 Dec, 2014 11:20 PM

    Marat Komarov's Avatar

    From debug log it's visible, that Scalarizr executes script, then killed by someone, and started again

    2014-12-18 22:58:22,517+00:00 - DEBUG - scalarizr.handlers.script_executor.1418943502.76 - Executing /bin/bash
      /usr/local/bin/scalr-scripting.1418943502.76/install_security_updates_precise
      1>/var/log/scalarizr/scripting/install_security_updates_precise.BeforeHostUp.d90be2d4-369c-4eed-a0c8-3d782c30d6d0-out.log
      2>/var/log/scalarizr/scripting/install_security_updates_precise.BeforeHostUp.d90be2d4-369c-4eed-a0c8-3d782c30d6d0-err.log
      timeout: 180 seconds
    2014-12-18 22:58:22,528+00:00 - DEBUG - scalarizr.handlers.script_executor.1418943502.76 - Communicating with /bin/bash (pid: 1798)
    2014-12-18 22:58:33,355+00:00 - INFO - scalarizr.app - [pid: 2447] Starting scalarizr 2.12.5
    

    Regards,
    Marat

  5. 5 Posted by dan on 18 Dec, 2014 11:26 PM

    dan's Avatar

    the error started happening at the same time that the scalarizr version was
    increased on this farm from 2.12.4 to 2.12.5. also, our other farms
    running the same script are not having this issue. the script in question
    does nothing related to scalarizr at all. here are the contents:

      apt-get update
      apt-get install -y openssl libssl1.0.0

  6. Support Staff 6 Posted by Marat Komarov on 18 Dec, 2014 11:36 PM

    Marat Komarov's Avatar

    There are several app servers on this farm, initialized fine with 2.12.5. For example 54.177.124.130 and 54.215.39.153.

    I'm sure this issue is not related to 2.12.5, and i agree that apt-get update should't be the cause.

    Maybe crobtab? or any of other tasks restart Scalarizr?

  7. 7 Posted by dan on 18 Dec, 2014 11:42 PM

    dan's Avatar

    there are 2 that succeeded. there were many many others that failed, and i
    killed them. there is currently a server sitting in this state now that i
    have not yet killed. i have been giving them 30 minutes, then terminating
    them.

  8. Support Staff 8 Posted by Igor Savchenko on 18 Dec, 2014 11:45 PM

    Igor Savchenko's Avatar

    As an experiment, can you please try to disable all BeforeHostUp scripts and see if instance initialize or not. If yes, then start to turn them on one by one to see which one causing trouble and then we will help you to find the reason. If instances will fail to initialize without scripts, then will try to dig down and find what restarts scalarizr.

    Thanks,
    Igor

  9. Support Staff 9 Posted by Igor Savchenko on 18 Dec, 2014 11:45 PM

    Igor Savchenko's Avatar

    Initially I mentioned HostUp scripts, but meant BeforeHostUp ones.

  10. 10 Posted by dan on 18 Dec, 2014 11:46 PM

    dan's Avatar

    would it be ok to make them hostup scripts instead of removing them? its a
    pain to remove them completely and then put them back in.

  11. 11 Posted by dan on 18 Dec, 2014 11:56 PM

    dan's Avatar

    The server started. I don't understand why my scripts, which work on other
    farms with scalarizr version 3.1.24 work, and previously worked on this
    farm with 2.12.4, but don't work on this farm with 2.12.5.

    I suppose this solution will work, having the scripts run on HostUp, but
    isn't optimal

  12. Support Staff 12 Posted by Igor Savchenko on 18 Dec, 2014 11:59 PM

    Igor Savchenko's Avatar

    Instead of removing them, you can just set Execution target to "DO NOT EXECUTE".

    Execute them on HostUp would solve issue. But I still would advise you to find out what causing scalarizr restart.

    Regards,
    Igor

  13. 13 Posted by dan on 19 Dec, 2014 12:39 AM

    dan's Avatar

    its something dependent on that one version of scalarizr. there is quite a
    bit in those scripts, and startup times are so long, it'll take way too
    long to pinpoint it, so i'll leave it for now.

  14. Marat Komarov closed this discussion on 20 Jan, 2015 11:34 AM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions

02 Jul, 2019 07:54 PM
07 Jun, 2019 07:12 PM
02 May, 2019 04:04 PM
28 Mar, 2019 05:24 PM
22 Feb, 2019 08:11 PM