Scalr connect communicate to one of our servers after an image snapshot start yesterda

Fred Rees's Avatar

Fred Rees

08 Sep, 2014 06:30 PM

Scalr is having issues communicating on our app-apache64-centos-ebs-php-configured-crontasks #1 server. I started a server snapshot yesterday which appears did not complete and has left the server in an awkward state.

From the log (/var/log/scalarizr_update.log):

2014-09-08 14:18:11,821 - INFO - upd.client - Starting updclient (pid: 1644)
2014-09-08 14:18:11,822 - DEBUG - upd.client - Starting API on port 8008
2014-09-08 14:18:11,897 - DEBUG - scalarizr.util - system: dmidecode -s system-uuid
2014-09-08 14:18:11,906 - DEBUG - scalarizr.updclient.api - dmidecide returns empty UUID
2014-09-08 14:18:11,914 - DEBUG - scalarizr.util.metadata.cloudstack - Use DHCP leases file: /var/lib/dhclient/dhclient-eth0.leases
2014-09-08 14:18:11,915 - DEBUG - scalarizr.util.metadata.cloudstack - Use DHCP server: 169.254.1.0
2014-09-08 14:18:11,915 - DEBUG - scalarizr.util.metadata.file - Try '/etc/.scalr-user-data': not exists
2014-09-08 14:18:11,915 - DEBUG - scalarizr.util.metadata.file - Try '/etc/scalr/private.d/.user-data': not exists
2014-09-08 14:18:11,916 - DEBUG - scalarizr.util.metadata.gce - Try 'http://metadata/computeMetadata/v1': <urlopen error [Errno -2] Name or service not known>
2014-09-08 14:18:11,918 - DEBUG - scalarizr.util.metadata.ec2 - matched instance_id
2014-09-08 14:18:11,919 - DEBUG - scalarizr.util.metadata.openstack-query - Try 'http://169.254.169.254/openstack/latest/meta_data.json': HTTP Error 404: Not Found
2014-09-08 14:18:11,921 - DEBUG - scalarizr.util.metadata.ec2 - matched user_data
2014-09-08 14:18:21,914 - DEBUG - scalarizr.util.metadata.cloudstack - Try 'http://169.254.1.0/latest/instance-id': <urlopen error timed out>
2014-09-08 14:18:21,914 - DEBUG - scalarizr.util.metadata - provider for 'instance_id': <scalarizr.util.metadata.Ec2Pvd object at 0x8406850>
2014-09-08 14:18:21,915 - DEBUG - scalarizr.util.metadata - provider for 'user_data': <scalarizr.util.metadata.Ec2Pvd object at 0x8406850>
2014-09-08 14:18:21,918 - DEBUG - scalarizr.updclient.api - Checking /etc/scalr/private.d/update.status
2014-09-08 14:18:21,919 - DEBUG - scalarizr.updclient.api - Serial number in lock file matches machine one
2014-09-08 14:18:21,919 - DEBUG - scalarizr.updclient.api - Apply /etc/scalr/private.d/update.status settings
2014-09-08 14:18:21,919 - DEBUG - scalarizr.updclient.api - Initializing database connection
2014-09-08 14:18:21,920 - DEBUG - scalarizr.config - Bootstrap INI configuration (reload=False)
2014-09-08 14:18:21,920 - DEBUG - scalarizr.config - Loading main configuration
2014-09-08 14:18:21,921 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/config.ini
2014-09-08 14:18:21,921 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/private.d/config.ini
2014-09-08 14:18:21,922 - DEBUG - scalarizr.config - Loading platform configuration
2014-09-08 14:18:21,922 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/ec2.ini
2014-09-08 14:18:21,922 - DEBUG - scalarizr.config - Loading behaviours configuration
2014-09-08 14:18:21,922 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/app.ini
2014-09-08 14:18:21,923 - DEBUG - scalarizr.config - Loading handlers configuration
2014-09-08 14:18:21,923 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/hooks.ini
2014-09-08 14:18:21,923 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/script_executor.ini
2014-09-08 14:18:21,923 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/ip_list_builder.ini
2014-09-08 14:18:21,924 - DEBUG - scalarizr.updclient.api - Initializing QueryEnv
2014-09-08 14:18:21,924 - DEBUG - scalarizr.queryenv - Call QueryEnv: https://my.scalr.com/query-env/2012-04-17/get-latest-version
2014-09-08 14:18:21,925 - DEBUG - scalarizr.queryenv - QueryEnv request: operation=get-latest-version&version=2012-04-17
2014-09-08 14:18:22,225 - INFO - upd.client - Stopping updclient
2014-09-08 14:18:22,226 - DEBUG - upd.client - Kill child processes
2014-09-08 14:18:22,274 - INFO - upd.client - Stopped
2014-09-08 14:18:22,274 - ERROR - upd.client - Detailed exception information below:
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/scalarizr/updclient/app.py", line 172, in do_start
    self.api.bootstrap()
  File "/usr/lib/python2.6/site-packages/scalarizr/updclient/api.py", line 417, in bootstrap
    self._init_services()
  File "/usr/lib/python2.6/site-packages/scalarizr/updclient/api.py", line 244, in _init_services
    wait_until(init_queryenv, timeout=120, sleep=10)
  File "/usr/lib/python2.6/site-packages/scalarizr/util/__init__.py", line 301, in wait_until
    while not target(*args, **kwargs):
  File "/usr/lib/python2.6/site-packages/scalarizr/updclient/api.py", line 236, in init_queryenv
    self._init_queryenv()
  File "/usr/lib/python2.6/site-packages/scalarizr/updclient/api.py", line 195, in _init_queryenv
    api_version=self.queryenv.get_latest_version())
  File "/usr/lib/python2.6/site-packages/scalarizr/queryenv.py", line 241, in get_latest_version
    return self._request("get-latest-version",{}, self._read_get_latest_version_response)
  File "/usr/lib/python2.6/site-packages/scalarizr/queryenv.py", line 282, in _request
    xml = self.fetch(command, params, log_response=False)
  File "/usr/lib/python2.6/site-packages/scalarizr/queryenv.py", line 93, in fetch
    raise InvalidSignatureError(msg)
InvalidSignatureError: Signature doesn't match

I also see the following from the scalr daemon:

# service scalarizr status
scalarizr dead but pid file exists
# service scalarizr stop
# service scalarizr start
Starting scalarizr: [ OK ]
# service scalarizr status
scalarizr (pid 1804) is running...

# ps awx | grep sca
 1804 ? Sl 0:00 /usr/bin/python2.6 /usr/bin/scalarizr -z -c /etc/scalr
 1902 pts/0 R+ 0:00 grep sca

However the scalr admin still cannot communicate with the server and then in a few minutes I get:

[root@ip-10-98-90-227 ~]# service scalarizr status
scalarizr dead but pid file exists

I did try to remove the pid file and restart but that did not help. I also tried a hard reboot of the server and that did not help either.

  1. 1 Posted by Nick Demyanchuk on 09 Sep, 2014 08:58 AM

    Nick Demyanchuk's Avatar

    We've fixed broken server for you, and our engineers are working on fix to prevent such errors in future.

  2. Support Staff 2 Posted by Igor Savchenko on 22 Sep, 2014 11:36 PM

    Igor Savchenko's Avatar

    Issue was fixed. Closing this thread.

    Regards,
    Igor

  3. 3 Posted by Tamas Kalman on 15 Jul, 2016 06:35 AM

    Tamas Kalman's Avatar

    Hello,

    can't install/update/start Scalarizer on Ubuntu 12.04

    2016-07-15 06:31:39,855+00:00 - DEBUG - scalarizr.app - Reloading configuration after user-data applying
    2016-07-15 06:31:39,855+00:00 - DEBUG - scalarizr.config - Bootstrap INI configuration (reload=True)
    2016-07-15 06:31:39,856+00:00 - DEBUG - scalarizr.config - Loading main configuration
    2016-07-15 06:31:39,856+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/config.ini
    2016-07-15 06:31:39,857+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/private.d/config.ini
    2016-07-15 06:31:39,858+00:00 - DEBUG - scalarizr.config - Loading platform configuration
    2016-07-15 06:31:39,858+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/ec2.ini
    2016-07-15 06:31:39,859+00:00 - DEBUG - scalarizr.config - Loading behaviours configuration
    2016-07-15 06:31:39,859+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/app.ini
    2016-07-15 06:31:39,860+00:00 - DEBUG - scalarizr.config - Loading handlers configuration
    2016-07-15 06:31:39,860+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/ip_list_builder.ini
    2016-07-15 06:31:39,861+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/script_executor.ini
    2016-07-15 06:31:39,862+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/hooks.ini
    2016-07-15 06:31:39,862+00:00 - DEBUG - scalarizr.config - Bootstrap INI configuration (reload=True)
    2016-07-15 06:31:39,863+00:00 - DEBUG - scalarizr.config - Loading main configuration
    2016-07-15 06:31:39,863+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/config.ini
    2016-07-15 06:31:39,864+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/private.d/config.ini
    2016-07-15 06:31:39,865+00:00 - DEBUG - scalarizr.config - Loading platform configuration
    2016-07-15 06:31:39,865+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/ec2.ini
    2016-07-15 06:31:39,866+00:00 - DEBUG - scalarizr.config - Loading behaviours configuration
    2016-07-15 06:31:39,866+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/app.ini
    2016-07-15 06:31:39,867+00:00 - DEBUG - scalarizr.config - Loading handlers configuration
    2016-07-15 06:31:39,867+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/ip_list_builder.ini
    2016-07-15 06:31:39,868+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/script_executor.ini
    2016-07-15 06:31:39,869+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/hooks.ini
    2016-07-15 06:31:39,870+00:00 - DEBUG - scalarizr.app - Initialize platform
    2016-07-15 06:31:40,110+00:00 - DEBUG - scalarizr.util - Wait 0.10 seconds before the next attempt
    2016-07-15 06:31:40,111+00:00 - DEBUG - scalarizr.app - Open SQLite database (file: /etc/scalr/private.d/db.sqlite)
    2016-07-15 06:31:40,266+00:00 - DEBUG - scalarizr.config - Bootstrap INI configuration (reload=False)
    2016-07-15 06:31:40,267+00:00 - DEBUG - scalarizr.config - Bootstrap INI configuration (reload=False)
    2016-07-15 06:31:40,269+00:00 - DEBUG - scalarizr.config - Importing module scalarizr.handlers.apache
    2016-07-15 06:31:40,303+00:00 - DEBUG - scalarizr.config - Lookup config container in module scalarizr.handlers.apache
    2016-07-15 06:31:40,303+00:00 - DEBUG - scalarizr.config - Importing module scalarizr.handlers.lifecycle
    2016-07-15 06:31:41,041+00:00 - DEBUG - scalarizr.config - Lookup config container in module scalarizr.handlers.lifecycle
    2016-07-15 06:31:41,041+00:00 - DEBUG - scalarizr.config - Importing module scalarizr.handlers.ip_list_builder
    2016-07-15 06:31:41,044+00:00 - DEBUG - scalarizr.config - Lookup config container in module scalarizr.handlers.ip_list_builder
    2016-07-15 06:31:41,045+00:00 - DEBUG - scalarizr.config - Importing module scalarizr.handlers.script_executor
    2016-07-15 06:31:41,045+00:00 - DEBUG - scalarizr.config - Lookup config container in module scalarizr.handlers.script_executor
    2016-07-15 06:31:41,046+00:00 - DEBUG - scalarizr.config - Importing module scalarizr.handlers.hooks
    2016-07-15 06:31:41,047+00:00 - DEBUG - scalarizr.config - Lookup config container in module scalarizr.handlers.hooks
    2016-07-15 06:31:41,048+00:00 - DEBUG - scalarizr.config - Importing module scalarizr.handlers.ssh_auth_keys
    2016-07-15 06:31:41,051+00:00 - DEBUG - scalarizr.config - Lookup config container in module scalarizr.handlers.ssh_auth_keys
    2016-07-15 06:31:41,051+00:00 - DEBUG - scalarizr.config - Importing module scalarizr.handlers.deploy
    2016-07-15 06:31:41,052+00:00 - DEBUG - scalarizr.config - Lookup config container in module scalarizr.handlers.deploy
    2016-07-15 06:31:41,052+00:00 - DEBUG - scalarizr.config - Importing module scalarizr.handlers.ec2.lifecycle
    2016-07-15 06:31:41,055+00:00 - DEBUG - scalarizr.config - Lookup config container in module scalarizr.handlers.ec2.lifecycle
    2016-07-15 06:31:41,055+00:00 - DEBUG - scalarizr.config - Importing module scalarizr.handlers.ec2.rebundle
    2016-07-15 06:31:41,149+00:00 - DEBUG - scalarizr.config - Lookup config container in module scalarizr.handlers.ec2.rebundle
    2016-07-15 06:31:41,150+00:00 - DEBUG - scalarizr.config - Importing module scalarizr.handlers.ec2.ebs
    2016-07-15 06:31:41,154+00:00 - DEBUG - scalarizr.config - Lookup config container in module scalarizr.handlers.ec2.ebs
    2016-07-15 06:31:41,155+00:00 - DEBUG - scalarizr.node - Reloading /etc/scalr/private.d/config.ini
    2016-07-15 06:31:41,156+00:00 - DEBUG - scalarizr.app - Got scalr url: 'https://my.scalr.net'
    2016-07-15 06:31:41,157+00:00 - DEBUG - scalarizr.app - Initialize QueryEnv client
    2016-07-15 06:31:41,157+00:00 - DEBUG - scalarizr.node - Reloading /etc/scalr/public.d/config.ini
    2016-07-15 06:31:41,161+00:00 - DEBUG - scalarizr.queryenv - Call QueryEnv: https://my.scalr.net/query-env/2012-04-17/get-latest-version
    2016-07-15 06:31:41,161+00:00 - DEBUG - scalarizr.queryenv - QueryEnv request: {'operation': 'get-latest-version', 'version': '2012-04-17'}
    2016-07-15 06:31:41,360+00:00 - ERROR - scalarizr.app - 403 Client Error: Forbidden for url: https://my.scalr.net/query-env/2012-04-17/get-latest-version?operation=get-latest-version&version=2012-04-17
    Traceback (most recent call last):
      File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.9.0-py2.7.egg/scalarizr/app.py", line 562, in main
        svs.start()
      File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.9.0-py2.7.egg/scalarizr/app.py", line 722, in start
        self._init_services()
      File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.9.0-py2.7.egg/scalarizr/app.py", line 845, in _init_services
        queryenv = new_queryenv()
      File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.9.0-py2.7.egg/scalarizr/queryenv.py", line 690, in new_queryenv
        api_version = queryenv_svc.get_latest_version()
      File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.9.0-py2.7.egg/scalarizr/queryenv.py", line 278, in get_latest_version
        return self._request("get-latest-version", {}, self._read_get_latest_version_response)
      File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.9.0-py2.7.egg/scalarizr/queryenv.py", line 321, in _request
        xml = self.fetch(command, params, log_response=False)
      File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.9.0-py2.7.egg/scalarizr/queryenv.py", line 146, in fetch
        self._process_request_exception(sys.exc_info()[1])
      File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.9.0-py2.7.egg/scalarizr/queryenv.py", line 124, in _process_request_exception
        raise InvalidSignatureError(str(exc))
    InvalidSignatureError: 403 Client Error: Forbidden for url: https://my.scalr.net/query-env/2012-04-17/get-latest-version?operation=get-latest-version&version=2012-04-17

    Please advise.

  4. Support Staff 4 Posted by Igor Savchenko on 15 Jul, 2016 04:07 PM

    Igor Savchenko's Avatar

    Hi Tamas,

    What is your Hosted Scalr account email address?

    Thanks,
    Igor

  5. marc closed this discussion on 22 Jul, 2016 09:11 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac