openstack

openstack overcloud deploy: plan-environment.yaml not found

For one reason or another, I sometimes run into the following error when trying to deploy an OpenStack TripleO overcloud in my lab:

ClientException: Object GET failed: https://192.0.2.2:13808/v1
  /AUTH_1a5df8c242d848e8b01b974a72afcf8c
  /overcloud/plan-environment.yaml 404 Not Found

This usually occurs after impatiently cancelling an overcloud deployment – strongly not recommended of course. The overcloud plan, which is stored in a Swift container, is left missing one or more objects.

Given that Mistral (OpenStack’s workflow service) is meant to first delete the existing plan before continuing, it’s an odd error. The solution, it turns out, is not quite as straight forward as manually deleting the plan.

Deleting an overcloud plan is a restricted operation when that plan is tied to an existing stack, and deleting the stack would mean deleting my overcloud. This was not a fresh deployment, but rather a stack update, so funnily enough this was not my preferred approach. Given the overcloud plan is just stored in a Swift object container though, we can modify it directly:

## Source undercloud environment variables
$ . stackrc
## Recursively delete all objects and the container
$ openstack container delete --recursive overcloud

From here, running the overcloud deployment again creates the Swift object container and populates it with the deployment plan, however this is still prone to errors. My own deployment kept failing with Mistral validation errors, despite being a “valid” plan previously.

The solution is to run the undercloud installation again. This is essentially a bunch of idempotent workflows, so should be completely safe to run and re-run, though take care that your undercloud config file has not changed unless you expect it to.

$ openstack undercloud install

Once the install has completed you should notice that the Swift overcloud container has been recreated and populated with all the required plan objects. You can now kick off that overcloud deploy.

3 thoughts on “openstack overcloud deploy: plan-environment.yaml not found

  1. Hi Matt,

    We are also facing same issue while running the redeployment on our OpenStack cloud. The re-deployment was failed due to network connection fluctuation (our bad … it was not executed in a “screen”).

    What are your views if we backup & delete the swift overcloud container with “swift delete overcloud” command and then re-run the deployment, will it have any impact on running cloud platform.

    (This is a cloud platform with 300+ instances running)

    Liked by 1 person

    1. Hi Amit

      In my experience there were no issues with deleting the overcloud plan and then kicking off the overcloud deploy/update again, although as I mentioned I did need to run the undercloud install again first in order to populate a new default overcloud plan.

      Just be sure to take the standard precautions … database backups, test in another environment first, etc

      Cheers,
      Matt

      Like

  2. Following is the error we are getting …. if it helps:

    http://10.151.5.50:8080 “GET /v1/AUTH_9ee3af68b5ec4bbd9fbfcfea344c67f2/overcloud/plan-environment.yaml HTTP/1.1” 200 233
    ‘passwords’
    Traceback (most recent call last):
    File “/usr/lib/python2.7/site-packages/cliff/app.py”, line 400, in run_subcommand
    result = cmd.run(parsed_args)
    File “/usr/lib/python2.7/site-packages/tripleoclient/command.py”, line 25, in run
    super(Command, self).run(parsed_args)
    File “/usr/lib/python2.7/site-packages/osc_lib/command/command.py”, line 41, in run
    return super(Command, self).run(parsed_args)
    File “/usr/lib/python2.7/site-packages/cliff/command.py”, line 184, in run
    return_code = self.take_action(parsed_args) or 0
    File “/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py”, line 973, in take_action
    self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
    File “/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py”, line 428, in _deploy_tripleo_heat_templates_tmpdir
    new_tht_root, tht_root)
    File “/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py”, line 453, in _deploy_tripleo_heat_templates
    type(self)._keep_env_on_update)
    File “/usr/lib/python2.7/site-packages/tripleoclient/workflows/plan_management.py”, line 170, in update_plan_from_templates
    passwords = _load_passwords(swift_client, name)
    File “/usr/lib/python2.7/site-packages/tripleoclient/workflows/plan_management.py”, line 256, in _load_passwords
    return plan_env[‘passwords’]
    KeyError: ‘passwords’
    clean_up DeployOvercloud: ‘passwords’
    Traceback (most recent call last):
    File “/usr/lib/python2.7/site-packages/osc_lib/shell.py”, line 134, in run
    ret_val = super(OpenStackShell, self).run(argv)
    File “/usr/lib/python2.7/site-packages/cliff/app.py”, line 279, in run
    result = self.run_subcommand(remainder)
    File “/usr/lib/python2.7/site-packages/osc_lib/shell.py”, line 169, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
    File “/usr/lib/python2.7/site-packages/cliff/app.py”, line 400, in run_subcommand
    result = cmd.run(parsed_args)
    File “/usr/lib/python2.7/site-packages/tripleoclient/command.py”, line 25, in run
    super(Command, self).run(parsed_args)
    File “/usr/lib/python2.7/site-packages/osc_lib/command/command.py”, line 41, in run
    return super(Command, self).run(parsed_args)
    File “/usr/lib/python2.7/site-packages/cliff/command.py”, line 184, in run
    return_code = self.take_action(parsed_args) or 0
    File “/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py”, line 973, in take_action
    self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
    File “/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py”, line 428, in _deploy_tripleo_heat_templates_tmpdir
    new_tht_root, tht_root)
    File “/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py”, line 453, in _deploy_tripleo_heat_templates
    type(self)._keep_env_on_update)
    File “/usr/lib/python2.7/site-packages/tripleoclient/workflows/plan_management.py”, line 170, in update_plan_from_templates
    passwords = _load_passwords(swift_client, name)
    File “/usr/lib/python2.7/site-packages/tripleoclient/workflows/plan_management.py”, line 256, in _load_passwords
    return plan_env[‘passwords’]
    KeyError: ‘passwords’

    END return value: 1

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.