For one reason or another, I sometimes run into the following error when trying to deploy an OpenStack TripleO overcloud in my lab:
ClientException: Object GET failed: https://192.0.2.2:13808/v1 /AUTH_1a5df8c242d848e8b01b974a72afcf8c /overcloud/plan-environment.yaml 404 Not Found
This usually occurs after impatiently cancelling an overcloud deployment – strongly not recommended of course. The overcloud plan, which is stored in a Swift container, is left missing one or more objects.
Given that Mistral (OpenStack’s workflow service) is meant to first delete the existing plan before continuing, it’s an odd error. The solution, it turns out, is not quite as straight forward as manually deleting the plan.
Deleting an overcloud plan is a restricted operation when that plan is tied to an existing stack, and deleting the stack would mean deleting my overcloud. This was not a fresh deployment, but rather a stack update, so funnily enough this was not my preferred approach. Given the overcloud plan is just stored in a Swift object container though, we can modify it directly:
## Source undercloud environment variables $ . stackrc ## Recursively delete all objects and the container $ openstack container delete --recursive overcloud
From here, running the overcloud deployment again creates the Swift object container and populates it with the deployment plan, however this is still prone to errors. My own deployment kept failing with Mistral validation errors, despite being a “valid” plan previously.
The solution is to run the undercloud installation again. This is essentially a bunch of idempotent workflows, so should be completely safe to run and re-run, though take care that your undercloud config file has not changed unless you expect it to.
$ openstack undercloud install
Once the install has completed you should notice that the Swift overcloud container has been recreated and populated with all the required plan objects. You can now kick off that overcloud deploy.
Hi Matt,
We are also facing same issue while running the redeployment on our OpenStack cloud. The re-deployment was failed due to network connection fluctuation (our bad … it was not executed in a “screen”).
What are your views if we backup & delete the swift overcloud container with “swift delete overcloud” command and then re-run the deployment, will it have any impact on running cloud platform.
(This is a cloud platform with 300+ instances running)
LikeLiked by 1 person
Hi Amit
In my experience there were no issues with deleting the overcloud plan and then kicking off the overcloud deploy/update again, although as I mentioned I did need to run the undercloud install again first in order to populate a new default overcloud plan.
Just be sure to take the standard precautions … database backups, test in another environment first, etc
Cheers,
Matt
LikeLike
Following is the error we are getting …. if it helps:
http://10.151.5.50:8080 “GET /v1/AUTH_9ee3af68b5ec4bbd9fbfcfea344c67f2/overcloud/plan-environment.yaml HTTP/1.1” 200 233
‘passwords’
Traceback (most recent call last):
File “/usr/lib/python2.7/site-packages/cliff/app.py”, line 400, in run_subcommand
result = cmd.run(parsed_args)
File “/usr/lib/python2.7/site-packages/tripleoclient/command.py”, line 25, in run
super(Command, self).run(parsed_args)
File “/usr/lib/python2.7/site-packages/osc_lib/command/command.py”, line 41, in run
return super(Command, self).run(parsed_args)
File “/usr/lib/python2.7/site-packages/cliff/command.py”, line 184, in run
return_code = self.take_action(parsed_args) or 0
File “/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py”, line 973, in take_action
self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
File “/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py”, line 428, in _deploy_tripleo_heat_templates_tmpdir
new_tht_root, tht_root)
File “/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py”, line 453, in _deploy_tripleo_heat_templates
type(self)._keep_env_on_update)
File “/usr/lib/python2.7/site-packages/tripleoclient/workflows/plan_management.py”, line 170, in update_plan_from_templates
passwords = _load_passwords(swift_client, name)
File “/usr/lib/python2.7/site-packages/tripleoclient/workflows/plan_management.py”, line 256, in _load_passwords
return plan_env[‘passwords’]
KeyError: ‘passwords’
clean_up DeployOvercloud: ‘passwords’
Traceback (most recent call last):
File “/usr/lib/python2.7/site-packages/osc_lib/shell.py”, line 134, in run
ret_val = super(OpenStackShell, self).run(argv)
File “/usr/lib/python2.7/site-packages/cliff/app.py”, line 279, in run
result = self.run_subcommand(remainder)
File “/usr/lib/python2.7/site-packages/osc_lib/shell.py”, line 169, in run_subcommand
ret_value = super(OpenStackShell, self).run_subcommand(argv)
File “/usr/lib/python2.7/site-packages/cliff/app.py”, line 400, in run_subcommand
result = cmd.run(parsed_args)
File “/usr/lib/python2.7/site-packages/tripleoclient/command.py”, line 25, in run
super(Command, self).run(parsed_args)
File “/usr/lib/python2.7/site-packages/osc_lib/command/command.py”, line 41, in run
return super(Command, self).run(parsed_args)
File “/usr/lib/python2.7/site-packages/cliff/command.py”, line 184, in run
return_code = self.take_action(parsed_args) or 0
File “/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py”, line 973, in take_action
self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
File “/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py”, line 428, in _deploy_tripleo_heat_templates_tmpdir
new_tht_root, tht_root)
File “/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py”, line 453, in _deploy_tripleo_heat_templates
type(self)._keep_env_on_update)
File “/usr/lib/python2.7/site-packages/tripleoclient/workflows/plan_management.py”, line 170, in update_plan_from_templates
passwords = _load_passwords(swift_client, name)
File “/usr/lib/python2.7/site-packages/tripleoclient/workflows/plan_management.py”, line 256, in _load_passwords
return plan_env[‘passwords’]
KeyError: ‘passwords’
END return value: 1
LikeLike