问题描述
我正在使用RHOSP 16.1在RHEL 8.2环境中部署Openstack。 对于这些过程，我使用了RHOSP 16.1的RedHat官方文档： https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/pdf/director_installation_and_usage/Red_Hat_OpenStack_Platform-16.1-Director_Installation_and_Usage-en-US.pdf
我已经成功安装了Undercloud，但是在对Overcloud节点进行自省时遇到了错误。 内省（使用--provide选项）后，节点应该处于“可用”状态，但是在执行过程中，节点仍处于“可用”状态。 内省重试限制已达到。错误是“自省超时”，如Mistral日志中所示。 您是否知道导致错误的原因？
我尝试了以下对策无济于事：
- 已禁用selinux
- （通过virsh）重新创建Overcloud节点
- 确认可以从Undercloud（通过ipmitool）访问和控制Overcloud节点
我的猜测是，该问题最有可能是由于服务之间的许可（nova 中子，nova ironic等）引起的。 但是，这是否真的很奇怪，因为在undercloud安装期间没有手动配置服务。 还是我错过了什么？喜欢软件包或必备软件吗？
下面分别是在undercloud安装和节点导入期间使用的undercloud.conf和overcloud-stackenv.json文件：
(undercloud) [stack@undercloud ~]$ cat undercloud.conf
[DEFAULT]
container_images_file = /home/stack/containers-prepare-parameter.yaml
deployment_user = stack
enabled_hardware_types = ipmi,redfish,ilo,idrac,manual-management
generate_service_certificate = false
inspection_interface = br-ctlplane
ipxe_enabled = true
local_interface = ens4
local_ip = 10.10.10.10/24
local_subnet = ctlplane-subnet
overcloud_domain_name = overcloud.domain
subnets = ctlplane-subnet
undercloud_hostname = undercloud.home
undercloud_log_file = install-undercloud.log
[ctlplane-subnet]
cidr = 10.10.10.0/24
dhcp_end = 10.10.10.100
dhcp_start = 10.10.10.20
gateway = 10.10.10.10
inspection_iprange = 10.10.10.101,10.10.10.150
masquerade = true
(undercloud) [stack@undercloud ~]$ cat overcloud-stackenv.json
{
"nodes": [
{
"arch": "x86_64","disk": "80","memory": "12288","name": "new-overcloud-compute","pm_user": "admin","pm_addr": "10.10.10.10","pm_password": "password","pm_port": "6001","pm_type": "pxe_ipmitool","mac": [
"52:54:00:b4:c6:f0"
],"cpu": "2","capabilities": "profile:compute,boot_option:local"
},{
"arch": "x86_64","name": "new-overcloud-controller","pm_port": "6002","mac": [
"52:54:00:b4:c6:f8"
],"capabilities": "profile:control,boot_option:local"
}
]
}
我也尝试将pm_type的值更改为ipmitool，但是遇到的问题仍然相同。
vbmc list的输出：
(undercloud) [stack@undercloud ~]$ vbmc list
+--------------------------+---------+-------------+------+
| Domain name | Status | Address | Port |
+--------------------------+---------+-------------+------+
| new-overcloud-compute | running | 10.10.10.10 | 6001 |
| new-overcloud-controller | running | 10.10.10.10 | 6002 |
+--------------------------+---------+-------------+------+
关于遇到的实际错误，请参见以下详细信息：
(undercloud) [stack@undercloud ~]$ openstack overcloud node import --introspect --provide overcloud-stackenv.json
Waiting for messages on queue 'tripleo' with no timeout.
2 node(s) successfully moved to the "manageable" state.
Successfully registered node UUID 2b4db183-7bbf-488c-8754-54174f0ba47e
Successfully registered node UUID 8aa29d4b-dd52-42a7-9b25-05d0999fbe3e
Waiting for introspection to finish...
Waiting for messages on queue 'tripleo' with no timeout.
Introspection of node attempt Failed:8aa29d4b-dd52-42a7-9b25-05d0999fbe3e.
Introspection of node attempt Failed:2b4db183-7bbf-488c-8754-54174f0ba47e.
retrying 2 nodes that Failed introspection. Attempt 1 of 3
Introspection of node attempt Failed:2b4db183-7bbf-488c-8754-54174f0ba47e.
Introspection of node attempt Failed:8aa29d4b-dd52-42a7-9b25-05d0999fbe3e.
retrying 2 nodes that Failed introspection. Attempt 2 of 3
Introspection of node attempt Failed:2b4db183-7bbf-488c-8754-54174f0ba47e.
Introspection of node attempt Failed:8aa29d4b-dd52-42a7-9b25-05d0999fbe3e.
retrying 2 nodes that Failed introspection. Attempt 3 of 3
Introspection of node attempt Failed:8aa29d4b-dd52-42a7-9b25-05d0999fbe3e.
Introspection of node attempt Failed:2b4db183-7bbf-488c-8754-54174f0ba47e.
Retry limit reached with 2 nodes still failing introspection
Introspection completed with errors: Retry limit reached with 2 nodes still failing introspection
sys:1: ResourceWarning: unclosed <socket.socket fd=4,family=AddressFamily.AF_INET,type=SocketKind.soCK_STREAM,proto=6,laddr=('10.10.10.10',60614)>
sys:1: ResourceWarning: unclosed <socket.socket fd=6,48436)>
我可以通过ipmitool命令控制/访问节点。
(undercloud) [stack@undercloud ~]$ ipmitool -I lanplus -U admin -P password -H 10.10.10.10 -p 6001 power status
Chassis Power is on
(undercloud) [stack@undercloud ~]$ ipmitool -I lanplus -U admin -P password -H 10.10.10.10 -p 6002 power status
Chassis Power is on
请注意，节点仍处于“可管理”状态。
(undercloud) [stack@undercloud ~]$ openstack bareMetal node list
+--------------------------------------+--------------------------+---------------+-------------+--------------------+-------------+
| UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+--------------------------+---------------+-------------+--------------------+-------------+
| 2b4db183-7bbf-488c-8754-54174f0ba47e | new-overcloud-compute | None | power on | manageable | False |
| 8aa29d4b-dd52-42a7-9b25-05d0999fbe3e | new-overcloud-controller | None | power on | manageable | False |
+--------------------------------------+--------------------------+---------------+-------------+--------------------+-------------+
(undercloud) [stack@undercloud ~]$ nova service-list
+--------------------------------------+----------------+-----------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | disabled Reason | Forced down |
+--------------------------------------+----------------+-----------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| a0e54463-0afa-49b0-bc1b-2af14ad7eb63 | nova-conductor | undercloud.overcloud.domain | internal | enabled | up | 2020-09-28T07:39:51.000000 | - | False |
| 88f4bf62-e052-4e3a-8114-f6d492001cf6 | nova-scheduler | undercloud.overcloud.domain | internal | enabled | up | 2020-09-28T07:39:48.000000 | - | False |
| ef9ccd89-f74e-4628-a1fc-4f4aed8bb422 | nova-compute | undercloud.overcloud.domain | nova | enabled | up | 2020-09-28T07:39:50.000000 | - | False |
+--------------------------------------+----------------+-----------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
(undercloud) [stack@undercloud ~]$ nova hypervisor-list
+--------------------------------------+--------------------------------------+-------+---------+
| ID | Hypervisor hostname | State | Status |
+--------------------------------------+--------------------------------------+-------+---------+
| 8aa29d4b-dd52-42a7-9b25-05d0999fbe3e | 8aa29d4b-dd52-42a7-9b25-05d0999fbe3e | up | enabled |
| 2b4db183-7bbf-488c-8754-54174f0ba47e | 2b4db183-7bbf-488c-8754-54174f0ba47e | up | enabled |
+--------------------------------------+--------------------------------------+-------+---------+
引擎日志
[root@undercloud containers]# grep -iE "(fail|error|warn|denied)" mistral/engine.log
...
2020-09-28 13:28:19.014 6 DEBUG mistral.expressions.yaql_expression [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Starting to evaluate YAQL expression. [expression='<% $.get('status') = "Failed" %>'] evaluate /usr/lib/python3.6/site-packages/mistral/expressions/yaql_expression.py:175
2020-09-28 13:28:19.019 6 DEBUG mistral.expressions.yaql_expression [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Starting to evaluate YAQL expression. [expression='<% "Workflow Failed due to message status. Status:{} Message:{}".format($.get('status'),$.get('message')) %>'] evaluate /usr/lib/python3.6/site-packages/mistral/expressions/yaql_expression.py:175
2020-09-28 13:28:19.026 6 DEBUG mistral.workflow.direct_workflow [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Found commands: [Fail [workflow=tripleo.messaging.v1.send]] _find_next_commands_for_task /usr/lib/python3.6/site-packages/mistral/workflow/direct_workflow.py:157
2020-09-28 13:28:19.058 6 INFO workflow_trace [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Workflow 'tripleo.messaging.v1.send' [RUNNING -> ERROR,msg=Workflow Failed due to message status. Status:Failed Message:Introspection of node attempt Failed:2b4db183-7bbf-488c-8754-54174f0ba47e.] (execution_id=ff5cf8ee-52f6-4b3c-b4a3-0a9ec220c52a)
2020-09-28 13:28:19.111 6 INFO mistral.engine.engine_server [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Received RPC request 'on_action_complete'[action_ex_id=ff5cf8ee-52f6-4b3c-b4a3-0a9ec220c52a,result=Result [data=None,error=Workflow Failed due to message status. Status:Failed Message:Introspection of node attempt Failed:2b...,cancel=False]]
2020-09-28 13:28:19.146 6 INFO workflow_trace [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Task 'send_message' (c4b3466b-34b1-4889-a84a-f38a8ec913db) [RUNNING -> ERROR,msg=Workflow Failed due to message status. Status:Failed Message:Introspection of node attempt Failed:2b4db183-7bbf-488c-8754-54174f0ba47e.] (execution_id=5e0c77f6-9f54-4fa7-bbfe-908ceab17e7e)
2020-09-28 13:28:19.284 6 WARNING oslo_db.sqlalchemy.utils [-] Unique keys not in sort_keys. The sorting order may be unstable.
2020-09-28 13:28:19.330 6 INFO workflow_trace [-] Workflow 'tripleo.bareMetal.v1._introspect' [RUNNING -> ERROR,msg=Failure caused by error in tasks: send_message
send_message [task_ex_id=c4b3466b-34b1-4889-a84a-f38a8ec913db] -> Workflow Failed due to message status. Status:Failed Message:Introspection of node attempt Failed:2b4db183-7bbf-488c-8754-54174f0ba47e.
[wf_ex_id=ff5cf8ee-52f6-4b3c-b4a3-0a9ec220c52a,idx=0]: Workflow Failed due to message status. Status:Failed Message:Introspection of node attempt Failed:2b4db183-7bbf-488c-8754-54174f0ba47e.
2020-09-28 13:28:19.349 6 INFO mistral.engine.engine_server [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Received RPC request 'on_action_complete'[action_ex_id=5e0c77f6-9f54-4fa7-bbfe-908ceab17e7e,error=Failure caused by error in tasks: send_message
2020-09-28 13:28:20.126 6 INFO mistral.engine.engine_server [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Received RPC request 'on_action_complete'[action_ex_id=0d02fd62-adf9-4db8-a942-df47c7236e65,error=None,cancel=False]]
2020-09-28 13:28:20.148 6 INFO workflow_trace [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Action 'std.noop' (0d02fd62-adf9-4db8-a942-df47c7236e65)(task=check_status) [RUNNING -> SUCCESS,Result [data=None,cancel=False]]
2020-09-28 13:28:20.192 6 DEBUG mistral.expressions.yaql_expression [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Starting to evaluate YAQL expression. [expression='<% $.get('status') = "Failed" %>'] evaluate /usr/lib/python3.6/site-packages/mistral/expressions/yaql_expression.py:175
2020-09-28 13:28:20.196 6 DEBUG mistral.expressions.yaql_expression [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Starting to evaluate YAQL expression. [expression='<% "Workflow Failed due to message status. Status:{} Message:{}".format($.get('status'),$.get('message')) %>'] evaluate /usr/lib/python3.6/site-packages/mistral/expressions/yaql_expression.py:175
2020-09-28 13:28:20.203 6 DEBUG mistral.workflow.direct_workflow [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Found commands: [Fail [workflow=tripleo.messaging.v1.send]] _find_next_commands_for_task /usr/lib/python3.6/site-packages/mistral/workflow/direct_workflow.py:157
2020-09-28 13:28:20.243 6 INFO workflow_trace [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Workflow 'tripleo.messaging.v1.send' [RUNNING -> ERROR,msg=Workflow Failed due to message status. Status:Failed Message:Introspection of node attempt Failed:8aa29d4b-dd52-42a7-9b25-05d0999fbe3e.] (execution_id=83375607-678b-4654-acc9-a8c36edace4b)
2020-09-28 13:28:20.275 6 INFO mistral.engine.engine_server [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Received RPC request 'on_action_complete'[action_ex_id=83375607-678b-4654-acc9-a8c36edace4b,error=Workflow Failed due to message status. Status:Failed Message:Introspection of node attempt Failed:8a...,cancel=False]]
2020-09-28 13:28:20.299 6 INFO workflow_trace [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Task 'send_message' (e339574e-2cd8-4c0a-9d26-d5826d90bdbc) [RUNNING -> ERROR,msg=Workflow Failed due to message status. Status:Failed Message:Introspection of node attempt Failed:8aa29d4b-dd52-42a7-9b25-05d0999fbe3e.] (execution_id=39407fb5-354f-4989-88e5-cf627b12f2d6)
2020-09-28 13:28:20.389 6 WARNING oslo_db.sqlalchemy.utils [-] Unique keys not in sort_keys. The sorting order may be unstable.
2020-09-28 13:28:20.434 6 INFO workflow_trace [-] Workflow 'tripleo.bareMetal.v1._introspect' [RUNNING -> ERROR,msg=Failure caused by error in tasks: send_message
send_message [task_ex_id=e339574e-2cd8-4c0a-9d26-d5826d90bdbc] -> Workflow Failed due to message status. Status:Failed Message:Introspection of node attempt Failed:8aa29d4b-dd52-42a7-9b25-05d0999fbe3e.
[wf_ex_id=83375607-678b-4654-acc9-a8c36edace4b,idx=0]: Workflow Failed due to message status. Status:Failed Message:Introspection of node attempt Failed:8aa29d4b-dd52-42a7-9b25-05d0999fbe3e.
2020-09-28 13:28:20.455 6 INFO mistral.engine.engine_server [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Received RPC request 'on_action_complete'[action_ex_id=39407fb5-354f-4989-88e5-cf627b12f2d6,error=Failure caused by error in tasks: send_message
2020-09-28 13:28:21.336 6 INFO workflow_trace [req-02272ea3-36ec-49a7-9042-e7917691b87c 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Task 'introspect_nodes' (79017565-04d9-499a-93f9-cd15aca38765) [RUNNING -> ERROR,msg=One or more actions had Failed.] (execution_id=43188fcc-e4a0-489e-9032-a33fd8ae6902)
2020-09-28 13:28:21.356 6 DEBUG mistral.workflow.direct_workflow [req-02272ea3-36ec-49a7-9042-e7917691b87c 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Found commands: [Run task [workflow=tripleo.bareMetal.v1.introspect,task=max_retry_attempts_reached,waif_flag=False,triggered_by=[{'task_id': '79017565-04d9-499a-93f9-cd15aca38765','event': 'on-error'}]]] _find_next_commands_for_task /usr/lib/python3.6/site-packages/mistral/workflow/direct_workflow.py:157
2020-09-28 13:28:21.418 6 INFO mistral.engine.engine_server [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Received RPC request 'on_action_complete'[action_ex_id=0c23a7fb-af8e-44f5-b0c4-a9f0094d599c,cancel=False]]
2020-09-28 13:28:21.434 6 INFO workflow_trace [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Action 'std.noop' (0c23a7fb-af8e-44f5-b0c4-a9f0094d599c)(task=max_retry_attempts_reached) [RUNNING -> SUCCESS,cancel=False]]
2020-09-28 13:28:21.460 6 DEBUG mistral.expressions.yaql_expression [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Starting to evaluate YAQL expression. [expression='<% 'Retry limit reached with {0} nodes still failing introspection'.format($.Failed_introspection.len()) %>'] evaluate /usr/lib/python3.6/site-packages/mistral/expressions/yaql_expression.py:175
2020-09-28 13:28:21.517 6 DEBUG mistral.expressions.yaql_expression [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Starting to evaluate YAQL expression. [expression='<% { introspected_nodes => $.get('introspected_nodes',[]),Failed_introspection => $.get('Failed_introspection',[]) } %>'] evaluate /usr/lib/python3.6/site-packages/mistral/expressions/yaql_expression.py:175
2020-09-28 13:28:21.546 6 INFO workflow_trace [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Starting workflow [name=tripleo.messaging.v1.send,input={'queue_name': 'tripleo','type': 'tripleo.bareMetal.v1.introspect','status': 'Failed','message...]
...
2020-09-28 13:28:23.874 6 DEBUG mistral.expressions.yaql_expression [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Starting to evaluate YAQL expression. [expression='<% $.get('status') = "Failed" %>'] evaluate /usr/lib/python3.6/site-packages/mistral/expressions/yaql_expression.py:175
2020-09-28 13:28:23.881 6 DEBUG mistral.expressions.yaql_expression [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Starting to evaluate YAQL expression. [expression='<% "Workflow Failed due to message status. Status:{} Message:{}".format($.get('status'),$.get('message')) %>'] evaluate /usr/lib/python3.6/site-packages/mistral/expressions/yaql_expression.py:175
2020-09-28 13:28:23.887 6 DEBUG mistral.workflow.direct_workflow [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Found commands: [Fail [workflow=tripleo.messaging.v1.send]] _find_next_commands_for_task /usr/lib/python3.6/site-packages/mistral/workflow/direct_workflow.py:157
2020-09-28 13:28:23.918 6 INFO workflow_trace [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Workflow 'tripleo.messaging.v1.send' [RUNNING -> ERROR,msg=Workflow Failed due to message status. Status:Failed Message:Retry limit reached with 2 nodes still failing introspection] (execution_id=89c35cc0-c179-490e-ad7c-b50bc2a1c36c)
2020-09-28 13:28:23.945 6 INFO mistral.engine.engine_server [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Received RPC request 'on_action_complete'[action_ex_id=89c35cc0-c179-490e-ad7c-b50bc2a1c36c,error=Workflow Failed due to message status. Status:Failed Message:Retry limit reached with 2 nodes still ...,cancel=False]]
2020-09-28 13:28:23.966 6 INFO workflow_trace [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Task 'send_message' (c7218062-acc2-4baa-be01-ad7ebf5fef27) [RUNNING -> ERROR,msg=Workflow Failed due to message status. Status:Failed Message:Retry limit reached with 2 nodes still failing introspection] (execution_id=43188fcc-e4a0-489e-9032-a33fd8ae6902)
2020-09-28 13:28:24.052 6 WARNING oslo_db.sqlalchemy.utils [-] Unique keys not in sort_keys. The sorting order may be unstable.
2020-09-28 13:28:24.099 6 INFO workflow_trace [-] Workflow 'tripleo.bareMetal.v1.introspect' [RUNNING -> ERROR,msg=Failure caused by error in tasks: send_message
send_message [task_ex_id=c7218062-acc2-4baa-be01-ad7ebf5fef27] -> Workflow Failed due to message status. Status:Failed Message:Retry limit reached with 2 nodes still failing introspection
[wf_ex_id=89c35cc0-c179-490e-ad7c-b50bc2a1c36c,idx=0]: Workflow Failed due to message status. Status:Failed Message:Retry limit reached with 2 nodes still failing introspection
主要执行日志
[root@undercloud containers]# grep -iE "(fail|error|warn|denied)" mistral/executor.log
2020-09-28 12:06:52.602 6 WARNING tripleo_common.utils.nodes [req-331aca0d-d9b8-4a7c-8836-3fcdc9eea72b 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Replacing deprecated driver pxe_ipmitool with the hardware type ipmi,please update your inventory
2020-09-28 12:06:54.870 6 WARNING tripleo_common.utils.nodes [req-331aca0d-d9b8-4a7c-8836-3fcdc9eea72b 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Replacing deprecated driver pxe_ipmitool with the hardware type ipmi,please update your inventory
...
2020-09-28 12:06:58.866 6 DEBUG mistral.executors.executor_server [req-331aca0d-d9b8-4a7c-8836-3fcdc9eea72b 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Received RPC request 'run_action'[action_ex_id=299c5f9c-ecd0-4fd1-9879-bbd4ec501b7a,action_cls_str=mistral.actions.openstack.actions.IronicAction,action_cls_attrs={'client_method_name': 'node.get'},params={'node_id': '2b4db183-7bbf-488c-8754-54174f0ba47e','fields': ['provision_state','last_error']},timeout=1200] run_action /usr/lib/python3.6/site-packages/mistral/executors/executor_server.py:94
2020-09-28 12:07:02.854 6 DEBUG mistral.executors.executor_server [req-fda60dfa-6dfb-41b3-b7c8-ff0baa8ff9bc 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Received RPC request 'run_action'[action_ex_id=515881f8-c2e1-4b35-a53e-36ba4bd4c431,timeout=1200] run_action /usr/lib/python3.6/site-packages/mistral/executors/executor_server.py:94
2020-09-28 12:07:03.063 6 DEBUG mistral.executors.executor_server [req-fda60dfa-6dfb-41b3-b7c8-ff0baa8ff9bc 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Received RPC request 'run_action'[action_ex_id=51e88369-a9d1-4469-b714-b8580c5be64d,params={'node_id': '8aa29d4b-dd52-42a7-9b25-05d0999fbe3e',timeout=1200] run_action /usr/lib/python3.6/site-packages/mistral/executors/executor_server.py:94
2020-09-28 12:27:18.382 6 WARNING mistral.actions.openstack.base [req-fda60dfa-6dfb-41b3-b7c8-ff0baa8ff9bc 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Traceback (most recent call last):
...
2020-09-28 13:28:15.618 6 WARNING mistral.actions.openstack.base [req-02272ea3-36ec-49a7-9042-e7917691b87c 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Traceback (most recent call last):
ironic_inspector_client.v1.WaitTimeoutError: Timeout while waiting for introspection of nodes ['2b4db183-7bbf-488c-8754-54174f0ba47e']
: ironic_inspector_client.v1.WaitTimeoutError: Timeout while waiting for introspection of nodes ['2b4db183-7bbf-488c-8754-54174f0ba47e']
2020-09-28 13:28:15.619 6 WARNING mistral.executors.default_executor [req-02272ea3-36ec-49a7-9042-e7917691b87c 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] The action raised an exception [action_ex_id=7aefe54e-46c7-41f1-abfe-cd8dca9336c9,msg='BareMetalintrospectionAction.wait_for_finish Failed: Timeout while waiting for introspection of nodes ['2b4db183-7bbf-488c-8754-54174f0ba47e']',action_cls='<class 'mistral.actions.action_factory.BareMetalintrospectionAction'>',attributes='{'client_method_name': 'wait_for_finish'}',params='{'uuids': ['2b4db183-7bbf-488c-8754-54174f0ba47e'],'max_retries': 120,'retry_interval': 10}']: mistral.exceptions.ActionException: BareMetalintrospectionAction.wait_for_finish Failed: Timeout while waiting for introspection of nodes ['2b4db183-7bbf-488c-8754-54174f0ba47e']
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor Traceback (most recent call last):
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor File "/usr/lib/python3.6/site-packages/mistral/actions/openstack/base.py",line 117,in run
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor result = method(**self._kwargs_for_run)
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor File "/usr/lib/python3.6/site-packages/ironic_inspector_client/v1.py",line 265,in wait_for_finish
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor "of nodes %s") % new_active_node_ids)
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor ironic_inspector_client.v1.WaitTimeoutError: Timeout while waiting for introspection of nodes ['2b4db183-7bbf-488c-8754-54174f0ba47e']
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor During handling of the above exception,another exception occurred:
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor Traceback (most recent call last):
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor File "/usr/lib/python3.6/site-packages/mistral/executors/default_executor.py",line 133,in _do_run_action
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor result = action.run(action_ctx)
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor File "/usr/lib/python3.6/site-packages/mistral/actions/openstack/base.py",line 130,in run
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor (self.__class__.__name__,self.client_method_name,str(e))
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor mistral.exceptions.ActionException: BareMetalintrospectionAction.wait_for_finish Failed: Timeout while waiting for introspection of nodes ['2b4db183-7bbf-488c-8754-54174f0ba47e']
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor
NOVA计算机日志
[root@undercloud containers]# grep -iE "(fail|error|warn|denied)" nova/nova-compute.log
2020-09-28 12:07:15.103 6 ERROR nova.compute.manager [req-76b53cba-540c-4069-bb12-85b5f2b71f14 - - - - -] No compute node record for host undercloud.overcloud.domain: nova.exception_Remote.ComputeHostNotFound_Remote: Compute host undercloud.overcloud.domain Could not be found.
2020-09-28 12:07:15.143 6 WARNING nova.compute.resource_tracker [req-76b53cba-540c-4069-bb12-85b5f2b71f14 - - - - -] No compute node record for undercloud.overcloud.domain:8aa29d4b-dd52-42a7-9b25-05d0999fbe3e: nova.exception_Remote.ComputeHostNotFound_Remote: Compute host undercloud.overcloud.domain Could not be found.
2020-09-28 12:07:16.805 6 WARNING nova.compute.resource_tracker [req-76b53cba-540c-4069-bb12-85b5f2b71f14 - - - - -] No compute node record for undercloud.overcloud.domain:2b4db183-7bbf-488c-8754-54174f0ba47e: nova.exception_Remote.ComputeHostNotFound_Remote: Compute host undercloud.overcloud.domain Could not be found.
任何帮助/输入将不胜感激。非常感谢！
这个问题也在https://access.redhat.com/discussions/5444481。
解决方法
暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！
如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。
小编邮箱:dio#foxmail.com (将#修改为@）