2025-10-11 16:45:41,801 - INFO - Domain Default found not creating 2025-10-11 16:45:41,942 - INFO - Project ctest-TestRoutersBasic-81768906 not found, creating it 2025-10-11 16:45:42,392 - INFO - Created Project:ctest-TestRoutersBasic-81768906, ID : a64fce7a-46dd-4bb7-985d-c5a6d13e2a26 2025-10-11 16:45:43,850 - INFO - ================================================================================ 2025-10-11 16:45:43,850 - INFO - STARTING TEST : test_basic_snat_behavior_without_external_connectivity 2025-10-11 16:45:43,850 - INFO - TEST DESCRIPTION : Create an external network, a router set router-gateway to external network launch a private network and attach it to router validate left vm pinging right vm through Snat 2025-10-11 16:45:43,973 - DEBUG - Nothing to compare xmpp stats {'10.0.0.254': {'10.20.0.254': '0'}} with 2025-10-11 16:45:43,973 - INFO - Initial checks done. Running the testcase now 2025-10-11 16:45:43,973 - INFO - 2025-10-11 16:45:44,662 - INFO - Default SG to be edited for allow all on project: ctest-TestRoutersBasic-81768906 2025-10-11 16:45:44,758 - INFO - Adding rules to the default security group in Project ctest-TestRoutersBasic-81768906 2025-10-11 16:45:45,073 - DEBUG - Reading firewall policy {'fq_name': ['default-domain', 'ctest-TestRoutersBasic-81768906', 'default']} 2025-10-11 16:45:45,371 - DEBUG - Response for create_network : {'network': {'id': '347486ad-b16d-489d-a302-05721c5117cc', 'name': 'ctest-vn-71995652', 'tenant_id': 'a64fce7a46dd4bb7985dc5a6d13e2a26', 'project_id': 'a64fce7a46dd4bb7985dc5a6d13e2a26', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestRoutersBasic-81768906', 'ctest-vn-71995652'], 'port_security_enabled': True, 'description': ''}} 2025-10-11 16:45:45,601 - DEBUG - Response for create_subnet : {'subnet': {'id': 'e26656e6-f3e8-46e5-bdfa-6954fd46a564', 'name': '', 'tenant_id': 'a64fce7a46dd4bb7985dc5a6d13e2a26', 'network_id': '347486ad-b16d-489d-a302-05721c5117cc', 'ip_version': 4, 'cidr': '23.219.150.128/26', 'allocation_pools': [{'start': '23.219.150.130', 'end': '23.219.150.190'}], 'gateway_ip': '23.219.150.129', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '23.219.150.130', 'tags': [], 'project_id': 'a64fce7a46dd4bb7985dc5a6d13e2a26'}} 2025-10-11 16:45:45,620 - INFO - Created VN ctest-vn-71995652 2025-10-11 16:45:45,674 - DEBUG - VN ctest-vn-71995652 UUID is 347486ad-b16d-489d-a302-05721c5117cc 2025-10-11 16:45:45,982 - DEBUG - Response for create_network : {'network': {'id': '7e931aad-1dab-4d5c-9c3a-fd34fdb9a2cb', 'name': 'ctest-ext_vn-89097485', 'tenant_id': 'a64fce7a46dd4bb7985dc5a6d13e2a26', 'project_id': 'a64fce7a46dd4bb7985dc5a6d13e2a26', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': True, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestRoutersBasic-81768906', 'ctest-ext_vn-89097485'], 'port_security_enabled': True, 'description': ''}} 2025-10-11 16:45:46,157 - DEBUG - Response for create_subnet : {'subnet': {'id': '83925a1a-7ac8-472b-80f8-08223b32fe6e', 'name': '', 'tenant_id': 'a64fce7a46dd4bb7985dc5a6d13e2a26', 'network_id': '7e931aad-1dab-4d5c-9c3a-fd34fdb9a2cb', 'ip_version': 4, 'cidr': '3.190.205.192/26', 'allocation_pools': [{'start': '3.190.205.194', 'end': '3.190.205.254'}], 'gateway_ip': '3.190.205.193', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '3.190.205.194', 'tags': [], 'project_id': 'a64fce7a46dd4bb7985dc5a6d13e2a26'}} 2025-10-11 16:45:46,175 - INFO - Created VN ctest-ext_vn-89097485 2025-10-11 16:45:46,226 - DEBUG - VN ctest-ext_vn-89097485 UUID is 7e931aad-1dab-4d5c-9c3a-fd34fdb9a2cb 2025-10-11 16:45:46,452 - DEBUG - Services list from nova: [, , ] 2025-10-11 16:45:47,486 - INFO - VM ([]) created on node: (cn-jenkins-deploy-platform-ansible-os-4287-1), Zone: (nova:cn-jenkins-deploy-platform-ansible-os-4287-1) 2025-10-11 16:45:48,711 - INFO - VM ([]) created on node: (cn-jenkins-deploy-platform-ansible-os-4287-1), Zone: (nova:cn-jenkins-deploy-platform-ansible-os-4287-1) 2025-10-11 16:45:48,776 - INFO - Setting gateway for router b314a417-e9cf-4a34-8bc1-b6b03c6f6e5d to network 7e931aad-1dab-4d5c-9c3a-fd34fdb9a2cb 2025-10-11 16:45:48,890 - INFO - Adding interface with subnet_id e26656e6-f3e8-46e5-bdfa-6954fd46a564, port_id None to router b314a417-e9cf-4a34-8bc1-b6b03c6f6e5d 2025-10-11 16:45:49,255 - INFO - Waiting for VM ctest-TestRoutersBasic-81768906-35933601 to be up.. 2025-10-11 16:45:49,334 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-10-11 16:45:54,398 - DEBUG - VM is in ERROR state now 2025-10-11 16:45:54,398 - ERROR - VM in error state. 2025-10-11 16:45:54,398 - ERROR - VM is not in ACTIVE state 2025-10-11 16:45:54,482 - ERROR - Fatal Nova Exception while getting VM detail 2025-10-11 16:45:54,482 - DEBUG - None 2025-10-11 16:45:54,482 - INFO - Waiting for VM ctest-TestRoutersBasic-81768906-77278478 to be up.. 2025-10-11 16:45:54,534 - DEBUG - VM is in ERROR state now 2025-10-11 16:45:54,534 - ERROR - VM in error state. 2025-10-11 16:45:54,534 - ERROR - VM is not in ACTIVE state 2025-10-11 16:45:54,609 - ERROR - Fatal Nova Exception while getting VM detail 2025-10-11 16:45:54,609 - DEBUG - None 2025-10-11 16:45:54,609 - INFO - VM name : ctest-TestRoutersBasic-81768906-77278478 2025-10-11 16:45:54,664 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:45:59,728 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:46:04,795 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:46:09,856 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:46:14,917 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:46:19,973 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:46:19,973 - ERROR - VM 0aa75746-e818-4414-89f0-5c671b749799 is not launched yet 2025-10-11 16:46:21,974 - INFO - VM name : ctest-TestRoutersBasic-81768906-77278478 2025-10-11 16:46:22,036 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:46:27,110 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:46:32,173 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:46:37,237 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:46:42,298 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:46:47,359 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:46:47,359 - ERROR - VM 0aa75746-e818-4414-89f0-5c671b749799 is not launched yet 2025-10-11 16:46:49,359 - INFO - VM name : ctest-TestRoutersBasic-81768906-77278478 2025-10-11 16:46:49,421 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:46:54,483 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:46:59,552 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:47:04,611 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:47:09,669 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:47:14,727 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:47:14,727 - ERROR - VM 0aa75746-e818-4414-89f0-5c671b749799 is not launched yet 2025-10-11 16:47:16,728 - INFO - VM name : ctest-TestRoutersBasic-81768906-77278478 2025-10-11 16:47:16,787 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:47:21,848 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:47:26,922 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:47:31,976 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:47:37,048 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:47:42,105 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:47:42,106 - ERROR - VM 0aa75746-e818-4414-89f0-5c671b749799 is not launched yet 2025-10-11 16:47:44,106 - INFO - VM name : ctest-TestRoutersBasic-81768906-77278478 2025-10-11 16:47:44,164 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:47:49,223 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:47:54,285 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:47:59,344 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:48:04,404 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:48:09,466 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:48:09,466 - ERROR - VM 0aa75746-e818-4414-89f0-5c671b749799 is not launched yet 2025-10-11 16:48:11,467 - INFO - VM name : ctest-TestRoutersBasic-81768906-77278478 2025-10-11 16:48:11,525 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:48:16,587 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:48:21,649 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:48:26,815 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:48:31,874 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:48:36,943 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:48:36,943 - ERROR - VM 0aa75746-e818-4414-89f0-5c671b749799 is not launched yet 2025-10-11 16:48:36,944 - INFO - VM name : ctest-TestRoutersBasic-81768906-77278478 2025-10-11 16:48:37,028 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:48:42,095 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:48:47,155 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:48:52,215 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:48:57,278 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:49:02,342 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:49:02,342 - ERROR - VM 0aa75746-e818-4414-89f0-5c671b749799 is not launched yet 2025-10-11 16:49:04,342 - INFO - VM name : ctest-TestRoutersBasic-81768906-77278478 2025-10-11 16:49:04,397 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:49:09,462 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:49:14,519 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:49:19,575 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:49:24,633 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:49:29,692 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:49:29,692 - ERROR - VM 0aa75746-e818-4414-89f0-5c671b749799 is not launched yet 2025-10-11 16:49:31,693 - INFO - VM name : ctest-TestRoutersBasic-81768906-77278478 2025-10-11 16:49:31,749 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:49:36,810 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:49:41,903 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:49:46,960 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:49:52,015 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:49:57,081 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:49:57,081 - ERROR - VM 0aa75746-e818-4414-89f0-5c671b749799 is not launched yet 2025-10-11 16:49:59,082 - INFO - VM name : ctest-TestRoutersBasic-81768906-77278478 2025-10-11 16:49:59,139 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:50:04,202 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:50:09,265 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:50:14,322 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:50:19,390 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:50:24,448 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:50:24,448 - ERROR - VM 0aa75746-e818-4414-89f0-5c671b749799 is not launched yet 2025-10-11 16:50:26,449 - INFO - VM name : ctest-TestRoutersBasic-81768906-77278478 2025-10-11 16:50:26,503 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:50:31,558 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:50:36,619 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:50:41,680 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:50:46,788 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:50:51,847 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:50:51,848 - ERROR - VM 0aa75746-e818-4414-89f0-5c671b749799 is not launched yet 2025-10-11 16:50:53,848 - INFO - VM name : ctest-TestRoutersBasic-81768906-77278478 2025-10-11 16:50:53,902 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:50:58,967 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:51:04,026 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:51:09,090 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:51:14,149 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:51:19,213 - DEBUG - VM ctest-TestRoutersBasic-81768906-77278478 : Status=ERROR, Addresses : {} 2025-10-11 16:51:19,213 - ERROR - VM 0aa75746-e818-4414-89f0-5c671b749799 is not launched yet 2025-10-11 16:51:19,214 - INFO - Deleting interface with subnet_id e26656e6-f3e8-46e5-bdfa-6954fd46a564, port_id None from router b314a417-e9cf-4a34-8bc1-b6b03c6f6e5d 2025-10-11 16:51:19,389 - INFO - Skip interface_detach for VM ctest-TestRoutersBasic-81768906-77278478 in state ERROR 2025-10-11 16:51:19,389 - INFO - Deleting VM ctest-TestRoutersBasic-81768906-77278478 2025-10-11 16:51:19,414 - ERROR - VM ctest-TestRoutersBasic-81768906-77278478 has failed to come up 2025-10-11 16:51:19,414 - ERROR - Fault seen in nova show is: {'code': 500, 'created': '2025-10-11T16:45:50Z', 'message': "Failure running os_vif plugin plug method: Failed to plug VIF VIFGeneric(active=True,address=02:a6:dd:7f:46:ae,has_traffic_filtering=True,id=a6dd7f46-aea0-4279-9337-8c6b73d7b51a,network=Network(7e931aad-1dab-4d5c-9c3a-fd34fdb9a2cb),plugin='vrouter',port_p", 'details': 'Traceback (most recent call last):\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_vif/__init__.py", line 77, in plug\n plugin.plug(vif, instance_info)\n File "/opt/plugin/site-packages/vif_plug_vrouter/vrouter.py", line 271, in plug\n self._vrouter_port_add(instance_info, vif)\n File "/opt/plugin/site-packages/vif_plug_vrouter/vrouter.py", line 263, in _vrouter_port_add\n vhostuser_socket, vhostuser_mode)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap\n r_call_timeout)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 215, in remote_call\n raise exc_type(*result[2])\nvif_plug_vrouter.exception.VrouterPortControlError: Error during the call to vrouter-port-control: (\'vrouter-port-control\', \'--oper=add\', \'--uuid=a6dd7f46-aea0-4279-9337-8c6b73d7b51a\', \'--instance_uuid=0aa75746-e818-4414-89f0-5c671b749799\', \'--vn_uuid=7e931aad-1dab-4d5c-9c3a-fd34fdb9a2cb\', \'--vm_project_uuid=a64fce7a46dd4bb7985dc5a6d13e2a26\', \'--ip_address=3.190.205.195\', \'--ipv6_address=None\', \'--vm_name=ctest-TestRoutersBasic-81768906-77278478\', \'--mac=02:a6:dd:7f:46:ae\', \'--tap_name=tapa6dd7f46-ae\', \'--port_type=NovaVMPort\', \'--tx_vlan_id=-1\', \'--rx_vlan_id=-1\')\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/vif.py", line 699, in _plug_os_vif\n os_vif.plug(vif, instance_info)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_vif/__init__.py", line 82, in plug\n raise os_vif.exception.PlugException(vif=vif, err=err)\nos_vif.exception.PlugException: Failed to plug VIF VIFGeneric(active=True,address=02:a6:dd:7f:46:ae,has_traffic_filtering=True,id=a6dd7f46-aea0-4279-9337-8c6b73d7b51a,network=Network(7e931aad-1dab-4d5c-9c3a-fd34fdb9a2cb),plugin=\'vrouter\',port_profile=,preserve_on_delete=False,vif_name=\'tapa6dd7f46-ae\'). Got error: Error during the call to vrouter-port-control: (\'vrouter-port-control\', \'--oper=add\', \'--uuid=a6dd7f46-aea0-4279-9337-8c6b73d7b51a\', \'--instance_uuid=0aa75746-e818-4414-89f0-5c671b749799\', \'--vn_uuid=7e931aad-1dab-4d5c-9c3a-fd34fdb9a2cb\', \'--vm_project_uuid=a64fce7a46dd4bb7985dc5a6d13e2a26\', \'--ip_address=3.190.205.195\', \'--ipv6_address=None\', \'--vm_name=ctest-TestRoutersBasic-81768906-77278478\', \'--mac=02:a6:dd:7f:46:ae\', \'--tap_name=tapa6dd7f46-ae\', \'--port_type=NovaVMPort\', \'--tx_vlan_id=-1\', \'--rx_vlan_id=-1\')\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 2521, in _build_and_run_instance\n accel_info=accel_info)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 4339, in spawn\n cleanup_instance_disks=created_disks)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7523, in _create_guest_with_network\n cleanup_instance_disks=cleanup_instance_disks)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7492, in _create_guest_with_network\n self.plug_vifs(instance, network_info)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1386, in plug_vifs\n self.vif_driver.plug(instance, vif)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/vif.py", line 723, in plug\n self._plug_os_vif(instance, vif_obj)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/vif.py", line 703, in _plug_os_vif\n raise exception.InternalError(msg)\nnova.exception.InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFGeneric(active=True,address=02:a6:dd:7f:46:ae,has_traffic_filtering=True,id=a6dd7f46-aea0-4279-9337-8c6b73d7b51a,network=Network(7e931aad-1dab-4d5c-9c3a-fd34fdb9a2cb),plugin=\'vrouter\',port_profile=,preserve_on_delete=False,vif_name=\'tapa6dd7f46-ae\'). Got error: Error during the call to vrouter-port-control: (\'vrouter-port-control\', \'--oper=add\', \'--uuid=a6dd7f46-aea0-4279-9337-8c6b73d7b51a\', \'--instance_uuid=0aa75746-e818-4414-89f0-5c671b749799\', \'--vn_uuid=7e931aad-1dab-4d5c-9c3a-fd34fdb9a2cb\', \'--vm_project_uuid=a64fce7a46dd4bb7985dc5a6d13e2a26\', \'--ip_address=3.190.205.195\', \'--ipv6_address=None\', \'--vm_name=ctest-TestRoutersBasic-81768906-77278478\', \'--mac=02:a6:dd:7f:46:ae\', \'--tap_name=tapa6dd7f46-ae\', \'--port_type=NovaVMPort\', \'--tx_vlan_id=-1\', \'--rx_vlan_id=-1\')\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 2337, in _do_build_and_run_instance\n filter_properties, request_spec, accel_uuids)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 2625, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=str(e))\nnova.exception.RescheduledException: Build of instance 0aa75746-e818-4414-89f0-5c671b749799 was re-scheduled: Failure running os_vif plugin plug method: Failed to plug VIF VIFGeneric(active=True,address=02:a6:dd:7f:46:ae,has_traffic_filtering=True,id=a6dd7f46-aea0-4279-9337-8c6b73d7b51a,network=Network(7e931aad-1dab-4d5c-9c3a-fd34fdb9a2cb),plugin=\'vrouter\',port_profile=,preserve_on_delete=False,vif_name=\'tapa6dd7f46-ae\'). Got error: Error during the call to vrouter-port-control: (\'vrouter-port-control\', \'--oper=add\', \'--uuid=a6dd7f46-aea0-4279-9337-8c6b73d7b51a\', \'--instance_uuid=0aa75746-e818-4414-89f0-5c671b749799\', \'--vn_uuid=7e931aad-1dab-4d5c-9c3a-fd34fdb9a2cb\', \'--vm_project_uuid=a64fce7a46dd4bb7985dc5a6d13e2a26\', \'--ip_address=3.190.205.195\', \'--ipv6_address=None\', \'--vm_name=ctest-TestRoutersBasic-81768906-77278478\', \'--mac=02:a6:dd:7f:46:ae\', \'--tap_name=tapa6dd7f46-ae\', \'--port_type=NovaVMPort\', \'--tx_vlan_id=-1\', \'--rx_vlan_id=-1\')\n'} 2025-10-11 16:51:19,415 - INFO - Skip interface_detach for VM ctest-TestRoutersBasic-81768906-35933601 in state ERROR 2025-10-11 16:51:19,415 - INFO - Deleting VM ctest-TestRoutersBasic-81768906-35933601 2025-10-11 16:51:19,415 - ERROR - VM ctest-TestRoutersBasic-81768906-35933601 has failed to come up 2025-10-11 16:51:19,415 - ERROR - Fault seen in nova show is: {'code': 500, 'created': '2025-10-11T16:45:49Z', 'message': "Failure running os_vif plugin plug method: Failed to plug VIF VIFGeneric(active=True,address=02:83:97:c0:1a:6d,has_traffic_filtering=True,id=8397c01a-6d3c-4496-aebd-26f1c25b3434,network=Network(347486ad-b16d-489d-a302-05721c5117cc),plugin='vrouter',port_p", 'details': 'Traceback (most recent call last):\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_vif/__init__.py", line 77, in plug\n plugin.plug(vif, instance_info)\n File "/opt/plugin/site-packages/vif_plug_vrouter/vrouter.py", line 271, in plug\n self._vrouter_port_add(instance_info, vif)\n File "/opt/plugin/site-packages/vif_plug_vrouter/vrouter.py", line 263, in _vrouter_port_add\n vhostuser_socket, vhostuser_mode)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap\n r_call_timeout)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 215, in remote_call\n raise exc_type(*result[2])\nvif_plug_vrouter.exception.VrouterPortControlError: Error during the call to vrouter-port-control: (\'vrouter-port-control\', \'--oper=add\', \'--uuid=8397c01a-6d3c-4496-aebd-26f1c25b3434\', \'--instance_uuid=ad47c041-ca41-4638-bf51-483e5c27f44d\', \'--vn_uuid=347486ad-b16d-489d-a302-05721c5117cc\', \'--vm_project_uuid=a64fce7a46dd4bb7985dc5a6d13e2a26\', \'--ip_address=23.219.150.131\', \'--ipv6_address=None\', \'--vm_name=ctest-TestRoutersBasic-81768906-35933601\', \'--mac=02:83:97:c0:1a:6d\', \'--tap_name=tap8397c01a-6d\', \'--port_type=NovaVMPort\', \'--tx_vlan_id=-1\', \'--rx_vlan_id=-1\')\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/vif.py", line 699, in _plug_os_vif\n os_vif.plug(vif, instance_info)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_vif/__init__.py", line 82, in plug\n raise os_vif.exception.PlugException(vif=vif, err=err)\nos_vif.exception.PlugException: Failed to plug VIF VIFGeneric(active=True,address=02:83:97:c0:1a:6d,has_traffic_filtering=True,id=8397c01a-6d3c-4496-aebd-26f1c25b3434,network=Network(347486ad-b16d-489d-a302-05721c5117cc),plugin=\'vrouter\',port_profile=,preserve_on_delete=False,vif_name=\'tap8397c01a-6d\'). Got error: Error during the call to vrouter-port-control: (\'vrouter-port-control\', \'--oper=add\', \'--uuid=8397c01a-6d3c-4496-aebd-26f1c25b3434\', \'--instance_uuid=ad47c041-ca41-4638-bf51-483e5c27f44d\', \'--vn_uuid=347486ad-b16d-489d-a302-05721c5117cc\', \'--vm_project_uuid=a64fce7a46dd4bb7985dc5a6d13e2a26\', \'--ip_address=23.219.150.131\', \'--ipv6_address=None\', \'--vm_name=ctest-TestRoutersBasic-81768906-35933601\', \'--mac=02:83:97:c0:1a:6d\', \'--tap_name=tap8397c01a-6d\', \'--port_type=NovaVMPort\', \'--tx_vlan_id=-1\', \'--rx_vlan_id=-1\')\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 2521, in _build_and_run_instance\n accel_info=accel_info)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 4339, in spawn\n cleanup_instance_disks=created_disks)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7523, in _create_guest_with_network\n cleanup_instance_disks=cleanup_instance_disks)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7492, in _create_guest_with_network\n self.plug_vifs(instance, network_info)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1386, in plug_vifs\n self.vif_driver.plug(instance, vif)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/vif.py", line 723, in plug\n self._plug_os_vif(instance, vif_obj)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/vif.py", line 703, in _plug_os_vif\n raise exception.InternalError(msg)\nnova.exception.InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFGeneric(active=True,address=02:83:97:c0:1a:6d,has_traffic_filtering=True,id=8397c01a-6d3c-4496-aebd-26f1c25b3434,network=Network(347486ad-b16d-489d-a302-05721c5117cc),plugin=\'vrouter\',port_profile=,preserve_on_delete=False,vif_name=\'tap8397c01a-6d\'). Got error: Error during the call to vrouter-port-control: (\'vrouter-port-control\', \'--oper=add\', \'--uuid=8397c01a-6d3c-4496-aebd-26f1c25b3434\', \'--instance_uuid=ad47c041-ca41-4638-bf51-483e5c27f44d\', \'--vn_uuid=347486ad-b16d-489d-a302-05721c5117cc\', \'--vm_project_uuid=a64fce7a46dd4bb7985dc5a6d13e2a26\', \'--ip_address=23.219.150.131\', \'--ipv6_address=None\', \'--vm_name=ctest-TestRoutersBasic-81768906-35933601\', \'--mac=02:83:97:c0:1a:6d\', \'--tap_name=tap8397c01a-6d\', \'--port_type=NovaVMPort\', \'--tx_vlan_id=-1\', \'--rx_vlan_id=-1\')\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 2337, in _do_build_and_run_instance\n filter_properties, request_spec, accel_uuids)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 2625, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=str(e))\nnova.exception.RescheduledException: Build of instance ad47c041-ca41-4638-bf51-483e5c27f44d was re-scheduled: Failure running os_vif plugin plug method: Failed to plug VIF VIFGeneric(active=True,address=02:83:97:c0:1a:6d,has_traffic_filtering=True,id=8397c01a-6d3c-4496-aebd-26f1c25b3434,network=Network(347486ad-b16d-489d-a302-05721c5117cc),plugin=\'vrouter\',port_profile=,preserve_on_delete=False,vif_name=\'tap8397c01a-6d\'). Got error: Error during the call to vrouter-port-control: (\'vrouter-port-control\', \'--oper=add\', \'--uuid=8397c01a-6d3c-4496-aebd-26f1c25b3434\', \'--instance_uuid=ad47c041-ca41-4638-bf51-483e5c27f44d\', \'--vn_uuid=347486ad-b16d-489d-a302-05721c5117cc\', \'--vm_project_uuid=a64fce7a46dd4bb7985dc5a6d13e2a26\', \'--ip_address=23.219.150.131\', \'--ipv6_address=None\', \'--vm_name=ctest-TestRoutersBasic-81768906-35933601\', \'--mac=02:83:97:c0:1a:6d\', \'--tap_name=tap8397c01a-6d\', \'--port_type=NovaVMPort\', \'--tx_vlan_id=-1\', \'--rx_vlan_id=-1\')\n'} 2025-10-11 16:51:19,415 - INFO - Deleting VN ctest-ext_vn-89097485 2025-10-11 16:51:19,463 - DEBUG - VN 7e931aad-1dab-4d5c-9c3a-fd34fdb9a2cb still in use: Unable to complete operation on network 7e931aad-1dab-4d5c-9c3a-fd34fdb9a2cb. There are one or more ports still in use on the network. Neutron server returns request_ids: ['req-a03d3922-3525-47ec-a2fb-6d84cbbd7aa9'] 2025-10-11 16:51:19,463 - WARNING - Deleting VN ctest-ext_vn-89097485 failed..Will retry 2025-10-11 16:51:21,560 - DEBUG - Response for deleting network () 2025-10-11 16:51:21,560 - INFO - Deleting VN ctest-vn-71995652 2025-10-11 16:51:21,661 - DEBUG - Response for deleting network () 2025-10-11 16:51:21,778 - ERROR -
Traceback (most recent call last):
  File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper
    result = function(self, *args, **kwargs)
  File "/contrail-test/scripts/neutron/test_routers_basic.py", line 52, in test_basic_snat_behavior_without_external_connectivity
    vm2_fixture.vm_ip), 'Ping from vm_left to vm_right through snat failed'
  File "/contrail-test/fixtures/vm_test.py", line 354, in vm_ip
    assert self.verify_vm_launched()
AssertionError
2025-10-11 16:51:21,784 - DEBUG - No XMPP flaps were noticed during the test 2025-10-11 16:51:21,785 - INFO -
2025-10-11 16:51:21,785 - INFO - END TEST : test_basic_snat_behavior_without_external_connectivity : FAILED[0:05:38]
2025-10-11 16:51:21,785 - INFO - -------------------------------------------------------------------------------- 2025-10-11 16:51:22,465 - INFO - Deleted project: ctest-TestRoutersBasic-81768906, ID : a64fce7a-46dd-4bb7-985d-c5a6d13e2a26