2025-10-10 12:01:55,127 - INFO - Domain Default found not creating
2025-10-10 12:01:55,341 - INFO - Project ctest-TestRoutersBasic-03257302 not found, creating it
2025-10-10 12:01:55,881 - INFO - Created Project:ctest-TestRoutersBasic-03257302, ID : 51e2d69a-8261-4ddf-9a54-18075c66e1c1 
2025-10-10 12:01:57,579 - INFO - ================================================================================
2025-10-10 12:01:57,579 - INFO - STARTING TEST    : test_basic_snat_behavior_without_external_connectivity
2025-10-10 12:01:57,579 - INFO - TEST DESCRIPTION : Create an external network, a router
        set router-gateway to external network
        launch a private network and attach it to router
        validate left vm pinging right vm through Snat
       
2025-10-10 12:01:57,723 - DEBUG - Nothing to compare xmpp stats {'10.0.0.27': {'10.20.0.254': '0'}} with
2025-10-10 12:01:57,723 - INFO - Initial checks done. Running the testcase now
2025-10-10 12:01:57,723 - INFO - 
2025-10-10 12:01:58,507 - INFO - Default SG to be edited for allow all on project: ctest-TestRoutersBasic-03257302
2025-10-10 12:01:58,611 - INFO - Adding rules to the default security group in Project ctest-TestRoutersBasic-03257302
2025-10-10 12:01:58,954 - DEBUG - Reading firewall policy {'fq_name': ['default-domain', 'ctest-TestRoutersBasic-03257302', 'default']}
2025-10-10 12:01:59,299 - DEBUG - Response for create_network : {'network': {'id': '54f5ea95-84f1-4955-8a23-4d51d5c88f42', 'name': 'ctest-vn-87441222', 'tenant_id': '51e2d69a82614ddf9a5418075c66e1c1', 'project_id': '51e2d69a82614ddf9a5418075c66e1c1', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestRoutersBasic-03257302', 'ctest-vn-87441222'], 'port_security_enabled': True, 'description': ''}}
2025-10-10 12:01:59,534 - DEBUG - Response for create_subnet : {'subnet': {'id': 'c9386653-c2f4-4649-8af7-056ed43691f7', 'name': '', 'tenant_id': '51e2d69a82614ddf9a5418075c66e1c1', 'network_id': '54f5ea95-84f1-4955-8a23-4d51d5c88f42', 'ip_version': 4, 'cidr': '157.39.218.64/26', 'allocation_pools': [{'start': '157.39.218.66', 'end': '157.39.218.126'}], 'gateway_ip': '157.39.218.65', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '157.39.218.66', 'tags': [], 'project_id': '51e2d69a82614ddf9a5418075c66e1c1'}}
2025-10-10 12:01:59,559 - INFO - Created VN ctest-vn-87441222
2025-10-10 12:01:59,615 - DEBUG - VN ctest-vn-87441222 UUID is 54f5ea95-84f1-4955-8a23-4d51d5c88f42
2025-10-10 12:01:59,973 - DEBUG - Response for create_network : {'network': {'id': '6db2d707-f66b-4201-a1cc-370b30f2cdb6', 'name': 'ctest-ext_vn-35121328', 'tenant_id': '51e2d69a82614ddf9a5418075c66e1c1', 'project_id': '51e2d69a82614ddf9a5418075c66e1c1', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': True, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestRoutersBasic-03257302', 'ctest-ext_vn-35121328'], 'port_security_enabled': True, 'description': ''}}
2025-10-10 12:02:00,163 - DEBUG - Response for create_subnet : {'subnet': {'id': '7687d968-c7f5-4286-87ad-c072588b3a23', 'name': '', 'tenant_id': '51e2d69a82614ddf9a5418075c66e1c1', 'network_id': '6db2d707-f66b-4201-a1cc-370b30f2cdb6', 'ip_version': 4, 'cidr': '113.84.224.0/26', 'allocation_pools': [{'start': '113.84.224.2', 'end': '113.84.224.62'}], 'gateway_ip': '113.84.224.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '113.84.224.2', 'tags': [], 'project_id': '51e2d69a82614ddf9a5418075c66e1c1'}}
2025-10-10 12:02:00,185 - INFO - Created VN ctest-ext_vn-35121328
2025-10-10 12:02:00,239 - DEBUG - VN ctest-ext_vn-35121328 UUID is 6db2d707-f66b-4201-a1cc-370b30f2cdb6
2025-10-10 12:02:00,499 - DEBUG - Services list from nova: [<Service: 1>, <Service: 1>, <Service: 2>]
2025-10-10 12:02:01,684 - INFO - VM ([<Server: ctest-TestRoutersBasic-03257302-42866929>]) created on node: (cn-jenkins-deploy-platform-ansible-os-4285-1), Zone: (nova:cn-jenkins-deploy-platform-ansible-os-4285-1)
2025-10-10 12:02:03,336 - INFO - VM ([<Server: ctest-TestRoutersBasic-03257302-09310850>]) created on node: (cn-jenkins-deploy-platform-ansible-os-4285-1), Zone: (nova:cn-jenkins-deploy-platform-ansible-os-4285-1)
2025-10-10 12:02:03,436 - INFO - Setting gateway for router ebdc3deb-582b-4550-bb3a-faa45b0e53c2 to network 6db2d707-f66b-4201-a1cc-370b30f2cdb6 
2025-10-10 12:02:03,515 - INFO - Adding interface with subnet_id c9386653-c2f4-4649-8af7-056ed43691f7, port_id None to router ebdc3deb-582b-4550-bb3a-faa45b0e53c2
2025-10-10 12:02:03,965 - INFO - Waiting for VM ctest-TestRoutersBasic-03257302-42866929 to be up..
2025-10-10 12:02:04,143 - DEBUG - VM <Server: ctest-TestRoutersBasic-03257302-42866929> is still in BUILD state, Expected: ACTIVE
2025-10-10 12:02:09,246 - DEBUG - VM <Server: ctest-TestRoutersBasic-03257302-42866929> is in ERROR state now
2025-10-10 12:02:09,247 - ERROR - VM in error state.
2025-10-10 12:02:09,247 - ERROR - VM is not in ACTIVE state
2025-10-10 12:02:09,349 - ERROR - Fatal Nova Exception while getting VM detail
2025-10-10 12:02:09,349 - DEBUG - None
2025-10-10 12:02:09,349 - INFO - Waiting for VM ctest-TestRoutersBasic-03257302-09310850 to be up..
2025-10-10 12:02:09,421 - DEBUG - VM <Server: ctest-TestRoutersBasic-03257302-09310850> is in ERROR state now
2025-10-10 12:02:09,421 - ERROR - VM in error state.
2025-10-10 12:02:09,421 - ERROR - VM is not in ACTIVE state
2025-10-10 12:02:09,528 - ERROR - Fatal Nova Exception while getting VM detail
2025-10-10 12:02:09,528 - DEBUG - None
2025-10-10 12:02:09,528 - INFO - VM name : ctest-TestRoutersBasic-03257302-09310850
2025-10-10 12:02:09,607 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:02:14,689 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:02:19,759 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:02:24,847 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:02:29,920 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:02:35,003 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:02:35,003 - ERROR - VM 21996100-15d6-4449-8dff-a55ad2ff4242 is not launched yet
2025-10-10 12:02:37,004 - INFO - VM name : ctest-TestRoutersBasic-03257302-09310850
2025-10-10 12:02:37,069 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:02:42,149 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:02:47,222 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:02:52,298 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:02:57,375 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:03:02,442 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:03:02,442 - ERROR - VM 21996100-15d6-4449-8dff-a55ad2ff4242 is not launched yet
2025-10-10 12:03:04,442 - INFO - VM name : ctest-TestRoutersBasic-03257302-09310850
2025-10-10 12:03:04,515 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:03:09,597 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:03:14,667 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:03:19,744 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:03:24,812 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:03:29,881 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:03:29,881 - ERROR - VM 21996100-15d6-4449-8dff-a55ad2ff4242 is not launched yet
2025-10-10 12:03:31,883 - INFO - VM name : ctest-TestRoutersBasic-03257302-09310850
2025-10-10 12:03:31,955 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:03:37,022 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:03:42,096 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:03:47,164 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:03:52,237 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:03:57,312 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:03:57,312 - ERROR - VM 21996100-15d6-4449-8dff-a55ad2ff4242 is not launched yet
2025-10-10 12:03:59,313 - INFO - VM name : ctest-TestRoutersBasic-03257302-09310850
2025-10-10 12:03:59,397 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:04:04,474 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:04:09,551 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:04:14,620 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:04:19,713 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:04:24,790 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:04:24,790 - ERROR - VM 21996100-15d6-4449-8dff-a55ad2ff4242 is not launched yet
2025-10-10 12:04:26,791 - INFO - VM name : ctest-TestRoutersBasic-03257302-09310850
2025-10-10 12:04:26,868 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:04:31,942 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:04:37,019 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:04:42,103 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:04:47,180 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:04:52,252 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:04:52,252 - ERROR - VM 21996100-15d6-4449-8dff-a55ad2ff4242 is not launched yet
2025-10-10 12:04:52,253 - INFO - VM name : ctest-TestRoutersBasic-03257302-09310850
2025-10-10 12:04:52,323 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:04:57,397 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:05:02,466 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:05:07,547 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:05:12,630 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:05:17,696 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:05:17,696 - ERROR - VM 21996100-15d6-4449-8dff-a55ad2ff4242 is not launched yet
2025-10-10 12:05:19,697 - INFO - VM name : ctest-TestRoutersBasic-03257302-09310850
2025-10-10 12:05:19,773 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:05:24,840 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:05:29,912 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:05:34,990 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:05:40,104 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:05:45,174 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:05:45,174 - ERROR - VM 21996100-15d6-4449-8dff-a55ad2ff4242 is not launched yet
2025-10-10 12:05:47,176 - INFO - VM name : ctest-TestRoutersBasic-03257302-09310850
2025-10-10 12:05:47,258 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:05:52,337 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:05:57,409 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:06:02,485 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:06:07,558 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:06:12,632 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:06:12,632 - ERROR - VM 21996100-15d6-4449-8dff-a55ad2ff4242 is not launched yet
2025-10-10 12:06:14,633 - INFO - VM name : ctest-TestRoutersBasic-03257302-09310850
2025-10-10 12:06:14,710 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:06:19,789 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:06:24,865 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:06:29,937 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:06:35,013 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:06:40,108 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:06:40,108 - ERROR - VM 21996100-15d6-4449-8dff-a55ad2ff4242 is not launched yet
2025-10-10 12:06:42,109 - INFO - VM name : ctest-TestRoutersBasic-03257302-09310850
2025-10-10 12:06:42,181 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:06:47,250 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:06:52,328 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:06:57,429 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:07:02,526 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:07:07,604 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:07:07,604 - ERROR - VM 21996100-15d6-4449-8dff-a55ad2ff4242 is not launched yet
2025-10-10 12:07:09,605 - INFO - VM name : ctest-TestRoutersBasic-03257302-09310850
2025-10-10 12:07:09,676 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:07:14,744 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:07:19,811 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:07:24,878 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:07:29,948 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:07:35,022 - DEBUG - VM ctest-TestRoutersBasic-03257302-09310850 : Status=ERROR, Addresses : {}
2025-10-10 12:07:35,022 - ERROR - VM 21996100-15d6-4449-8dff-a55ad2ff4242 is not launched yet
2025-10-10 12:07:35,022 - INFO - Deleting interface with subnet_id c9386653-c2f4-4649-8af7-056ed43691f7, port_id None from router ebdc3deb-582b-4550-bb3a-faa45b0e53c2
2025-10-10 12:07:35,266 - INFO - Skip interface_detach for VM ctest-TestRoutersBasic-03257302-09310850 in state ERROR
2025-10-10 12:07:35,266 - INFO - Deleting VM ctest-TestRoutersBasic-03257302-09310850
2025-10-10 12:07:35,292 - ERROR - VM ctest-TestRoutersBasic-03257302-09310850 has failed to come up
2025-10-10 12:07:35,292 - ERROR - Fault seen in nova show <vm-uuid> is:  {'code': 500, 'created': '2025-10-10T12:02:05Z', 'message': "Failure running os_vif plugin plug method: Failed to plug VIF VIFGeneric(active=True,address=02:2c:ee:3d:d2:fd,has_traffic_filtering=True,id=2cee3dd2-fdbe-42c0-9318-2441b4f47839,network=Network(6db2d707-f66b-4201-a1cc-370b30f2cdb6),plugin='vrouter',port_p", 'details': 'Traceback (most recent call last):\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_vif/__init__.py", line 77, in plug\n    plugin.plug(vif, instance_info)\n  File "/opt/plugin/site-packages/vif_plug_vrouter/vrouter.py", line 271, in plug\n    self._vrouter_port_add(instance_info, vif)\n  File "/opt/plugin/site-packages/vif_plug_vrouter/vrouter.py", line 263, in _vrouter_port_add\n    vhostuser_socket, vhostuser_mode)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap\n    r_call_timeout)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 215, in remote_call\n    raise exc_type(*result[2])\nvif_plug_vrouter.exception.VrouterPortControlError: Error during the call to vrouter-port-control: (\'vrouter-port-control\', \'--oper=add\', \'--uuid=2cee3dd2-fdbe-42c0-9318-2441b4f47839\', \'--instance_uuid=21996100-15d6-4449-8dff-a55ad2ff4242\', \'--vn_uuid=6db2d707-f66b-4201-a1cc-370b30f2cdb6\', \'--vm_project_uuid=51e2d69a82614ddf9a5418075c66e1c1\', \'--ip_address=113.84.224.3\', \'--ipv6_address=None\', \'--vm_name=ctest-TestRoutersBasic-03257302-09310850\', \'--mac=02:2c:ee:3d:d2:fd\', \'--tap_name=tap2cee3dd2-fd\', \'--port_type=NovaVMPort\', \'--tx_vlan_id=-1\', \'--rx_vlan_id=-1\')\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/vif.py", line 699, in _plug_os_vif\n    os_vif.plug(vif, instance_info)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_vif/__init__.py", line 82, in plug\n    raise os_vif.exception.PlugException(vif=vif, err=err)\nos_vif.exception.PlugException: Failed to plug VIF VIFGeneric(active=True,address=02:2c:ee:3d:d2:fd,has_traffic_filtering=True,id=2cee3dd2-fdbe-42c0-9318-2441b4f47839,network=Network(6db2d707-f66b-4201-a1cc-370b30f2cdb6),plugin=\'vrouter\',port_profile=<?>,preserve_on_delete=False,vif_name=\'tap2cee3dd2-fd\'). Got error: Error during the call to vrouter-port-control: (\'vrouter-port-control\', \'--oper=add\', \'--uuid=2cee3dd2-fdbe-42c0-9318-2441b4f47839\', \'--instance_uuid=21996100-15d6-4449-8dff-a55ad2ff4242\', \'--vn_uuid=6db2d707-f66b-4201-a1cc-370b30f2cdb6\', \'--vm_project_uuid=51e2d69a82614ddf9a5418075c66e1c1\', \'--ip_address=113.84.224.3\', \'--ipv6_address=None\', \'--vm_name=ctest-TestRoutersBasic-03257302-09310850\', \'--mac=02:2c:ee:3d:d2:fd\', \'--tap_name=tap2cee3dd2-fd\', \'--port_type=NovaVMPort\', \'--tx_vlan_id=-1\', \'--rx_vlan_id=-1\')\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 2521, in _build_and_run_instance\n    accel_info=accel_info)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 4339, in spawn\n    cleanup_instance_disks=created_disks)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7523, in _create_guest_with_network\n    cleanup_instance_disks=cleanup_instance_disks)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n    self.force_reraise()\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n    raise self.value\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7492, in _create_guest_with_network\n    self.plug_vifs(instance, network_info)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1386, in plug_vifs\n    self.vif_driver.plug(instance, vif)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/vif.py", line 723, in plug\n    self._plug_os_vif(instance, vif_obj)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/vif.py", line 703, in _plug_os_vif\n    raise exception.InternalError(msg)\nnova.exception.InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFGeneric(active=True,address=02:2c:ee:3d:d2:fd,has_traffic_filtering=True,id=2cee3dd2-fdbe-42c0-9318-2441b4f47839,network=Network(6db2d707-f66b-4201-a1cc-370b30f2cdb6),plugin=\'vrouter\',port_profile=<?>,preserve_on_delete=False,vif_name=\'tap2cee3dd2-fd\'). Got error: Error during the call to vrouter-port-control: (\'vrouter-port-control\', \'--oper=add\', \'--uuid=2cee3dd2-fdbe-42c0-9318-2441b4f47839\', \'--instance_uuid=21996100-15d6-4449-8dff-a55ad2ff4242\', \'--vn_uuid=6db2d707-f66b-4201-a1cc-370b30f2cdb6\', \'--vm_project_uuid=51e2d69a82614ddf9a5418075c66e1c1\', \'--ip_address=113.84.224.3\', \'--ipv6_address=None\', \'--vm_name=ctest-TestRoutersBasic-03257302-09310850\', \'--mac=02:2c:ee:3d:d2:fd\', \'--tap_name=tap2cee3dd2-fd\', \'--port_type=NovaVMPort\', \'--tx_vlan_id=-1\', \'--rx_vlan_id=-1\')\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 2337, in _do_build_and_run_instance\n    filter_properties, request_spec, accel_uuids)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 2625, in _build_and_run_instance\n    instance_uuid=instance.uuid, reason=str(e))\nnova.exception.RescheduledException: Build of instance 21996100-15d6-4449-8dff-a55ad2ff4242 was re-scheduled: Failure running os_vif plugin plug method: Failed to plug VIF VIFGeneric(active=True,address=02:2c:ee:3d:d2:fd,has_traffic_filtering=True,id=2cee3dd2-fdbe-42c0-9318-2441b4f47839,network=Network(6db2d707-f66b-4201-a1cc-370b30f2cdb6),plugin=\'vrouter\',port_profile=<?>,preserve_on_delete=False,vif_name=\'tap2cee3dd2-fd\'). Got error: Error during the call to vrouter-port-control: (\'vrouter-port-control\', \'--oper=add\', \'--uuid=2cee3dd2-fdbe-42c0-9318-2441b4f47839\', \'--instance_uuid=21996100-15d6-4449-8dff-a55ad2ff4242\', \'--vn_uuid=6db2d707-f66b-4201-a1cc-370b30f2cdb6\', \'--vm_project_uuid=51e2d69a82614ddf9a5418075c66e1c1\', \'--ip_address=113.84.224.3\', \'--ipv6_address=None\', \'--vm_name=ctest-TestRoutersBasic-03257302-09310850\', \'--mac=02:2c:ee:3d:d2:fd\', \'--tap_name=tap2cee3dd2-fd\', \'--port_type=NovaVMPort\', \'--tx_vlan_id=-1\', \'--rx_vlan_id=-1\')\n'}
2025-10-10 12:07:35,292 - INFO - Skip interface_detach for VM ctest-TestRoutersBasic-03257302-42866929 in state ERROR
2025-10-10 12:07:35,292 - INFO - Deleting VM ctest-TestRoutersBasic-03257302-42866929
2025-10-10 12:07:35,292 - ERROR - VM ctest-TestRoutersBasic-03257302-42866929 has failed to come up
2025-10-10 12:07:35,292 - ERROR - Fault seen in nova show <vm-uuid> is:  {'code': 500, 'created': '2025-10-10T12:02:04Z', 'message': "Failure running os_vif plugin plug method: Failed to plug VIF VIFGeneric(active=True,address=02:e6:ec:f2:2d:d7,has_traffic_filtering=True,id=e6ecf22d-d7bc-4f6c-bfa5-1964a928efc0,network=Network(54f5ea95-84f1-4955-8a23-4d51d5c88f42),plugin='vrouter',port_p", 'details': 'Traceback (most recent call last):\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_vif/__init__.py", line 77, in plug\n    plugin.plug(vif, instance_info)\n  File "/opt/plugin/site-packages/vif_plug_vrouter/vrouter.py", line 271, in plug\n    self._vrouter_port_add(instance_info, vif)\n  File "/opt/plugin/site-packages/vif_plug_vrouter/vrouter.py", line 263, in _vrouter_port_add\n    vhostuser_socket, vhostuser_mode)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap\n    r_call_timeout)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 215, in remote_call\n    raise exc_type(*result[2])\nvif_plug_vrouter.exception.VrouterPortControlError: Error during the call to vrouter-port-control: (\'vrouter-port-control\', \'--oper=add\', \'--uuid=e6ecf22d-d7bc-4f6c-bfa5-1964a928efc0\', \'--instance_uuid=4569ba14-7830-40c1-87ef-2b3a9473dd34\', \'--vn_uuid=54f5ea95-84f1-4955-8a23-4d51d5c88f42\', \'--vm_project_uuid=51e2d69a82614ddf9a5418075c66e1c1\', \'--ip_address=157.39.218.67\', \'--ipv6_address=None\', \'--vm_name=ctest-TestRoutersBasic-03257302-42866929\', \'--mac=02:e6:ec:f2:2d:d7\', \'--tap_name=tape6ecf22d-d7\', \'--port_type=NovaVMPort\', \'--tx_vlan_id=-1\', \'--rx_vlan_id=-1\')\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/vif.py", line 699, in _plug_os_vif\n    os_vif.plug(vif, instance_info)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_vif/__init__.py", line 82, in plug\n    raise os_vif.exception.PlugException(vif=vif, err=err)\nos_vif.exception.PlugException: Failed to plug VIF VIFGeneric(active=True,address=02:e6:ec:f2:2d:d7,has_traffic_filtering=True,id=e6ecf22d-d7bc-4f6c-bfa5-1964a928efc0,network=Network(54f5ea95-84f1-4955-8a23-4d51d5c88f42),plugin=\'vrouter\',port_profile=<?>,preserve_on_delete=False,vif_name=\'tape6ecf22d-d7\'). Got error: Error during the call to vrouter-port-control: (\'vrouter-port-control\', \'--oper=add\', \'--uuid=e6ecf22d-d7bc-4f6c-bfa5-1964a928efc0\', \'--instance_uuid=4569ba14-7830-40c1-87ef-2b3a9473dd34\', \'--vn_uuid=54f5ea95-84f1-4955-8a23-4d51d5c88f42\', \'--vm_project_uuid=51e2d69a82614ddf9a5418075c66e1c1\', \'--ip_address=157.39.218.67\', \'--ipv6_address=None\', \'--vm_name=ctest-TestRoutersBasic-03257302-42866929\', \'--mac=02:e6:ec:f2:2d:d7\', \'--tap_name=tape6ecf22d-d7\', \'--port_type=NovaVMPort\', \'--tx_vlan_id=-1\', \'--rx_vlan_id=-1\')\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 2521, in _build_and_run_instance\n    accel_info=accel_info)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 4339, in spawn\n    cleanup_instance_disks=created_disks)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7523, in _create_guest_with_network\n    cleanup_instance_disks=cleanup_instance_disks)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n    self.force_reraise()\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n    raise self.value\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7492, in _create_guest_with_network\n    self.plug_vifs(instance, network_info)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1386, in plug_vifs\n    self.vif_driver.plug(instance, vif)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/vif.py", line 723, in plug\n    self._plug_os_vif(instance, vif_obj)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/vif.py", line 703, in _plug_os_vif\n    raise exception.InternalError(msg)\nnova.exception.InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFGeneric(active=True,address=02:e6:ec:f2:2d:d7,has_traffic_filtering=True,id=e6ecf22d-d7bc-4f6c-bfa5-1964a928efc0,network=Network(54f5ea95-84f1-4955-8a23-4d51d5c88f42),plugin=\'vrouter\',port_profile=<?>,preserve_on_delete=False,vif_name=\'tape6ecf22d-d7\'). Got error: Error during the call to vrouter-port-control: (\'vrouter-port-control\', \'--oper=add\', \'--uuid=e6ecf22d-d7bc-4f6c-bfa5-1964a928efc0\', \'--instance_uuid=4569ba14-7830-40c1-87ef-2b3a9473dd34\', \'--vn_uuid=54f5ea95-84f1-4955-8a23-4d51d5c88f42\', \'--vm_project_uuid=51e2d69a82614ddf9a5418075c66e1c1\', \'--ip_address=157.39.218.67\', \'--ipv6_address=None\', \'--vm_name=ctest-TestRoutersBasic-03257302-42866929\', \'--mac=02:e6:ec:f2:2d:d7\', \'--tap_name=tape6ecf22d-d7\', \'--port_type=NovaVMPort\', \'--tx_vlan_id=-1\', \'--rx_vlan_id=-1\')\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 2337, in _do_build_and_run_instance\n    filter_properties, request_spec, accel_uuids)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 2625, in _build_and_run_instance\n    instance_uuid=instance.uuid, reason=str(e))\nnova.exception.RescheduledException: Build of instance 4569ba14-7830-40c1-87ef-2b3a9473dd34 was re-scheduled: Failure running os_vif plugin plug method: Failed to plug VIF VIFGeneric(active=True,address=02:e6:ec:f2:2d:d7,has_traffic_filtering=True,id=e6ecf22d-d7bc-4f6c-bfa5-1964a928efc0,network=Network(54f5ea95-84f1-4955-8a23-4d51d5c88f42),plugin=\'vrouter\',port_profile=<?>,preserve_on_delete=False,vif_name=\'tape6ecf22d-d7\'). Got error: Error during the call to vrouter-port-control: (\'vrouter-port-control\', \'--oper=add\', \'--uuid=e6ecf22d-d7bc-4f6c-bfa5-1964a928efc0\', \'--instance_uuid=4569ba14-7830-40c1-87ef-2b3a9473dd34\', \'--vn_uuid=54f5ea95-84f1-4955-8a23-4d51d5c88f42\', \'--vm_project_uuid=51e2d69a82614ddf9a5418075c66e1c1\', \'--ip_address=157.39.218.67\', \'--ipv6_address=None\', \'--vm_name=ctest-TestRoutersBasic-03257302-42866929\', \'--mac=02:e6:ec:f2:2d:d7\', \'--tap_name=tape6ecf22d-d7\', \'--port_type=NovaVMPort\', \'--tx_vlan_id=-1\', \'--rx_vlan_id=-1\')\n'}
2025-10-10 12:07:35,293 - INFO - Deleting VN ctest-ext_vn-35121328 
2025-10-10 12:07:35,361 - DEBUG - VN 6db2d707-f66b-4201-a1cc-370b30f2cdb6 still in use: Unable to complete operation on network 6db2d707-f66b-4201-a1cc-370b30f2cdb6. There are one or more ports still in use on the network.
Neutron server returns request_ids: ['req-10801a2d-5b25-40bb-858b-99e8beffb8c9']
2025-10-10 12:07:35,361 - WARNING - Deleting VN ctest-ext_vn-35121328 failed..Will retry
2025-10-10 12:07:37,500 - DEBUG - Response for deleting network ()
2025-10-10 12:07:37,500 - INFO - Deleting VN ctest-vn-87441222 
2025-10-10 12:07:37,629 - DEBUG - Response for deleting network ()
2025-10-10 12:07:37,767 - ERROR - <pre>Traceback (most recent call last):
  File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper
    result = function(self, *args, **kwargs)
  File "/contrail-test/scripts/neutron/test_routers_basic.py", line 52, in test_basic_snat_behavior_without_external_connectivity
    vm2_fixture.vm_ip), 'Ping from vm_left to vm_right through snat failed'
  File "/contrail-test/fixtures/vm_test.py", line 354, in vm_ip
    assert self.verify_vm_launched()
AssertionError
</pre>

2025-10-10 12:07:37,774 - DEBUG - No XMPP flaps were noticed during the test
2025-10-10 12:07:37,774 - INFO - 
2025-10-10 12:07:37,774 - INFO - END TEST : test_basic_snat_behavior_without_external_connectivity : FAILED[0:05:40]
2025-10-10 12:07:37,774 - INFO - --------------------------------------------------------------------------------
2025-10-10 12:07:38,573 - INFO - Deleted project: ctest-TestRoutersBasic-03257302, ID : 51e2d69a-8261-4ddf-9a54-18075c66e1c1