2025-07-31 02:42:34,407 - INFO - Domain Default found not creating 2025-07-31 02:42:34,788 - INFO - Project ctest-TestDSNAT-89822341 not found, creating it 2025-07-31 02:42:35,441 - INFO - Created Project:ctest-TestDSNAT-89822341, ID : 43e31bc7-88c7-40c9-8ed8-da5fdaa0784b 2025-07-31 02:42:37,499 - INFO - ================================================================================ 2025-07-31 02:42:37,500 - INFO - STARTING TEST : test_dsnat_basic 2025-07-31 02:42:37,500 - INFO - TEST DESCRIPTION : create a VN and enable fabric SNAT launch two VMs in that VN verify ping between the VN and ping to the external IP disable fabric SNAT verify that the ping the external IP fails 2025-07-31 02:42:38,873 - DEBUG - Nothing to compare xmpp stats {'10.0.0.27': {'10.20.0.193': '0', '10.20.0.18': '0'}, '10.0.0.57': {'10.20.0.18': '0', '10.20.0.193': '0'}} with 2025-07-31 02:42:38,873 - INFO - Initial checks done. Running the testcase now 2025-07-31 02:42:38,873 - INFO - 2025-07-31 02:42:38,874 - INFO - Create VN, enable FABRIC SNAT and verify its routing instance 2025-07-31 02:42:39,632 - DEBUG - Response for create_network : {'network': {'id': 'ce49c2d3-5e54-42a5-a843-709fca4720e5', 'name': 'ctest-dsnat_vn-80763764', 'tenant_id': '43e31bc788c740c98ed8da5fdaa0784b', 'project_id': '43e31bc788c740c98ed8da5fdaa0784b', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestDSNAT-89822341', 'ctest-dsnat_vn-80763764'], 'port_security_enabled': True, 'description': ''}} 2025-07-31 02:42:39,934 - DEBUG - Response for create_subnet : {'subnet': {'id': '3aeda360-96af-4b0a-a3f4-46243b903a4a', 'name': '', 'tenant_id': '43e31bc788c740c98ed8da5fdaa0784b', 'network_id': 'ce49c2d3-5e54-42a5-a843-709fca4720e5', 'ip_version': 4, 'cidr': '185.24.82.0/26', 'allocation_pools': [{'start': '185.24.82.2', 'end': '185.24.82.62'}], 'gateway_ip': '185.24.82.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '185.24.82.2', 'tags': [], 'project_id': '43e31bc788c740c98ed8da5fdaa0784b'}} 2025-07-31 02:42:39,962 - INFO - Created VN ctest-dsnat_vn-80763764 2025-07-31 02:42:39,981 - DEBUG - VN ctest-dsnat_vn-80763764 UUID is ce49c2d3-5e54-42a5-a843-709fca4720e5 2025-07-31 02:42:40,185 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/ce49c2d3-5e54-42a5-a843-709fca4720e5 2025-07-31 02:42:40,502 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/ce49c2d3-5e54-42a5-a843-709fca4720e5 2025-07-31 02:42:40,547 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/87a186ea-a951-44d9-ad8e-0d191863198f 2025-07-31 02:42:40,705 - DEBUG - Services list from nova: [, , , ] 2025-07-31 02:42:40,737 - DEBUG - Image by name ubuntu not found 2025-07-31 02:42:40,737 - DEBUG - Installing image ubuntu 2025-07-31 02:42:40,737 - DEBUG - Image info {'name': 'ubuntu.vmdk.gz', 'webserver': '', 'location': '/images/converts/', 'username': 'ubuntu', 'password': 'ubuntu', 'type': 'os', 'flavor': 'contrail_flavor_tiny', 'params': '--container-format bare --disk-format vmdk --property vmware_disktype="sparse" --property vmware_adaptertype="ide" --property hw_scsi_model=virtio-scsi --property hw_disk_bus_model=virtio-scsi --property hw_disk_bus=virtio', 'name_docker': 'phusion-baseimage-enablesshd', 'vctmpl': 'ubuntu.vmtx', 'vcname': 'ubuntu-disk1.vmdk'} 2025-07-31 02:42:40,737 - DEBUG - Image build path http://nexus.opensdn.io/repository///images/converts//ubuntu.vmdk.gz 2025-07-31 02:42:40,737 - DEBUG - Download image from web http://nexus.opensdn.io/repository///images/converts//ubuntu.vmdk.gz 2025-07-31 02:42:40,747 - DEBUG - Image download (local=True) (proxy=None): Execute cmd: wget -nv http://nexus.opensdn.io/repository///images/converts//ubuntu.vmdk.gz -O /contrail-test/images/ubuntu.vmdk.gz 2025-07-31 02:42:40,776 - DEBUG - Image has been downloaded to /contrail-test/images/ubuntu.vmdk.gz 2025-07-31 02:42:40,776 - DEBUG - Unzip image 2025-07-31 02:42:40,777 - DEBUG - Image download (local=True) (proxy=None): Execute cmd: gunzip -f /contrail-test/images/ubuntu.vmdk.gz 2025-07-31 02:42:40,789 - DEBUG - Try to create image in user context 2025-07-31 02:42:40,893 - INFO - Deleting VN ctest-dsnat_vn-80763764 2025-07-31 02:42:41,096 - DEBUG - Response for deleting network () 2025-07-31 02:42:42,390 - ERROR - 
Traceback (most recent call last):
  File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper
    result = function(self, *args, **kwargs)
  File "/contrail-test/scripts/dsnat/test_dsnat.py", line 30, in test_dsnat_basic
    image_name='ubuntu')
  File "/contrail-test/common/base.py", line 281, in create_vm
    **kwargs)
  File "/contrail-test/common/base.py", line 254, in create_only_vm
    vm_obj.setUp()
  File "/contrail-test/fixtures/vm_test.py", line 188, in setUp
    self.create()
  File "/contrail-test/fixtures/vm_test.py", line 240, in create
    fixed_ips=self.fixed_ips)
  File "/contrail-test/fixtures/openstack.py", line 118, in create_vm
    return self.nova_h.create_vm(vm_name=vm_name, image_name=image_name, vn_ids=vn_ids, **kwargs)
  File "/contrail-test/fixtures/nova_test.py", line 577, in create_vm
    image = self.get_image(image_name=image_name)
  File "/contrail-test/fixtures/nova_test.py", line 198, in get_image
    self._install_image(image_name=image_name)
  File "/contrail-test/fixtures/nova_test.py", line 338, in _install_image
    return self.copy_and_glance(build_path, image_name, params)
  File "/contrail-test/fixtures/nova_test.py", line 407, in copy_and_glance
    image_id = self.glance_h.create_image(generic_image_name, image_path_real, **params)
  File "/contrail-test/fixtures/glance_test.py", line 50, in create_image
    self.upload_image(obj['id'], filename)
  File "/contrail-test/fixtures/glance_test.py", line 54, in upload_image
    self.obj.images.upload(uuid, open(filename, 'rb'))
FileNotFoundError: [Errno 2] No such file or directory: '/contrail-test/images/ubuntu.vmdk'
2025-07-31 02:42:42,422 - DEBUG - No XMPP flaps were noticed during the test 2025-07-31 02:42:42,422 - INFO - 2025-07-31 02:42:42,422 - INFO - END TEST : test_dsnat_basic : FAILED[0:00:05] 2025-07-31 02:42:42,422 - INFO - -------------------------------------------------------------------------------- 2025-07-31 02:42:42,431 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3732-1', 'an-jenkins-deploy-platform-ansible-os-3732-2']} 2025-07-31 02:42:42,431 - INFO - ================================================================================ 2025-07-31 02:42:42,431 - INFO - STARTING TEST : test_dsnat_bug_1749695 2025-07-31 02:42:42,432 - INFO - TEST DESCRIPTION : Testcase to verify the bug 1749695 create test VN , associate a policy to allow all traffic between VN and ip fabric associate the policy to the test VN and fabric VN set the test VN fabric mode as l3 verify the ping succeeds for intra VN and to the fabric IP 2025-07-31 02:42:43,735 - DEBUG - Nothing to compare xmpp stats {'10.0.0.27': {'10.20.0.193': '0', '10.20.0.18': '0'}, '10.0.0.57': {'10.20.0.18': '0', '10.20.0.193': '0'}} with 2025-07-31 02:42:43,736 - INFO - Initial checks done. Running the testcase now 2025-07-31 02:42:43,736 - INFO - 2025-07-31 02:42:43,736 - INFO - Create VN, enable FABRIC SNAT and verify its routing instance 2025-07-31 02:42:44,096 - DEBUG - Response for create_network : {'network': {'id': '39c57d50-64ac-4e6b-946a-49565fdeeb3e', 'name': 'ctest-dsnat_vn-50910904', 'tenant_id': '43e31bc788c740c98ed8da5fdaa0784b', 'project_id': '43e31bc788c740c98ed8da5fdaa0784b', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestDSNAT-89822341', 'ctest-dsnat_vn-50910904'], 'port_security_enabled': True, 'description': ''}} 2025-07-31 02:42:44,417 - DEBUG - Response for create_subnet : {'subnet': {'id': '426f2109-9313-4fff-bca4-654299fa6d39', 'name': '', 'tenant_id': '43e31bc788c740c98ed8da5fdaa0784b', 'network_id': '39c57d50-64ac-4e6b-946a-49565fdeeb3e', 'ip_version': 4, 'cidr': '125.48.232.64/26', 'allocation_pools': [{'start': '125.48.232.66', 'end': '125.48.232.126'}], 'gateway_ip': '125.48.232.65', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '125.48.232.66', 'tags': [], 'project_id': '43e31bc788c740c98ed8da5fdaa0784b'}} 2025-07-31 02:42:44,441 - INFO - Created VN ctest-dsnat_vn-50910904 2025-07-31 02:42:44,458 - DEBUG - VN ctest-dsnat_vn-50910904 UUID is 39c57d50-64ac-4e6b-946a-49565fdeeb3e 2025-07-31 02:42:44,641 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/39c57d50-64ac-4e6b-946a-49565fdeeb3e 2025-07-31 02:42:44,654 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/4b9e0ca6-ab8f-4307-b751-3fef56d6de84 2025-07-31 02:42:44,845 - DEBUG - Image by name ubuntu-traffic not found 2025-07-31 02:42:44,845 - DEBUG - Installing image ubuntu-traffic 2025-07-31 02:42:44,845 - DEBUG - Image info {'name': 'ubuntu-traffic.vmdk.gz', 'webserver': '', 'location': '/images/converts/', 'username': 'ubuntu', 'password': 'ubuntu', 'type': 'os', 'flavor': 'contrail_flavor_small', 'params': '--container-format bare --disk-format vmdk --property vmware_disktype="sparse" --property vmware_adaptertype="ide" --property hw_scsi_model=virtio-scsi --property hw_disk_bus_model=virtio-scsi --property hw_disk_bus=virtio', 'name_docker': 'ubuntu-traffic-docker', 'vctmpl': 'ubuntu-traffic.vmtx', 'vcname': 'ubuntu-traffic.vmdk', 'vmdk': 'ubuntu-traffic-disk1.vmdk', 'shrinked_vmdk': "'no'"} 2025-07-31 02:42:44,846 - DEBUG - Image build path http://nexus.opensdn.io/repository///images/converts//ubuntu-traffic.vmdk.gz 2025-07-31 02:42:44,846 - DEBUG - Download image from web http://nexus.opensdn.io/repository///images/converts//ubuntu-traffic.vmdk.gz 2025-07-31 02:42:44,864 - DEBUG - Image download (local=True) (proxy=None): Execute cmd: wget -nv http://nexus.opensdn.io/repository///images/converts//ubuntu-traffic.vmdk.gz -O /contrail-test/images/ubuntu-traffic.vmdk.gz 2025-07-31 02:42:50,063 - DEBUG - Image has been downloaded to /contrail-test/images/ubuntu-traffic.vmdk.gz 2025-07-31 02:42:50,065 - DEBUG - Unzip image 2025-07-31 02:42:50,065 - DEBUG - Image download (local=True) (proxy=None): Execute cmd: gunzip -f /contrail-test/images/ubuntu-traffic.vmdk.gz 2025-07-31 02:43:10,268 - DEBUG - Try to create image in user context 2025-07-31 02:43:21,042 - DEBUG - Image has been created in user context 2025-07-31 02:43:21,043 - DEBUG - Image download (local=True) (proxy=None): Execute cmd: rm -f /contrail-test/images/ubuntu-traffic.vmdk 2025-07-31 02:43:21,297 - DEBUG - Downloaded image has been removed 2025-07-31 02:43:21,345 - DEBUG - Uploaded image status = active 2025-07-31 02:43:22,804 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3732-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3732-1) 2025-07-31 02:43:24,484 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3732-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3732-2) 2025-07-31 02:43:24,484 - INFO - Waiting for VM ctest-TestDSNAT-89822341-34584198 to be up.. 2025-07-31 02:43:24,627 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-07-31 02:43:29,851 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-07-31 02:43:35,043 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-07-31 02:43:40,152 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-07-31 02:43:45,260 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-07-31 02:43:50,394 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-07-31 02:43:55,582 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-07-31 02:44:00,680 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-07-31 02:44:05,875 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-07-31 02:44:11,051 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-07-31 02:44:16,234 - DEBUG - VM is in ACTIVE state now 2025-07-31 02:44:16,235 - INFO - VM name : ctest-TestDSNAT-89822341-34584198 2025-07-31 02:44:16,389 - DEBUG - VM ctest-TestDSNAT-89822341-34584198 ID is 9bb9fef7-d015-4e9f-afc4-b3fa3169e8d5 2025-07-31 02:44:16,426 - DEBUG - VM ctest-TestDSNAT-89822341-34584198 launched on Node an-jenkins-deploy-platform-ansible-os-3732-1 2025-07-31 02:44:16,597 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine/9bb9fef7-d015-4e9f-afc4-b3fa3169e8d5 2025-07-31 02:44:16,997 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine/9bb9fef7-d015-4e9f-afc4-b3fa3169e8d5 2025-07-31 02:44:17,082 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine-interface/49fcf92b-495f-4d97-934b-20228a554bee 2025-07-31 02:44:20,327 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1004ms') 2025-07-31 02:44:20,327 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestDSNAT-89822341-34584198 failed! 2025-07-31 02:44:20,365 - DEBUG - Gateway for vn default-domain:ctest-TestDSNAT-89822341:ctest-dsnat_vn-50910904 is 125.48.232.65 and allocation pool is NOT set 2025-07-31 02:44:24,453 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1026ms') 2025-07-31 02:44:24,454 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestDSNAT-89822341-34584198 failed! 2025-07-31 02:44:24,474 - DEBUG - Gateway for vn default-domain:ctest-TestDSNAT-89822341:ctest-dsnat_vn-50910904 is 125.48.232.65 and allocation pool is NOT set 2025-07-31 02:44:28,550 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-07-31 02:44:28,551 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestDSNAT-89822341-34584198 failed! 2025-07-31 02:44:28,578 - DEBUG - Gateway for vn default-domain:ctest-TestDSNAT-89822341:ctest-dsnat_vn-50910904 is 125.48.232.65 and allocation pool is NOT set 2025-07-31 02:44:32,657 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms') 2025-07-31 02:44:32,657 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestDSNAT-89822341-34584198 failed! 2025-07-31 02:44:32,692 - DEBUG - Gateway for vn default-domain:ctest-TestDSNAT-89822341:ctest-dsnat_vn-50910904 is 125.48.232.65 and allocation pool is NOT set 2025-07-31 02:44:36,784 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1026ms') 2025-07-31 02:44:36,785 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestDSNAT-89822341-34584198 failed! 2025-07-31 02:44:36,811 - DEBUG - Gateway for vn default-domain:ctest-TestDSNAT-89822341:ctest-dsnat_vn-50910904 is 125.48.232.65 and allocation pool is NOT set 2025-07-31 02:44:40,905 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1030ms') 2025-07-31 02:44:40,905 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestDSNAT-89822341-34584198 failed! 2025-07-31 02:44:40,930 - DEBUG - Gateway for vn default-domain:ctest-TestDSNAT-89822341:ctest-dsnat_vn-50910904 is 125.48.232.65 and allocation pool is NOT set 2025-07-31 02:44:45,001 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2025-07-31 02:44:45,002 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestDSNAT-89822341-34584198 failed! 2025-07-31 02:44:45,021 - DEBUG - Gateway for vn default-domain:ctest-TestDSNAT-89822341:ctest-dsnat_vn-50910904 is 125.48.232.65 and allocation pool is NOT set 2025-07-31 02:44:49,099 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1005ms') 2025-07-31 02:44:49,100 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestDSNAT-89822341-34584198 failed! 2025-07-31 02:44:49,124 - DEBUG - Gateway for vn default-domain:ctest-TestDSNAT-89822341:ctest-dsnat_vn-50910904 is 125.48.232.65 and allocation pool is NOT set 2025-07-31 02:44:53,228 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=8.23 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 1 received, 50% packet loss, time 1029ms\r\nrtt min/avg/max/mdev = 8.229/8.229/8.229/0.000 ms') 2025-07-31 02:44:53,229 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-TestDSNAT-89822341-34584198 passed 2025-07-31 02:44:53,360 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 02:44:53,360 - DEBUG - Waiting to SSH to VM ctest-TestDSNAT-89822341-34584198, IP 125.48.232.67, Port 22 2025-07-31 02:44:53,433 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-31 02:44:53,630 - DEBUG - VM ctest-TestDSNAT-89822341-34584198 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-31 02:44:58,631 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 02:44:58,631 - DEBUG - Waiting to SSH to VM ctest-TestDSNAT-89822341-34584198, IP 125.48.232.67, Port 22 2025-07-31 02:44:58,700 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-31 02:44:58,822 - DEBUG - VM ctest-TestDSNAT-89822341-34584198 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-31 02:45:03,822 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 02:45:03,823 - DEBUG - Waiting to SSH to VM ctest-TestDSNAT-89822341-34584198, IP 125.48.232.67, Port 22 2025-07-31 02:45:03,896 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-31 02:45:04,086 - DEBUG - VM ctest-TestDSNAT-89822341-34584198 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-31 02:45:09,087 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 02:45:09,088 - DEBUG - Waiting to SSH to VM ctest-TestDSNAT-89822341-34584198, IP 125.48.232.67, Port 22 2025-07-31 02:45:09,159 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-31 02:45:09,356 - DEBUG - VM ctest-TestDSNAT-89822341-34584198 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-31 02:45:14,357 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 02:45:14,357 - DEBUG - Waiting to SSH to VM ctest-TestDSNAT-89822341-34584198, IP 125.48.232.67, Port 22 2025-07-31 02:45:14,535 - DEBUG - VM ctest-TestDSNAT-89822341-34584198 is ready for SSH connections 2025-07-31 02:45:14,535 - INFO - Waiting for VM ctest-TestDSNAT-89822341-78011642 to be up.. 2025-07-31 02:45:14,691 - DEBUG - VM is in ACTIVE state now 2025-07-31 02:45:14,691 - INFO - VM name : ctest-TestDSNAT-89822341-78011642 2025-07-31 02:45:14,788 - DEBUG - VM ctest-TestDSNAT-89822341-78011642 ID is 3cb8ad75-faaa-48d0-974d-3367b7ca9c9e 2025-07-31 02:45:14,788 - DEBUG - VM ctest-TestDSNAT-89822341-78011642 launched on Node an-jenkins-deploy-platform-ansible-os-3732-2 2025-07-31 02:45:14,890 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine/3cb8ad75-faaa-48d0-974d-3367b7ca9c9e 2025-07-31 02:45:14,903 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine-interface/fb303854-567a-4fa7-ae70-f82dd1e03b6e 2025-07-31 02:45:16,138 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=12.5 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=2.31 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 2.311/7.396/12.481/5.085 ms') 2025-07-31 02:45:16,138 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-TestDSNAT-89822341-78011642 passed 2025-07-31 02:45:16,266 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 02:45:16,266 - DEBUG - Waiting to SSH to VM ctest-TestDSNAT-89822341-78011642, IP 125.48.232.68, Port 22 2025-07-31 02:45:16,348 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-31 02:45:16,564 - DEBUG - VM ctest-TestDSNAT-89822341-78011642 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-31 02:45:21,566 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 02:45:21,566 - DEBUG - Waiting to SSH to VM ctest-TestDSNAT-89822341-78011642, IP 125.48.232.68, Port 22 2025-07-31 02:45:21,639 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-31 02:45:21,841 - DEBUG - VM ctest-TestDSNAT-89822341-78011642 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-31 02:45:26,842 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 02:45:26,842 - DEBUG - Waiting to SSH to VM ctest-TestDSNAT-89822341-78011642, IP 125.48.232.68, Port 22 2025-07-31 02:45:27,010 - DEBUG - VM ctest-TestDSNAT-89822341-78011642 is ready for SSH connections 2025-07-31 02:45:27,180 - DEBUG - Fetched VN: default-domain:default-project:ip-fabric(244e7543-3449-460b-b952-aa52d704ea86) with subnets None 2025-07-31 02:45:27,234 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = any, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestDSNAT-89822341:ctest-dsnat_vn-50910904, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:default-project:ip-fabric, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-07-31 02:45:27,578 - INFO - Attaching policy ctest-test-dsnat-99367505 to vn ctest-dsnat_vn-50910904 2025-07-31 02:45:28,112 - INFO - Attaching policy ctest-test-dsnat-99367505 to vn ip-fabric 2025-07-31 02:45:28,395 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 10.20.0.25, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 02:45:28,395 - DEBUG - ping -s 56 -c 3 -W 1 10.20.0.25 2025-07-31 02:46:22,068 - DEBUG - PING 10.20.0.25 (10.20.0.25) 56(84) bytes of data. --- 10.20.0.25 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2002ms 2025-07-31 02:46:22,069 - WARNING - Ping to IP 10.20.0.25 from VM ctest-TestDSNAT-89822341-34584198 failed 2025-07-31 02:46:23,070 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 10.20.0.25, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 02:46:23,071 - DEBUG - ping -s 56 -c 3 -W 1 10.20.0.25 2025-07-31 02:46:25,541 - DEBUG - PING 10.20.0.25 (10.20.0.25) 56(84) bytes of data. 64 bytes from 10.20.0.25: icmp_req=1 ttl=62 time=1.94 ms 64 bytes from 10.20.0.25: icmp_req=2 ttl=62 time=1.03 ms 64 bytes from 10.20.0.25: icmp_req=3 ttl=62 time=0.795 ms --- 10.20.0.25 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 0.795/1.259/1.949/0.498 ms 2025-07-31 02:46:25,542 - INFO - Ping to IP 10.20.0.25 from VM ctest-TestDSNAT-89822341-34584198 passed 2025-07-31 02:46:25,669 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 10.20.0.25, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 02:46:25,669 - DEBUG - ping -s 56 -c 3 -W 1 10.20.0.25 2025-07-31 02:46:28,143 - DEBUG - PING 10.20.0.25 (10.20.0.25) 56(84) bytes of data. 64 bytes from 10.20.0.25: icmp_req=1 ttl=62 time=2.59 ms 64 bytes from 10.20.0.25: icmp_req=2 ttl=62 time=0.833 ms 64 bytes from 10.20.0.25: icmp_req=3 ttl=62 time=0.997 ms --- 10.20.0.25 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2005ms rtt min/avg/max/mdev = 0.833/1.474/2.594/0.795 ms 2025-07-31 02:46:28,144 - INFO - Ping to IP 10.20.0.25 from VM ctest-TestDSNAT-89822341-34584198 passed 2025-07-31 02:46:28,144 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 02:46:28,144 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 125.48.232.68, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 02:46:28,144 - DEBUG - ping -s 56 -c 3 -W 1 125.48.232.68 2025-07-31 02:46:30,673 - DEBUG - PING 125.48.232.68 (125.48.232.68) 56(84) bytes of data. 64 bytes from 125.48.232.68: icmp_req=1 ttl=63 time=8.03 ms 64 bytes from 125.48.232.68: icmp_req=2 ttl=63 time=1.25 ms 64 bytes from 125.48.232.68: icmp_req=3 ttl=63 time=1.14 ms --- 125.48.232.68 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.147/3.479/8.032/3.219 ms 2025-07-31 02:46:30,673 - INFO - Ping to IP 125.48.232.68 from VM ctest-TestDSNAT-89822341-34584198 passed 2025-07-31 02:46:30,674 - INFO - Dettaching policy ctest-test-dsnat-99367505 from vn ip-fabric 2025-07-31 02:46:30,941 - INFO - Dettaching policy ctest-test-dsnat-99367505 from vn ctest-dsnat_vn-50910904 2025-07-31 02:46:31,215 - INFO - Deleted policy ctest-test-dsnat-99367505 2025-07-31 02:46:31,216 - INFO - Deleting VM ctest-TestDSNAT-89822341-78011642 2025-07-31 02:46:31,340 - INFO - Deleting VM ctest-TestDSNAT-89822341-34584198 2025-07-31 02:46:31,450 - INFO - Deleting VN ctest-dsnat_vn-50910904 2025-07-31 02:46:31,494 - DEBUG - VN 39c57d50-64ac-4e6b-946a-49565fdeeb3e still in use: Unable to complete operation on network 39c57d50-64ac-4e6b-946a-49565fdeeb3e. There are one or more ports still in use on the network. Neutron server returns request_ids: ['req-213da0aa-b6e7-4b04-b5a9-042354d3769e'] 2025-07-31 02:46:31,494 - WARNING - Deleting VN ctest-dsnat_vn-50910904 failed..Will retry 2025-07-31 02:46:33,759 - DEBUG - Response for deleting network () 2025-07-31 02:46:35,085 - DEBUG - No XMPP flaps were noticed during the test 2025-07-31 02:46:35,086 - INFO - END TEST : test_dsnat_bug_1749695 : PASSED[0:03:53] 2025-07-31 02:46:35,086 - INFO - -------------------------------------------------------------------------------- 2025-07-31 02:46:35,976 - INFO - Deleted project: ctest-TestDSNAT-89822341, ID : 43e31bc7-88c7-40c9-8ed8-da5fdaa0784b 2025-07-31 05:25:05,675 - INFO - Domain Default found not creating 2025-07-31 05:25:05,963 - INFO - Project ctest-TestDSNAT-37400992 not found, creating it 2025-07-31 05:25:06,633 - INFO - Created Project:ctest-TestDSNAT-37400992, ID : 51e0f304-2fb4-4d09-a6a8-5c8fb8b41f5d 2025-07-31 05:25:08,339 - DEBUG - Services list from nova: [, , , ] 2025-07-31 05:25:08,339 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3732-1', 'an-jenkins-deploy-platform-ansible-os-3732-2']} 2025-07-31 05:25:08,339 - INFO - ================================================================================ 2025-07-31 05:25:08,339 - INFO - STARTING TEST : test_dsnat_tcp_basic 2025-07-31 05:25:08,339 - INFO - TEST DESCRIPTION : Configure TCP port translation pool send TCP traffic from the VM to the fabric IP Verify the Port address translation happened and the port being used in the range of configured 2025-07-31 05:25:09,677 - DEBUG - Skipping xmpp flap check 2025-07-31 05:25:09,677 - INFO - Initial checks done. Running the testcase now 2025-07-31 05:25:09,677 - INFO - 2025-07-31 05:25:10,173 - INFO - Create VN, enable FABRIC SNAT and verify its routing instance 2025-07-31 05:25:10,729 - DEBUG - Response for create_network : {'network': {'id': 'e0195253-949f-4c37-8a5a-5b44e5d5d97d', 'name': 'ctest-dsnat_vn-36282497', 'tenant_id': '51e0f3042fb44d09a6a85c8fb8b41f5d', 'project_id': '51e0f3042fb44d09a6a85c8fb8b41f5d', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestDSNAT-37400992', 'ctest-dsnat_vn-36282497'], 'port_security_enabled': True, 'description': ''}} 2025-07-31 05:25:11,353 - DEBUG - Response for create_subnet : {'subnet': {'id': '72b12166-88c6-4f43-b1bc-64e9a5feb958', 'name': '', 'tenant_id': '51e0f3042fb44d09a6a85c8fb8b41f5d', 'network_id': 'e0195253-949f-4c37-8a5a-5b44e5d5d97d', 'ip_version': 4, 'cidr': '66.96.95.64/26', 'allocation_pools': [{'start': '66.96.95.66', 'end': '66.96.95.126'}], 'gateway_ip': '66.96.95.65', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '66.96.95.66', 'tags': [], 'project_id': '51e0f3042fb44d09a6a85c8fb8b41f5d'}} 2025-07-31 05:25:11,384 - INFO - Created VN ctest-dsnat_vn-36282497 2025-07-31 05:25:11,399 - DEBUG - VN ctest-dsnat_vn-36282497 UUID is e0195253-949f-4c37-8a5a-5b44e5d5d97d 2025-07-31 05:25:11,649 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/e0195253-949f-4c37-8a5a-5b44e5d5d97d 2025-07-31 05:25:11,977 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/e0195253-949f-4c37-8a5a-5b44e5d5d97d 2025-07-31 05:25:12,023 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/24538e50-24e2-472c-831f-205e45c9b2c7 2025-07-31 05:25:13,428 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3732-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3732-1) 2025-07-31 05:25:14,895 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3732-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3732-2) 2025-07-31 05:25:14,895 - INFO - Waiting for VM ctest-dsnat-vm1-24169334 to be up.. 2025-07-31 05:25:15,000 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-07-31 05:25:20,158 - DEBUG - VM is in ACTIVE state now 2025-07-31 05:25:20,158 - INFO - VM name : ctest-dsnat-vm1-24169334 2025-07-31 05:25:20,305 - DEBUG - VM ctest-dsnat-vm1-24169334 ID is 7c97b105-804c-4d9d-83d1-8f2737b75fae 2025-07-31 05:25:20,349 - DEBUG - VM ctest-dsnat-vm1-24169334 launched on Node an-jenkins-deploy-platform-ansible-os-3732-1 2025-07-31 05:25:20,496 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine/7c97b105-804c-4d9d-83d1-8f2737b75fae 2025-07-31 05:25:20,905 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine/7c97b105-804c-4d9d-83d1-8f2737b75fae 2025-07-31 05:25:20,973 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine-interface/d8e7a3d8-1dd8-49aa-85d8-1eba1b64ed8d 2025-07-31 05:25:24,202 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1016ms') 2025-07-31 05:25:24,202 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-dsnat-vm1-24169334 failed! 2025-07-31 05:25:24,231 - DEBUG - Gateway for vn default-domain:ctest-TestDSNAT-37400992:ctest-dsnat_vn-36282497 is 66.96.95.65 and allocation pool is NOT set 2025-07-31 05:25:28,333 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1034ms') 2025-07-31 05:25:28,333 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-dsnat-vm1-24169334 failed! 2025-07-31 05:25:28,361 - DEBUG - Gateway for vn default-domain:ctest-TestDSNAT-37400992:ctest-dsnat_vn-36282497 is 66.96.95.65 and allocation pool is NOT set 2025-07-31 05:25:32,461 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1030ms') 2025-07-31 05:25:32,462 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-dsnat-vm1-24169334 failed! 2025-07-31 05:25:32,488 - DEBUG - Gateway for vn default-domain:ctest-TestDSNAT-37400992:ctest-dsnat_vn-36282497 is 66.96.95.65 and allocation pool is NOT set 2025-07-31 05:25:36,587 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1028ms') 2025-07-31 05:25:36,588 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-dsnat-vm1-24169334 failed! 2025-07-31 05:25:36,614 - DEBUG - Gateway for vn default-domain:ctest-TestDSNAT-37400992:ctest-dsnat_vn-36282497 is 66.96.95.65 and allocation pool is NOT set 2025-07-31 05:25:40,718 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1034ms') 2025-07-31 05:25:40,718 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-dsnat-vm1-24169334 failed! 2025-07-31 05:25:40,738 - DEBUG - Gateway for vn default-domain:ctest-TestDSNAT-37400992:ctest-dsnat_vn-36282497 is 66.96.95.65 and allocation pool is NOT set 2025-07-31 05:25:44,811 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1006ms') 2025-07-31 05:25:44,811 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-dsnat-vm1-24169334 failed! 2025-07-31 05:25:44,829 - DEBUG - Gateway for vn default-domain:ctest-TestDSNAT-37400992:ctest-dsnat_vn-36282497 is 66.96.95.65 and allocation pool is NOT set 2025-07-31 05:25:48,903 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2025-07-31 05:25:48,904 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-dsnat-vm1-24169334 failed! 2025-07-31 05:25:48,932 - DEBUG - Gateway for vn default-domain:ctest-TestDSNAT-37400992:ctest-dsnat_vn-36282497 is 66.96.95.65 and allocation pool is NOT set 2025-07-31 05:25:52,998 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1006ms') 2025-07-31 05:25:52,998 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-dsnat-vm1-24169334 failed! 2025-07-31 05:25:53,030 - DEBUG - Gateway for vn default-domain:ctest-TestDSNAT-37400992:ctest-dsnat_vn-36282497 is 66.96.95.65 and allocation pool is NOT set 2025-07-31 05:25:55,103 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=8.66 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=5.64 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 5.642/7.150/8.658/1.508 ms') 2025-07-31 05:25:55,104 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-dsnat-vm1-24169334 passed 2025-07-31 05:25:55,204 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 05:25:55,204 - DEBUG - Waiting to SSH to VM ctest-dsnat-vm1-24169334, IP 66.96.95.67, Port 22 2025-07-31 05:25:55,275 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-31 05:25:55,467 - DEBUG - VM ctest-dsnat-vm1-24169334 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-31 05:26:00,468 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 05:26:00,469 - DEBUG - Waiting to SSH to VM ctest-dsnat-vm1-24169334, IP 66.96.95.67, Port 22 2025-07-31 05:26:00,538 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-31 05:26:00,656 - DEBUG - VM ctest-dsnat-vm1-24169334 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-31 05:26:05,657 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 05:26:05,658 - DEBUG - Waiting to SSH to VM ctest-dsnat-vm1-24169334, IP 66.96.95.67, Port 22 2025-07-31 05:26:05,736 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-31 05:26:05,856 - DEBUG - VM ctest-dsnat-vm1-24169334 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-31 05:26:10,857 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 05:26:10,857 - DEBUG - Waiting to SSH to VM ctest-dsnat-vm1-24169334, IP 66.96.95.67, Port 22 2025-07-31 05:26:10,930 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-31 05:26:11,047 - DEBUG - VM ctest-dsnat-vm1-24169334 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-31 05:26:16,048 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 05:26:16,048 - DEBUG - Waiting to SSH to VM ctest-dsnat-vm1-24169334, IP 66.96.95.67, Port 22 2025-07-31 05:26:16,227 - DEBUG - VM ctest-dsnat-vm1-24169334 is ready for SSH connections 2025-07-31 05:26:16,227 - INFO - Waiting for VM ctest-dsnat-vm2-05303020 to be up.. 2025-07-31 05:26:16,415 - DEBUG - VM is in ACTIVE state now 2025-07-31 05:26:16,416 - INFO - VM name : ctest-dsnat-vm2-05303020 2025-07-31 05:26:16,574 - DEBUG - VM ctest-dsnat-vm2-05303020 ID is 4de69f0a-5c66-4c23-b491-281cd580dbe6 2025-07-31 05:26:16,575 - DEBUG - VM ctest-dsnat-vm2-05303020 launched on Node an-jenkins-deploy-platform-ansible-os-3732-2 2025-07-31 05:26:16,737 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine/4de69f0a-5c66-4c23-b491-281cd580dbe6 2025-07-31 05:26:16,758 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine-interface/32c3819e-5bef-4c21-96e5-c4e3aa68a7c6 2025-07-31 05:26:18,028 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=3.60 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=4.14 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 3.598/3.867/4.136/0.269 ms') 2025-07-31 05:26:18,029 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-dsnat-vm2-05303020 passed 2025-07-31 05:26:18,111 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 05:26:18,112 - DEBUG - Waiting to SSH to VM ctest-dsnat-vm2-05303020, IP 66.96.95.68, Port 22 2025-07-31 05:26:18,180 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-31 05:26:18,309 - DEBUG - VM ctest-dsnat-vm2-05303020 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-31 05:26:23,309 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 05:26:23,310 - DEBUG - Waiting to SSH to VM ctest-dsnat-vm2-05303020, IP 66.96.95.68, Port 22 2025-07-31 05:26:23,381 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-31 05:26:23,583 - DEBUG - VM ctest-dsnat-vm2-05303020 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-31 05:26:28,584 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 05:26:28,584 - DEBUG - Waiting to SSH to VM ctest-dsnat-vm2-05303020, IP 66.96.95.68, Port 22 2025-07-31 05:26:28,764 - DEBUG - VM ctest-dsnat-vm2-05303020 is ready for SSH connections 2025-07-31 05:26:28,921 - DEBUG - command output is cat: /tmp/iperf_server_ctest-random-09479786.pid: No such file or directory kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2025-07-31 05:26:29,078 - DEBUG - command output is 2025-07-31 05:26:29,078 - INFO - Starting Iperf3 on ctest-dsnat-vm1-24169334, args: --port 6201 --length 1048576 --time 10 2025-07-31 05:26:29,078 - DEBUG - Iperf3 cmd : iperf3 -c 10.20.0.201 --port 6201 --length 1048576 --time 10 1>/tmp/iper3_ctest-random-09479786.log 2>/tmp/iper3_ctest-random-09479786.result 2025-07-31 05:26:29,078 - INFO - Excecuting cmds, ['iperf3 -s --port 6201 1>/tmp/iper3_ctest-random-09479786.log 2>/tmp/iper3_ctest-random-09479786.result'], on server 10.0.0.57 2025-07-31 05:26:29,199 - DEBUG - command output is [''] 2025-07-31 05:26:29,199 - DEBUG - Running remote_cmd, Cmd : iperf3 -c 10.20.0.201 --port 6201 --length 1048576 --time 10 1>/tmp/iper3_ctest-random-09479786.log 2>/tmp/iper3_ctest-random-09479786.result, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:26:29,199 - DEBUG - nohup iperf3 -c 10.20.0.201 --port 6201 --length 1048576 --time 10 1>/tmp/iper3_ctest-random-09479786.log 2>/tmp/iper3_ctest-random-09479786.result & echo $! > /tmp/iperf_client_ctest-random-09479786.pid 2025-07-31 05:26:47,308 - ERROR - Exception occured while running cmds ['iperf3 -c 10.20.0.201 --port 6201 --length 1048576 --time 10 1>/tmp/iper3_ctest-random-09479786.log 2>/tmp/iper3_ctest-random-09479786.result'] 2025-07-31 05:26:47,309 - ERROR - Timeout opening channel. Traceback (most recent call last): File "/contrail-test/tcutils/fabutils.py", line 169, in remote_cmd output = _run(cmd, timeout=timeout, pty=not as_daemon, shell=shell) File "/usr/local/lib/python3.6/site-packages/fabric/network.py", line 692, in host_prompting_wrapper return func(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/fabric/operations.py", line 1153, in sudo capture_buffer_size=capture_buffer_size, File "/usr/local/lib/python3.6/site-packages/fabric/operations.py", line 935, in _run_command channel=default_channel(), command=wrapped_command, pty=pty, File "/usr/local/lib/python3.6/site-packages/fabric/state.py", line 435, in default_channel chan = _open_session() File "/usr/local/lib/python3.6/site-packages/fabric/state.py", line 420, in _open_session session = transport.open_session(timeout=env.timeout) File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 923, in open_session timeout=timeout, File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 1050, in open_channel raise e File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 2109, in run ptype, m = self.packetizer.read_message() File "/usr/local/lib/python3.6/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/usr/local/lib/python3.6/site-packages/paramiko/packet.py", line 303, in read_all raise EOFError() EOFError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/fabric/state.py", line 435, in default_channel chan = _open_session() File "/usr/local/lib/python3.6/site-packages/fabric/state.py", line 420, in _open_session session = transport.open_session(timeout=env.timeout) File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 923, in open_session timeout=timeout, File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 1013, in open_channel raise SSHException("SSH session not active") paramiko.ssh_exception.SSHException: SSH session not active During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/contrail-test/fixtures/vm_test.py", line 2287, in run_cmd_on_vm logger=self.logger File "/contrail-test/tcutils/fabutils.py", line 178, in remote_cmd output = _run(cmd, timeout=timeout, pty=not as_daemon, shell=shell) File "/usr/local/lib/python3.6/site-packages/fabric/network.py", line 692, in host_prompting_wrapper return func(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/fabric/operations.py", line 1153, in sudo capture_buffer_size=capture_buffer_size, File "/usr/local/lib/python3.6/site-packages/fabric/operations.py", line 935, in _run_command channel=default_channel(), command=wrapped_command, pty=pty, File "/usr/local/lib/python3.6/site-packages/fabric/state.py", line 440, in default_channel chan = _open_session() File "/usr/local/lib/python3.6/site-packages/fabric/state.py", line 420, in _open_session session = transport.open_session(timeout=env.timeout) File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 923, in open_session timeout=timeout, File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 1054, in open_channel raise SSHException("Timeout opening channel.") paramiko.ssh_exception.SSHException: Timeout opening channel. 2025-07-31 05:26:51,058 - DEBUG - command output is /bin/bash: line 1: kill: (64465) - No such process 2025-07-31 05:26:51,173 - DEBUG - command output is 2025-07-31 05:26:51,173 - INFO - Deleting VM ctest-dsnat-vm2-05303020 2025-07-31 05:26:51,347 - INFO - Deleting VM ctest-dsnat-vm1-24169334 2025-07-31 05:26:51,525 - INFO - Deleting VN ctest-dsnat_vn-36282497 2025-07-31 05:26:51,619 - DEBUG - VN e0195253-949f-4c37-8a5a-5b44e5d5d97d still in use: Unable to complete operation on network e0195253-949f-4c37-8a5a-5b44e5d5d97d. There are one or more ports still in use on the network. Neutron server returns request_ids: ['req-0ab41501-a376-406a-9097-c4e887c8cbaf'] 2025-07-31 05:26:51,619 - WARNING - Deleting VN ctest-dsnat_vn-36282497 failed..Will retry 2025-07-31 05:26:53,816 - DEBUG - Response for deleting network () 2025-07-31 05:26:55,118 - ERROR - AttributeError Python 3.6.8: /usr/bin/python3 Thu Jul 31 05:26:50 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={}) 78 log.info('Initial checks done. Running the testcase now') 79 log.info('') 80 result = function(self, *args, **kwargs) 81 if self.inputs.upgrade: 82 pid = os.getpid() result = None function = self = args = () kwargs = {} /contrail-test/serial_scripts/dsnat/test_dsnat.py in test_dsnat_tcp_basic(self=) 108 self.iperf = None 109 110 assert self.verify_flow_with_port(vm1_fixture, vm2_fixture, port_range, **traffic) 111 112 @preposttest_wrapper self = self.verify_flow_with_port = .deco_retry.....test_dsnat_tcp_basic[sanity] id=0x7f4c7addb908>> vm1_fixture = vm2_fixture = port_range = [65000, 65001, 65002, 65003, 65004, 65005, 65006, 65007, 65008, 65009] traffic = {'port': 6201, 'udp': False} /contrail-test/tcutils/util.py in f_retry(*args=(, , , [65000, 65001, 65002, 65003, 65004, 65005, 65006, 65007, 65008, 65009]), **kwargs={'port': 6201, 'udp': False}) 82 mtries, mdelay = tries, delay # make mutable 83 84 result = f(*args, **kwargs) # first attempt 85 rv = result 86 final = False result undefined f = args = (, , , [65000, 65001, 65002, 65003, 65004, 65005, 65006, 65007, 65008, 65009]) kwargs = {'port': 6201, 'udp': False} /contrail-test/common/dsnat/base.py in verify_flow_with_port(self=, client_vm_fix=, dst_vm_fix=, port_range=[65000, 65001, 65002, 65003, 65004, 65005, 65006, 65007, 65008, 65009], **traffic={'port': 6201, 'udp': False}) 200 else: 201 proto = '6' 202 nat_port_used = self.get_nat_port_used_for_flow(client_vm_fix, proto, traffic['port']) 203 self.iperf.stop_iperf_on_server() 204 self.logger.info("Nat port being used for the flow is %s" %nat_port_used) nat_port_used undefined self = self.get_nat_port_used_for_flow = > client_vm_fix = proto = '6' traffic = {'port': 6201, 'udp': False} /contrail-test/common/dsnat/base.py in get_nat_port_used_for_flow(self=, vm_fix=, proto='6', port=6201, container='agent') 185 self.connections, vm_fix.vm_node_ip)) 186 flow_entry = server_fixture.get_flow_entry(dest_ip=vm_fix.vm_node_data_ip, 187 proto=proto, source_port = port,all_flows=True) 188 nat_port = [] 189 for flow in flow_entry: proto = '6' source_port undefined port = 6201 all_flows undefined /contrail-test/fixtures/compute_node_test.py in get_flow_entry(self=, flow_table=, index=None, source_ip=None, dest_ip='10.20.0.192', source_port=6201, dest_port=None, proto='6', vrf_id=None, refresh=True, show_evicted=True, all_flows=True) 561 all_flow_list = [] 562 for flow_entry_item in flow_table.items: 563 if reqd_entries.viewitems() <= flow_entry_item.viewitems(): 564 forward_flow = FlowEntry(flow_entry_item) 565 if flow_entry_item['rflow'] != '-1': reqd_entries = {'dip': '10.20.0.192', 'proto': '6', 'sport': '6201'} reqd_entries.viewitems undefined flow_entry_item = {'action': 'DROP', 'bytes': '400', 'd_vrf_id': '3', 'dip': '169.254.0.4', 'dport': '50195', 'drop_reason': 'Unknown', 'flags': ' ACTIVE | EVICTED ', 'gen_id': '1', 'index': '48', 'insight': '0', ...} flow_entry_item.viewitems undefined AttributeError: 'dict' object has no attribute 'viewitems' __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {} __dir__ = __doc__ = 'Attribute not found.' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = args = ("'dict' object has no attribute 'viewitems'",) with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper result = function(self, *args, **kwargs) File "/contrail-test/serial_scripts/dsnat/test_dsnat.py", line 110, in test_dsnat_tcp_basic assert self.verify_flow_with_port(vm1_fixture, vm2_fixture, port_range, **traffic) File "/contrail-test/tcutils/util.py", line 84, in f_retry result = f(*args, **kwargs) # first attempt File "/contrail-test/common/dsnat/base.py", line 202, in verify_flow_with_port nat_port_used = self.get_nat_port_used_for_flow(client_vm_fix, proto, traffic['port']) File "/contrail-test/common/dsnat/base.py", line 187, in get_nat_port_used_for_flow proto=proto, source_port = port,all_flows=True) File "/contrail-test/fixtures/compute_node_test.py", line 563, in get_flow_entry if reqd_entries.viewitems() <= flow_entry_item.viewitems(): AttributeError: 'dict' object has no attribute 'viewitems' 2025-07-31 05:26:55,122 - DEBUG - Skipping xmpp flap check 2025-07-31 05:26:55,122 - INFO - 2025-07-31 05:26:55,122 - INFO - END TEST : test_dsnat_tcp_basic : FAILED[0:01:47] 2025-07-31 05:26:55,123 - INFO - -------------------------------------------------------------------------------- 2025-07-31 05:26:55,931 - INFO - Deleted project: ctest-TestDSNAT-37400992, ID : 51e0f304-2fb4-4d09-a6a8-5c8fb8b41f5d