2025-07-31 04:54:47,769 - INFO - Domain Default found not creating 2025-07-31 04:54:47,950 - INFO - Project ctest-TestMtuProbe-15541328 not found, creating it 2025-07-31 04:54:48,668 - INFO - Created Project:ctest-TestMtuProbe-15541328, ID : e8822bde-b9be-4614-8a35-0e1e4b75b1f1 2025-07-31 04:54:50,470 - INFO - ================================================================================ 2025-07-31 04:54:50,471 - INFO - STARTING TEST : test_mtu_probe_inter_node 2025-07-31 04:54:50,471 - INFO - TEST DESCRIPTION : Description: check traffic configuring different tcp_mtu_probe vlaues on vm MTU probing values: 0, 1, 2 0 - Disabled (default) 1 - Disabled by default, Enabled only when ICMP black hole detected 2 - Always enabled, using the initial MSS value of tcp_base_mss system internally enables bit values for point 1&2 For test purpose, Idea is to check traffic works fine on configuring values Vm's across compute node steps: 1. create 1 VN and launch 2 VMs across compute node 2. client and server VMs across compute node 3. configure tcp_mtu_probe to 0, 1, 2 4. send icmp traffic assert if ping fails 5. send tcp traffic assert if tcp traffic fails Pass criteria: 1. If icmp/tcp traffic work fine configuring tcp_mtu_probe values 2025-07-31 04:54:51,891 - DEBUG - Nothing to compare xmpp stats {'10.0.0.27': {'10.20.0.193': '0', '10.20.0.18': '0'}, '10.0.0.57': {'10.20.0.18': '0', '10.20.0.193': '0'}} with 2025-07-31 04:54:51,891 - INFO - Initial checks done. Running the testcase now 2025-07-31 04:54:51,891 - INFO - 2025-07-31 04:54:51,922 - DEBUG - Services list from nova: [, , , ] 2025-07-31 04:54:51,923 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3732-1', 'an-jenkins-deploy-platform-ansible-os-3732-2']} 2025-07-31 04:54:52,855 - DEBUG - Response for create_network : {'network': {'id': '57f03fbd-2014-4b85-acb1-0d3ccb0e6724', 'name': 'ctest-vn-93417578', 'tenant_id': 'e8822bdeb9be46148a350e1e4b75b1f1', 'project_id': 'e8822bdeb9be46148a350e1e4b75b1f1', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestMtuProbe-15541328', 'ctest-vn-93417578'], 'port_security_enabled': True, 'description': ''}} 2025-07-31 04:54:53,128 - DEBUG - Response for create_subnet : {'subnet': {'id': '3bec82a0-bff0-4700-8e34-a34725e4c65e', 'name': '', 'tenant_id': 'e8822bdeb9be46148a350e1e4b75b1f1', 'network_id': '57f03fbd-2014-4b85-acb1-0d3ccb0e6724', 'ip_version': 4, 'cidr': '161.94.71.0/26', 'allocation_pools': [{'start': '161.94.71.2', 'end': '161.94.71.62'}], 'gateway_ip': '161.94.71.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '161.94.71.2', 'tags': [], 'project_id': 'e8822bdeb9be46148a350e1e4b75b1f1'}} 2025-07-31 04:54:53,153 - INFO - Created VN ctest-vn-93417578 2025-07-31 04:54:53,167 - DEBUG - VN ctest-vn-93417578 UUID is 57f03fbd-2014-4b85-acb1-0d3ccb0e6724 2025-07-31 04:54:53,252 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:54:53,587 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:54:53,640 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:54:53,683 - DEBUG - Requesting: http://10.0.0.41:8082/project/e8822bde-b9be-4614-8a35-0e1e4b75b1f1 2025-07-31 04:54:53,815 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/57f03fbd-2014-4b85-acb1-0d3ccb0e6724 2025-07-31 04:54:53,841 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/57f03fbd-2014-4b85-acb1-0d3ccb0e6724 2025-07-31 04:54:53,859 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/3a818f26-b148-455f-8164-2a9ad49a209f 2025-07-31 04:54:53,876 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/3a818f26-b148-455f-8164-2a9ad49a209f 2025-07-31 04:54:53,891 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/3ef1e4e9-8111-4c1b-aaf2-494466572728 2025-07-31 04:54:53,902 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-31 04:54:53,903 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/57f03fbd-2014-4b85-acb1-0d3ccb0e6724 2025-07-31 04:54:53,921 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/3a818f26-b148-455f-8164-2a9ad49a209f 2025-07-31 04:54:53,944 - INFO - Verified VN network id 11 for VN 57f03fbd-2014-4b85-acb1-0d3ccb0e6724 2025-07-31 04:54:53,944 - INFO - Verifications in API Server for VN ctest-vn-93417578 passed 2025-07-31 04:54:53,944 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/57f03fbd-2014-4b85-acb1-0d3ccb0e6724 2025-07-31 04:54:53,961 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/3a818f26-b148-455f-8164-2a9ad49a209f 2025-07-31 04:54:53,977 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/3a818f26-b148-455f-8164-2a9ad49a209f 2025-07-31 04:54:53,990 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/3ef1e4e9-8111-4c1b-aaf2-494466572728 2025-07-31 04:54:54,013 - DEBUG - Control-node 10.0.0.41 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestMtuProbe-15541328', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '6336634757229857669', 'uuid-lslong': '12443741800157308708'}, 'enable': 'true', 'created': '2025-07-31T04:54:52', 'last-modified': '2025-07-31T04:54:53', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.950413'} 2025-07-31 04:54:54,021 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-31 04:54:54,030 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestMtuProbe-15541328', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '6336634757229857669', 'uuid-lslong': '12443741800157308708'}, 'enable': 'true', 'created': '2025-07-31T04:54:52', 'last-modified': '2025-07-31T04:54:53', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.966223'} 2025-07-31 04:54:54,038 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-31 04:54:54,048 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestMtuProbe-15541328', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '6336634757229857669', 'uuid-lslong': '12443741800157308708'}, 'enable': 'true', 'created': '2025-07-31T04:54:52', 'last-modified': '2025-07-31T04:54:53', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.984584'} 2025-07-31 04:54:54,059 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-31 04:54:54,059 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn-93417578 passed 2025-07-31 04:54:54,060 - DEBUG - ====Verifying policy data for ctest-vn-93417578 in API_Server ====== 2025-07-31 04:54:54,060 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:54:54,075 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:54:54,103 - DEBUG - Requesting: http://10.0.0.41:8082/project/e8822bde-b9be-4614-8a35-0e1e4b75b1f1 2025-07-31 04:54:54,250 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/57f03fbd-2014-4b85-acb1-0d3ccb0e6724 2025-07-31 04:54:54,267 - DEBUG - =>VN ctest-vn-93417578 has no policy to be verified 2025-07-31 04:54:54,267 - DEBUG - Verifying the vn in opserver 2025-07-31 04:54:54,268 - DEBUG - Verifying the default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 virtual network link through opserver 10.0.0.41 2025-07-31 04:54:54,268 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks 2025-07-31 04:54:54,593 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks 2025-07-31 04:54:54,664 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-network/default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578?flat'} 2025-07-31 04:54:54,664 - DEBUG - Verifying the default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 virtual network link through opserver 10.0.0.38 2025-07-31 04:54:54,665 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-07-31 04:54:54,963 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-07-31 04:54:55,044 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578?flat'} 2025-07-31 04:54:55,044 - DEBUG - Verifying the default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 virtual network link through opserver 10.0.0.61 2025-07-31 04:54:55,044 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-07-31 04:54:55,365 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-07-31 04:54:55,423 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578?flat'} 2025-07-31 04:54:55,423 - INFO - Validated that VN default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 is found in opserver 2025-07-31 04:54:55,458 - DEBUG - Do not have enough data to verify VN in agent 2025-07-31 04:54:55,477 - DEBUG - VRF ids for VN ctest-vn-93417578: {} 2025-07-31 04:54:55,478 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3732-1', 'an-jenkins-deploy-platform-ansible-os-3732-2']} 2025-07-31 04:54:55,478 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3732-1', 'an-jenkins-deploy-platform-ansible-os-3732-2']} 2025-07-31 04:54:56,757 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3732-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3732-1) 2025-07-31 04:54:56,757 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3732-1', 'an-jenkins-deploy-platform-ansible-os-3732-2']} 2025-07-31 04:54:58,925 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3732-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3732-2) 2025-07-31 04:54:58,925 - INFO - Waiting for VM ctest-TestMtuProbe-15541328-27761176 to be up.. 2025-07-31 04:54:59,022 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-07-31 04:55:04,179 - DEBUG - VM is in ACTIVE state now 2025-07-31 04:55:04,179 - INFO - VM name : ctest-TestMtuProbe-15541328-27761176 2025-07-31 04:55:04,312 - DEBUG - VM ctest-TestMtuProbe-15541328-27761176 ID is 044c8598-01ad-4883-afc8-6a837d0239e4 2025-07-31 04:55:04,355 - DEBUG - VM ctest-TestMtuProbe-15541328-27761176 launched on Node an-jenkins-deploy-platform-ansible-os-3732-1 2025-07-31 04:55:04,504 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine/044c8598-01ad-4883-afc8-6a837d0239e4 2025-07-31 04:55:04,817 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine/044c8598-01ad-4883-afc8-6a837d0239e4 2025-07-31 04:55:04,871 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine-interface/36fedd06-a977-44ff-83d1-bdbcf3632683 2025-07-31 04:55:08,171 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1020ms') 2025-07-31 04:55:08,172 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-27761176 failed! 2025-07-31 04:55:08,196 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 is 161.94.71.1 and allocation pool is NOT set 2025-07-31 04:55:12,301 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1034ms') 2025-07-31 04:55:12,301 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-27761176 failed! 2025-07-31 04:55:12,329 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 is 161.94.71.1 and allocation pool is NOT set 2025-07-31 04:55:16,425 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1024ms') 2025-07-31 04:55:16,426 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-27761176 failed! 2025-07-31 04:55:16,455 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 is 161.94.71.1 and allocation pool is NOT set 2025-07-31 04:55:20,556 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1024ms') 2025-07-31 04:55:20,557 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-27761176 failed! 2025-07-31 04:55:20,580 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 is 161.94.71.1 and allocation pool is NOT set 2025-07-31 04:55:24,654 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1007ms') 2025-07-31 04:55:24,654 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-27761176 failed! 2025-07-31 04:55:24,681 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 is 161.94.71.1 and allocation pool is NOT set 2025-07-31 04:55:28,784 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1035ms') 2025-07-31 04:55:28,784 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-27761176 failed! 2025-07-31 04:55:28,805 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 is 161.94.71.1 and allocation pool is NOT set 2025-07-31 04:55:32,876 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1006ms') 2025-07-31 04:55:32,876 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-27761176 failed! 2025-07-31 04:55:32,895 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 is 161.94.71.1 and allocation pool is NOT set 2025-07-31 04:55:36,973 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1009ms') 2025-07-31 04:55:36,974 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-27761176 failed! 2025-07-31 04:55:37,000 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 is 161.94.71.1 and allocation pool is NOT set 2025-07-31 04:55:41,102 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1032ms') 2025-07-31 04:55:41,102 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-27761176 failed! 2025-07-31 04:55:41,128 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 is 161.94.71.1 and allocation pool is NOT set 2025-07-31 04:55:43,196 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=7.49 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=4.34 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 4.343/5.917/7.491/1.574 ms') 2025-07-31 04:55:43,197 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-27761176 passed 2025-07-31 04:55:43,327 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 04:55:43,327 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-27761176, IP 161.94.71.3, Port 22 2025-07-31 04:55:43,390 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-31 04:55:43,567 - DEBUG - VM ctest-TestMtuProbe-15541328-27761176 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-31 04:55:48,569 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 04:55:48,569 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-27761176, IP 161.94.71.3, Port 22 2025-07-31 04:55:48,632 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-31 04:55:48,815 - DEBUG - VM ctest-TestMtuProbe-15541328-27761176 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-31 04:55:53,816 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 04:55:53,816 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-27761176, IP 161.94.71.3, Port 22 2025-07-31 04:55:53,889 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-31 04:55:54,066 - DEBUG - VM ctest-TestMtuProbe-15541328-27761176 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-31 04:55:59,067 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 04:55:59,067 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-27761176, IP 161.94.71.3, Port 22 2025-07-31 04:55:59,133 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-31 04:55:59,304 - DEBUG - VM ctest-TestMtuProbe-15541328-27761176 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-31 04:56:04,304 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 04:56:04,305 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-27761176, IP 161.94.71.3, Port 22 2025-07-31 04:56:04,463 - DEBUG - VM ctest-TestMtuProbe-15541328-27761176 is ready for SSH connections 2025-07-31 04:56:04,463 - INFO - Waiting for VM ctest-TestMtuProbe-15541328-14386499 to be up.. 2025-07-31 04:56:04,558 - DEBUG - VM is in ACTIVE state now 2025-07-31 04:56:04,558 - INFO - VM name : ctest-TestMtuProbe-15541328-14386499 2025-07-31 04:56:04,640 - DEBUG - VM ctest-TestMtuProbe-15541328-14386499 ID is 3dfb85f0-8de8-4200-a578-553d6f3575a0 2025-07-31 04:56:04,641 - DEBUG - VM ctest-TestMtuProbe-15541328-14386499 launched on Node an-jenkins-deploy-platform-ansible-os-3732-2 2025-07-31 04:56:04,733 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine/3dfb85f0-8de8-4200-a578-553d6f3575a0 2025-07-31 04:56:04,747 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine-interface/77fefad4-1a29-46fb-a7ec-019f08f22475 2025-07-31 04:56:05,964 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=3.89 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.849 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.849/2.369/3.889/1.520 ms') 2025-07-31 04:56:05,964 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-14386499 passed 2025-07-31 04:56:06,084 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 04:56:06,084 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-14386499, IP 161.94.71.4, Port 22 2025-07-31 04:56:06,156 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-31 04:56:06,259 - DEBUG - VM ctest-TestMtuProbe-15541328-14386499 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-31 04:56:11,260 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 04:56:11,261 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-14386499, IP 161.94.71.4, Port 22 2025-07-31 04:56:11,442 - DEBUG - VM ctest-TestMtuProbe-15541328-14386499 is ready for SSH connections 2025-07-31 04:56:11,442 - DEBUG - Running remote_cmd, Cmd : echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 04:56:11,443 - DEBUG - echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing 2025-07-31 04:57:00,709 - DEBUG - None 2025-07-31 04:57:00,709 - DEBUG - Running remote_cmd, Cmd : echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:57:00,709 - DEBUG - echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing 2025-07-31 04:57:46,563 - DEBUG - None 2025-07-31 04:57:46,563 - INFO - {'echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing': None} 2025-07-31 04:57:46,563 - INFO - {'echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing': None} 2025-07-31 04:57:46,564 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 04:57:46,564 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 161.94.71.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 04:57:46,565 - DEBUG - ping -s 56 -c 3 -W 1 161.94.71.4 2025-07-31 04:57:49,082 - DEBUG - PING 161.94.71.4 (161.94.71.4) 56(84) bytes of data. 64 bytes from 161.94.71.4: icmp_req=1 ttl=64 time=11.7 ms 64 bytes from 161.94.71.4: icmp_req=2 ttl=64 time=1.18 ms 64 bytes from 161.94.71.4: icmp_req=3 ttl=64 time=1.02 ms --- 161.94.71.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.023/4.647/11.737/5.013 ms 2025-07-31 04:57:49,082 - INFO - Ping to IP 161.94.71.4 from VM ctest-TestMtuProbe-15541328-27761176 passed 2025-07-31 04:57:49,083 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 04:57:49,083 - DEBUG - Running remote_cmd, Cmd : rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:57:49,084 - DEBUG - nohup rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile & 2025-07-31 04:57:49,776 - DEBUG - total 132 drwxr-xr-x 11 ubuntu ubuntu 4096 Dec 29 2017 . drwxr-xr-x 3 root root 4096 Jan 4 2013 .. -rw------- 1 ubuntu ubuntu 2689 Dec 29 2017 .bash_history -rw-r--r-- 1 ubuntu ubuntu 220 Apr 3 2012 .bash_logout -rw-r--r-- 1 ubuntu ubuntu 3486 Apr 3 2012 .bashrc drwx------ 2 ubuntu ubuntu 4096 Jan 9 2013 .cache -rwxrwxr-x 1 ubuntu ubuntu 1431 Feb 5 2015 flow_test_pktgen.sh drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fontconfig drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fvwm drwxrwxr-x 3 ubuntu ubuntu 4096 Nov 29 2016 .jenkins -rw------- 1 ubuntu ubuntu 52 Jul 2 2014 .lesshst drwx------ 4 ubuntu ubuntu 4096 Jan 9 2013 .mozilla -rwxrwxr-x 1 ubuntu ubuntu 1378 Jun 27 2014 pktgen_new.sh -rw-r--r-- 1 ubuntu ubuntu 675 Apr 3 2012 .profile drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 scr drwx------ 2 ubuntu ubuntu 4096 Feb 3 2015 .ssh -rw-r--r-- 1 ubuntu ubuntu 0 Jan 9 2013 .sudo_as_admin_successful drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 traffic drwxr-xr-x 4 root root 4096 Jun 27 2014 traffic-1.0 -rw-rw-r-- 1 ubuntu ubuntu 16361 Jan 9 2015 traffic-1.0.tar.gz -rw------- 1 root root 4695 Dec 17 2017 .viminfo -rw------- 1 ubuntu ubuntu 152 Jul 2 2014 .Xauthority -rw------- 1 ubuntu ubuntu 29828 Jan 9 2013 .xsession-errors 2025-07-31 04:57:49,776 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -ll 5201 > testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:57:49,776 - DEBUG - nohup nc -4 -q 2 -w 5 -ll 5201 > testfile & 2025-07-31 04:57:50,312 - DEBUG - None 2025-07-31 04:57:50,313 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 04:57:50,313 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 161.94.71.4 5201 < testfile 2025-07-31 04:57:55,870 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.000870844 s, 115 kB/s 2025-07-31 04:57:55,870 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:57:55,870 - DEBUG - ls -l testfile 2025-07-31 04:57:56,282 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile 2025-07-31 04:57:56,282 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:57:59,283 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:57:59,284 - DEBUG - ls -l testfile 2025-07-31 04:57:59,664 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile 2025-07-31 04:57:59,664 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:58:02,665 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:58:02,666 - DEBUG - ls -l testfile 2025-07-31 04:58:03,075 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile 2025-07-31 04:58:03,075 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:58:06,077 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:58:06,077 - DEBUG - ls -l testfile 2025-07-31 04:58:06,508 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile 2025-07-31 04:58:06,509 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:58:09,510 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:58:09,511 - DEBUG - ls -l testfile 2025-07-31 04:58:09,889 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile 2025-07-31 04:58:09,889 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:58:12,890 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:58:12,891 - DEBUG - ls -l testfile 2025-07-31 04:58:13,285 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile 2025-07-31 04:58:13,286 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:58:16,287 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:58:16,287 - DEBUG - ls -l testfile 2025-07-31 04:58:16,709 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile 2025-07-31 04:58:16,710 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:58:19,710 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:58:19,711 - DEBUG - ls -l testfile 2025-07-31 04:58:20,109 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile 2025-07-31 04:58:20,109 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:58:23,110 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:58:23,111 - DEBUG - ls -l testfile 2025-07-31 04:58:23,466 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile 2025-07-31 04:58:23,467 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:58:26,467 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:58:26,467 - DEBUG - ls -l testfile 2025-07-31 04:58:26,877 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile 2025-07-31 04:58:26,878 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:58:29,879 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:58:29,880 - DEBUG - ls -l testfile 2025-07-31 04:58:30,227 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile 2025-07-31 04:58:30,227 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:58:30,228 - INFO - File transfer verification for file size 100 failed on the VM ctest-TestMtuProbe-15541328-14386499 2025-07-31 04:58:30,228 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 04:58:30,229 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 161.94.71.4 5201 < testfile 2025-07-31 04:58:32,836 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.00122353 s, 81.7 kB/s 2025-07-31 04:58:32,836 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:58:32,836 - DEBUG - ls -l testfile 2025-07-31 04:58:33,225 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile 2025-07-31 04:58:33,225 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:58:36,226 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:58:36,227 - DEBUG - ls -l testfile 2025-07-31 04:58:36,632 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile 2025-07-31 04:58:36,633 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:58:39,634 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:58:39,634 - DEBUG - ls -l testfile 2025-07-31 04:58:40,096 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile 2025-07-31 04:58:40,096 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:58:43,098 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:58:43,098 - DEBUG - ls -l testfile 2025-07-31 04:58:43,482 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile 2025-07-31 04:58:43,483 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:58:46,483 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:58:46,484 - DEBUG - ls -l testfile 2025-07-31 04:58:46,881 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile 2025-07-31 04:58:46,881 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:58:49,882 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:58:49,883 - DEBUG - ls -l testfile 2025-07-31 04:58:50,329 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile 2025-07-31 04:58:50,330 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:58:53,331 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:58:53,332 - DEBUG - ls -l testfile 2025-07-31 04:58:53,724 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile 2025-07-31 04:58:53,724 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:58:56,725 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:58:56,725 - DEBUG - ls -l testfile 2025-07-31 04:58:57,088 - DEBUG - -rw-r--r-- 1 root root 100 Jul 31 04:58 testfile 2025-07-31 04:58:57,089 - INFO - File transfer verification for file size 100 passed on the VM ctest-TestMtuProbe-15541328-14386499 2025-07-31 04:58:57,089 - DEBUG - Running remote_cmd, Cmd : echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 04:58:57,090 - DEBUG - echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing 2025-07-31 04:58:57,615 - DEBUG - None 2025-07-31 04:58:57,616 - DEBUG - Running remote_cmd, Cmd : echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:58:57,616 - DEBUG - echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing 2025-07-31 04:58:58,704 - DEBUG - None 2025-07-31 04:58:58,705 - INFO - {'echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing': None} 2025-07-31 04:58:58,705 - INFO - {'echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing': None} 2025-07-31 04:58:58,705 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 04:58:58,705 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 161.94.71.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 04:58:58,705 - DEBUG - ping -s 56 -c 3 -W 1 161.94.71.4 2025-07-31 04:59:01,174 - DEBUG - PING 161.94.71.4 (161.94.71.4) 56(84) bytes of data. 64 bytes from 161.94.71.4: icmp_req=1 ttl=64 time=7.47 ms 64 bytes from 161.94.71.4: icmp_req=2 ttl=64 time=1.31 ms 64 bytes from 161.94.71.4: icmp_req=3 ttl=64 time=1.21 ms --- 161.94.71.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 1.215/3.333/7.470/2.925 ms 2025-07-31 04:59:01,174 - INFO - Ping to IP 161.94.71.4 from VM ctest-TestMtuProbe-15541328-27761176 passed 2025-07-31 04:59:01,175 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 04:59:01,175 - DEBUG - Running remote_cmd, Cmd : rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:59:01,175 - DEBUG - nohup rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile & 2025-07-31 04:59:01,986 - DEBUG - total 132 drwxr-xr-x 11 ubuntu ubuntu 4096 Jul 31 04:59 . drwxr-xr-x 3 root root 4096 Jan 4 2013 .. -rw------- 1 ubuntu ubuntu 2689 Dec 29 2017 .bash_history -rw-r--r-- 1 ubuntu ubuntu 220 Apr 3 2012 .bash_logout -rw-r--r-- 1 ubuntu ubuntu 3486 Apr 3 2012 .bashrc drwx------ 2 ubuntu ubuntu 4096 Jan 9 2013 .cache -rwxrwxr-x 1 ubuntu ubuntu 1431 Feb 5 2015 flow_test_pktgen.sh drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fontconfig drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fvwm drwxrwxr-x 3 ubuntu ubuntu 4096 Nov 29 2016 .jenkins -rw------- 1 ubuntu ubuntu 52 Jul 2 2014 .lesshst drwx------ 4 ubuntu ubuntu 4096 Jan 9 2013 .mozilla -rwxrwxr-x 1 ubuntu ubuntu 1378 Jun 27 2014 pktgen_new.sh -rw-r--r-- 1 ubuntu ubuntu 675 Apr 3 2012 .profile drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 scr drwx------ 2 ubuntu ubuntu 4096 Feb 3 2015 .ssh -rw-r--r-- 1 ubuntu ubuntu 0 Jan 9 2013 .sudo_as_admin_successful drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 traffic drwxr-xr-x 4 root root 4096 Jun 27 2014 traffic-1.0 -rw-rw-r-- 1 ubuntu ubuntu 16361 Jan 9 2015 traffic-1.0.tar.gz -rw------- 1 root root 4695 Dec 17 2017 .viminfo -rw------- 1 ubuntu ubuntu 152 Jul 2 2014 .Xauthority -rw------- 1 ubuntu ubuntu 29828 Jan 9 2013 .xsession-errors 2025-07-31 04:59:01,986 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -ll 5201 > testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:59:01,986 - DEBUG - nohup nc -4 -q 2 -w 5 -ll 5201 > testfile & 2025-07-31 04:59:02,564 - DEBUG - None 2025-07-31 04:59:02,564 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10002 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 04:59:02,564 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10002 161.94.71.4 5201 < testfile 2025-07-31 04:59:08,103 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.000969596 s, 103 kB/s 2025-07-31 04:59:08,103 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:59:08,103 - DEBUG - ls -l testfile 2025-07-31 04:59:08,550 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile 2025-07-31 04:59:08,551 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:59:11,552 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:59:11,552 - DEBUG - ls -l testfile 2025-07-31 04:59:11,961 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile 2025-07-31 04:59:11,962 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:59:14,962 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:59:14,963 - DEBUG - ls -l testfile 2025-07-31 04:59:15,357 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile 2025-07-31 04:59:15,358 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:59:18,359 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:59:18,360 - DEBUG - ls -l testfile 2025-07-31 04:59:18,824 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile 2025-07-31 04:59:18,824 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:59:21,825 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:59:21,826 - DEBUG - ls -l testfile 2025-07-31 04:59:22,250 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile 2025-07-31 04:59:22,251 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:59:25,252 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:59:25,252 - DEBUG - ls -l testfile 2025-07-31 04:59:25,683 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile 2025-07-31 04:59:25,684 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:59:28,684 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:59:28,685 - DEBUG - ls -l testfile 2025-07-31 04:59:29,112 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile 2025-07-31 04:59:29,112 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:59:32,113 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:59:32,113 - DEBUG - ls -l testfile 2025-07-31 04:59:32,557 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile 2025-07-31 04:59:32,557 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:59:35,559 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:59:35,559 - DEBUG - ls -l testfile 2025-07-31 04:59:35,960 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile 2025-07-31 04:59:35,960 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:59:38,961 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:59:38,961 - DEBUG - ls -l testfile 2025-07-31 04:59:39,332 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile 2025-07-31 04:59:39,332 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:59:42,333 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:59:42,334 - DEBUG - ls -l testfile 2025-07-31 04:59:42,700 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile 2025-07-31 04:59:42,700 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:59:42,700 - INFO - File transfer verification for file size 100 failed on the VM ctest-TestMtuProbe-15541328-14386499 2025-07-31 04:59:42,701 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10002 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 04:59:42,701 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10002 161.94.71.4 5201 < testfile 2025-07-31 04:59:45,272 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.00103981 s, 96.2 kB/s 2025-07-31 04:59:45,273 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:59:45,273 - DEBUG - ls -l testfile 2025-07-31 04:59:45,693 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile 2025-07-31 04:59:45,694 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:59:48,694 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:59:48,695 - DEBUG - ls -l testfile 2025-07-31 04:59:49,089 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile 2025-07-31 04:59:49,090 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:59:52,090 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:59:52,091 - DEBUG - ls -l testfile 2025-07-31 04:59:52,528 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile 2025-07-31 04:59:52,528 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:59:55,536 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:59:55,537 - DEBUG - ls -l testfile 2025-07-31 04:59:55,907 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile 2025-07-31 04:59:55,908 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 04:59:58,908 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 04:59:58,909 - DEBUG - ls -l testfile 2025-07-31 04:59:59,291 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile 2025-07-31 04:59:59,291 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:00:02,292 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:00:02,293 - DEBUG - ls -l testfile 2025-07-31 05:00:02,715 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile 2025-07-31 05:00:02,715 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:00:05,716 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:00:05,716 - DEBUG - ls -l testfile 2025-07-31 05:00:06,083 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile 2025-07-31 05:00:06,083 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:00:09,084 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:00:09,085 - DEBUG - ls -l testfile 2025-07-31 05:00:09,434 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile 2025-07-31 05:00:09,435 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:00:12,435 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:00:12,436 - DEBUG - ls -l testfile 2025-07-31 05:00:12,830 - DEBUG - -rw-r--r-- 1 root root 100 Jul 31 05:00 testfile 2025-07-31 05:00:12,830 - INFO - File transfer verification for file size 100 passed on the VM ctest-TestMtuProbe-15541328-14386499 2025-07-31 05:00:12,830 - DEBUG - Running remote_cmd, Cmd : echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:00:12,830 - DEBUG - echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing 2025-07-31 05:00:13,259 - DEBUG - None 2025-07-31 05:00:13,259 - DEBUG - Running remote_cmd, Cmd : echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:00:13,260 - DEBUG - echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing 2025-07-31 05:00:13,819 - DEBUG - None 2025-07-31 05:00:13,820 - INFO - {'echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing': None} 2025-07-31 05:00:13,820 - INFO - {'echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing': None} 2025-07-31 05:00:13,820 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 05:00:13,821 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 161.94.71.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:00:13,821 - DEBUG - ping -s 56 -c 3 -W 1 161.94.71.4 2025-07-31 05:00:16,307 - DEBUG - PING 161.94.71.4 (161.94.71.4) 56(84) bytes of data. 64 bytes from 161.94.71.4: icmp_req=1 ttl=64 time=2.97 ms 64 bytes from 161.94.71.4: icmp_req=2 ttl=64 time=1.71 ms 64 bytes from 161.94.71.4: icmp_req=3 ttl=64 time=1.36 ms --- 161.94.71.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.363/2.019/2.978/0.694 ms 2025-07-31 05:00:16,308 - INFO - Ping to IP 161.94.71.4 from VM ctest-TestMtuProbe-15541328-27761176 passed 2025-07-31 05:00:16,308 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 05:00:16,308 - DEBUG - Running remote_cmd, Cmd : rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:00:16,308 - DEBUG - nohup rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile & 2025-07-31 05:00:17,006 - DEBUG - total 132 drwxr-xr-x 11 ubuntu ubuntu 4096 Jul 31 05:00 . drwxr-xr-x 3 root root 4096 Jan 4 2013 .. -rw------- 1 ubuntu ubuntu 2689 Dec 29 2017 .bash_history -rw-r--r-- 1 ubuntu ubuntu 220 Apr 3 2012 .bash_logout -rw-r--r-- 1 ubuntu ubuntu 3486 Apr 3 2012 .bashrc drwx------ 2 ubuntu ubuntu 4096 Jan 9 2013 .cache -rwxrwxr-x 1 ubuntu ubuntu 1431 Feb 5 2015 flow_test_pktgen.sh drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fontconfig drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fvwm drwxrwxr-x 3 ubuntu ubuntu 4096 Nov 29 2016 .jenkins -rw------- 1 ubuntu ubuntu 52 Jul 2 2014 .lesshst drwx------ 4 ubuntu ubuntu 4096 Jan 9 2013 .mozilla -rwxrwxr-x 1 ubuntu ubuntu 1378 Jun 27 2014 pktgen_new.sh -rw-r--r-- 1 ubuntu ubuntu 675 Apr 3 2012 .profile drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 scr drwx------ 2 ubuntu ubuntu 4096 Feb 3 2015 .ssh -rw-r--r-- 1 ubuntu ubuntu 0 Jan 9 2013 .sudo_as_admin_successful drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 traffic drwxr-xr-x 4 root root 4096 Jun 27 2014 traffic-1.0 -rw-rw-r-- 1 ubuntu ubuntu 16361 Jan 9 2015 traffic-1.0.tar.gz -rw------- 1 root root 4695 Dec 17 2017 .viminfo -rw------- 1 ubuntu ubuntu 152 Jul 2 2014 .Xauthority -rw------- 1 ubuntu ubuntu 29828 Jan 9 2013 .xsession-errors 2025-07-31 05:00:17,007 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -ll 5201 > testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:00:17,007 - DEBUG - nohup nc -4 -q 2 -w 5 -ll 5201 > testfile & 2025-07-31 05:00:17,552 - DEBUG - None 2025-07-31 05:00:17,553 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:00:17,553 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile 2025-07-31 05:00:23,165 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.00156089 s, 64.1 kB/s 2025-07-31 05:00:23,165 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:00:23,165 - DEBUG - ls -l testfile 2025-07-31 05:00:23,559 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile 2025-07-31 05:00:23,560 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:00:26,561 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:00:26,561 - DEBUG - ls -l testfile 2025-07-31 05:00:26,956 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile 2025-07-31 05:00:26,956 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:00:29,957 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:00:29,957 - DEBUG - ls -l testfile 2025-07-31 05:00:30,406 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile 2025-07-31 05:00:30,408 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:00:33,409 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:00:33,409 - DEBUG - ls -l testfile 2025-07-31 05:00:33,793 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile 2025-07-31 05:00:33,793 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:00:36,798 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:00:36,799 - DEBUG - ls -l testfile 2025-07-31 05:00:37,268 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile 2025-07-31 05:00:37,268 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:00:40,269 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:00:40,269 - DEBUG - ls -l testfile 2025-07-31 05:00:40,661 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile 2025-07-31 05:00:40,661 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:00:43,663 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:00:43,663 - DEBUG - ls -l testfile 2025-07-31 05:00:44,035 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile 2025-07-31 05:00:44,035 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:00:47,035 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:00:47,036 - DEBUG - ls -l testfile 2025-07-31 05:00:47,413 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile 2025-07-31 05:00:47,413 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:00:50,414 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:00:50,415 - DEBUG - ls -l testfile 2025-07-31 05:00:50,825 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile 2025-07-31 05:00:50,825 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:00:53,826 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:00:53,826 - DEBUG - ls -l testfile 2025-07-31 05:00:54,300 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile 2025-07-31 05:00:54,300 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:00:57,302 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:00:57,302 - DEBUG - ls -l testfile 2025-07-31 05:00:57,676 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile 2025-07-31 05:00:57,676 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:00:57,676 - INFO - File transfer verification for file size 100 failed on the VM ctest-TestMtuProbe-15541328-14386499 2025-07-31 05:00:57,677 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:00:57,677 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile 2025-07-31 05:01:00,249 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.00101222 s, 98.8 kB/s 2025-07-31 05:01:00,250 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:01:00,250 - DEBUG - ls -l testfile 2025-07-31 05:01:00,674 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile 2025-07-31 05:01:00,674 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:01:03,675 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:01:03,675 - DEBUG - ls -l testfile 2025-07-31 05:01:04,094 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile 2025-07-31 05:01:04,095 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:01:07,096 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:01:07,096 - DEBUG - ls -l testfile 2025-07-31 05:01:07,488 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile 2025-07-31 05:01:07,488 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:01:10,490 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:01:10,490 - DEBUG - ls -l testfile 2025-07-31 05:01:10,932 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile 2025-07-31 05:01:10,932 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:01:13,933 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:01:13,933 - DEBUG - ls -l testfile 2025-07-31 05:01:14,306 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile 2025-07-31 05:01:14,306 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:01:17,307 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:01:17,308 - DEBUG - ls -l testfile 2025-07-31 05:01:17,701 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile 2025-07-31 05:01:17,701 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:01:20,702 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:01:20,703 - DEBUG - ls -l testfile 2025-07-31 05:01:21,140 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile 2025-07-31 05:01:21,140 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:01:24,140 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:01:24,141 - DEBUG - ls -l testfile 2025-07-31 05:01:24,507 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:01:24,507 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:01:27,508 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:01:27,508 - DEBUG - ls -l testfile 2025-07-31 05:01:27,883 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:01:27,883 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:01:30,884 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:01:30,885 - DEBUG - ls -l testfile 2025-07-31 05:01:31,263 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:01:31,263 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:01:34,264 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:01:34,265 - DEBUG - ls -l testfile 2025-07-31 05:01:34,636 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:01:34,636 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:01:34,637 - INFO - File transfer verification for file size 100 failed on the VM ctest-TestMtuProbe-15541328-14386499 2025-07-31 05:01:34,637 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:01:34,637 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile 2025-07-31 05:01:35,237 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.00103085 s, 97.0 kB/s nc: bind failed: Address already in use 2025-07-31 05:01:35,238 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:01:35,238 - DEBUG - nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile 2025-07-31 05:01:35,847 - DEBUG - nc: bind failed: Address already in use 2025-07-31 05:01:41,848 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:01:41,848 - DEBUG - nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile 2025-07-31 05:01:42,331 - DEBUG - nc: bind failed: Address already in use 2025-07-31 05:01:48,333 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:01:48,334 - DEBUG - nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile 2025-07-31 05:01:49,253 - DEBUG - nc: bind failed: Address already in use 2025-07-31 05:01:55,254 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:01:55,255 - DEBUG - nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile 2025-07-31 05:02:00,825 - DEBUG - None 2025-07-31 05:02:00,825 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:02:00,825 - DEBUG - ls -l testfile 2025-07-31 05:02:01,294 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:02:01,294 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:02:04,296 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:02:04,296 - DEBUG - ls -l testfile 2025-07-31 05:02:04,751 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:02:04,751 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:02:07,752 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:02:07,753 - DEBUG - ls -l testfile 2025-07-31 05:02:08,165 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:02:08,165 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:02:11,165 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:02:11,166 - DEBUG - ls -l testfile 2025-07-31 05:02:11,530 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:02:11,531 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:02:14,532 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:02:14,533 - DEBUG - ls -l testfile 2025-07-31 05:02:14,925 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:02:14,925 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:02:17,926 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:02:17,926 - DEBUG - ls -l testfile 2025-07-31 05:02:18,318 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:02:18,319 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:02:21,320 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:02:21,320 - DEBUG - ls -l testfile 2025-07-31 05:02:21,809 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:02:21,809 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:02:24,810 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:02:24,811 - DEBUG - ls -l testfile 2025-07-31 05:02:25,202 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:02:25,203 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:02:28,204 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:02:28,204 - DEBUG - ls -l testfile 2025-07-31 05:02:28,556 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:02:28,556 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:02:31,556 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:02:31,557 - DEBUG - ls -l testfile 2025-07-31 05:02:31,997 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:02:31,998 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:02:34,999 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:02:34,999 - DEBUG - ls -l testfile 2025-07-31 05:02:35,361 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:02:35,361 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:02:35,361 - INFO - File transfer verification for file size 100 failed on the VM ctest-TestMtuProbe-15541328-14386499 2025-07-31 05:02:35,361 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:02:35,361 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile 2025-07-31 05:02:35,930 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.0113714 s, 8.8 kB/s 2025-07-31 05:02:35,930 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:02:35,931 - DEBUG - ls -l testfile 2025-07-31 05:02:36,391 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:02:36,391 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:02:39,392 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:02:39,393 - DEBUG - ls -l testfile 2025-07-31 05:02:39,779 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:02:39,779 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:02:42,779 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:02:42,780 - DEBUG - ls -l testfile 2025-07-31 05:02:43,151 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:02:43,151 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:02:46,153 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:02:46,153 - DEBUG - ls -l testfile 2025-07-31 05:02:46,525 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:02:46,527 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:02:49,528 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:02:49,529 - DEBUG - ls -l testfile 2025-07-31 05:02:49,865 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:02:49,865 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:02:52,867 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:02:52,867 - DEBUG - ls -l testfile 2025-07-31 05:02:53,265 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:02:53,265 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:02:56,266 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:02:56,267 - DEBUG - ls -l testfile 2025-07-31 05:02:56,656 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:02:56,657 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:02:59,658 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:02:59,659 - DEBUG - ls -l testfile 2025-07-31 05:03:00,033 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:03:00,033 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:03:03,034 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:03:03,034 - DEBUG - ls -l testfile 2025-07-31 05:03:03,426 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:03:03,426 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:03:06,428 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:03:06,429 - DEBUG - ls -l testfile 2025-07-31 05:03:06,838 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:03:06,839 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:03:09,840 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:03:09,841 - DEBUG - ls -l testfile 2025-07-31 05:03:10,225 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:03:10,225 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:03:10,225 - INFO - File transfer verification for file size 100 failed on the VM ctest-TestMtuProbe-15541328-14386499 2025-07-31 05:03:10,225 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:03:10,226 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile 2025-07-31 05:03:10,844 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.00134623 s, 74.3 kB/s 2025-07-31 05:03:10,844 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:03:10,844 - DEBUG - ls -l testfile 2025-07-31 05:03:11,208 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:03:11,208 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:03:14,209 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:03:14,210 - DEBUG - ls -l testfile 2025-07-31 05:03:14,670 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:03:14,670 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:03:17,671 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:03:17,672 - DEBUG - ls -l testfile 2025-07-31 05:03:18,063 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:03:18,064 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:03:21,064 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:03:21,065 - DEBUG - ls -l testfile 2025-07-31 05:03:21,440 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:03:21,441 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:03:24,441 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:03:24,442 - DEBUG - ls -l testfile 2025-07-31 05:03:24,847 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:03:24,847 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:03:27,848 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:03:27,849 - DEBUG - ls -l testfile 2025-07-31 05:03:28,255 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:03:28,256 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:03:31,256 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:03:31,257 - DEBUG - ls -l testfile 2025-07-31 05:03:31,617 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:03:31,617 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:03:34,618 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:03:34,618 - DEBUG - ls -l testfile 2025-07-31 05:03:35,010 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:03:35,010 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:03:38,010 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:03:38,011 - DEBUG - ls -l testfile 2025-07-31 05:03:38,392 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:03:38,392 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:03:41,394 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:03:41,394 - DEBUG - ls -l testfile 2025-07-31 05:03:41,917 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:03:41,918 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:03:44,919 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-07-31 05:03:44,919 - DEBUG - ls -l testfile 2025-07-31 05:03:45,284 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile 2025-07-31 05:03:45,284 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-31 05:03:45,284 - INFO - File transfer verification for file size 100 failed on the VM ctest-TestMtuProbe-15541328-14386499 2025-07-31 05:03:45,287 - INFO - Deleting VM ctest-TestMtuProbe-15541328-14386499 2025-07-31 05:03:45,404 - INFO - Deleting VM ctest-TestMtuProbe-15541328-27761176 2025-07-31 05:03:45,485 - INFO - Deleting VN ctest-vn-93417578 2025-07-31 05:03:45,646 - DEBUG - VN 57f03fbd-2014-4b85-acb1-0d3ccb0e6724 still in use: Unable to complete operation on network 57f03fbd-2014-4b85-acb1-0d3ccb0e6724. There are one or more ports still in use on the network. Neutron server returns request_ids: ['req-ff05892a-4f4f-4005-855b-2cde990de49c'] 2025-07-31 05:03:45,646 - WARNING - Deleting VN ctest-vn-93417578 failed..Will retry 2025-07-31 05:03:47,910 - DEBUG - Response for deleting network () 2025-07-31 05:03:47,910 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/3a818f26-b148-455f-8164-2a9ad49a209f 2025-07-31 05:03:47,954 - DEBUG - Response Code: 404 2025-07-31 05:03:47,954 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 05:03:47,969 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 05:03:48,035 - DEBUG - Requesting: http://10.0.0.41:8082/project/e8822bde-b9be-4614-8a35-0e1e4b75b1f1 2025-07-31 05:03:48,142 - INFO - Validated that VN ctest-vn-93417578 is not found in API Server 2025-07-31 05:03:48,157 - DEBUG - VN ctest-vn-93417578 is not present in Agent 10.0.0.27 2025-07-31 05:03:48,175 - DEBUG - VN ctest-vn-93417578 is not present in Agent 10.0.0.57 2025-07-31 05:03:48,175 - INFO - Validated that VN ctest-vn-93417578 is not in any agent 2025-07-31 05:03:48,217 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn-93417578 info 2025-07-31 05:03:49,618 - ERROR - AssertionError Python 3.6.8: /usr/bin/python3 Thu Jul 31 05:03:45 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={}) 78 log.info('Initial checks done. Running the testcase now') 79 log.info('') 80 result = function(self, *args, **kwargs) 81 if self.inputs.upgrade: 82 pid = os.getpid() result = None function = self = args = () kwargs = {} /contrail-test/tcutils/util.py in wrapper(self=, *func_args=(), **func_kwargs={}) 1039 raise testtools.TestCase.skipException(msg) 1040 1041 return f(self, *func_args, **func_kwargs) 1042 return wrapper 1043 return decorator f = self = func_args = () func_kwargs = {} /contrail-test/scripts/vrouter/test_tcp_mtu_probe.py in test_mtu_probe_inter_node(self=) 84 self.configure_tcp_mtu_probe(client,server,value) 85 self.verify_icmp_traffic(client,server) 86 self.verify_tcp_traffic(client,server, value) 87 return True 88 self = self.verify_tcp_traffic = > client = server = value = 2 /contrail-test/scripts/vrouter/test_tcp_mtu_probe.py in verify_tcp_traffic(self=, client=, server=, value=2) 42 if value is 2: 43 sport += value 44 assert self.send_nc_traffic(client, server, sport, dport, proto), "TCP traffic failed" 45 return True 46 self = self.send_nc_traffic = > client = server = sport = 10003 dport = 5201 proto = 'tcp' AssertionError: TCP traffic failed __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {} __dir__ = __doc__ = 'Assertion failed.' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = args = ('TCP traffic failed',) with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper result = function(self, *args, **kwargs) File "/contrail-test/tcutils/util.py", line 1041, in wrapper return f(self, *func_args, **func_kwargs) File "/contrail-test/scripts/vrouter/test_tcp_mtu_probe.py", line 86, in test_mtu_probe_inter_node self.verify_tcp_traffic(client,server, value) File "/contrail-test/scripts/vrouter/test_tcp_mtu_probe.py", line 44, in verify_tcp_traffic assert self.send_nc_traffic(client, server, sport, dport, proto), "TCP traffic failed" AssertionError: TCP traffic failed 2025-07-31 05:03:49,657 - DEBUG - No XMPP flaps were noticed during the test 2025-07-31 05:03:49,658 - INFO - 2025-07-31 05:03:49,658 - INFO - END TEST : test_mtu_probe_inter_node : FAILED[0:08:59] 2025-07-31 05:03:49,658 - INFO - -------------------------------------------------------------------------------- 2025-07-31 05:03:49,674 - INFO - ================================================================================ 2025-07-31 05:03:49,675 - INFO - STARTING TEST : test_mtu_probe_intra_node 2025-07-31 05:03:49,675 - INFO - TEST DESCRIPTION : Description: check traffic configuring different tcp_mtu_probe vlaues on vm MTU probing values: 0, 1, 2 0 - Disabled (default) 1 - Disabled by default, Enabled only when ICMP black hole detected 2 - Always enabled, using the initial MSS value of tcp_base_mss system internally enables bit values for point 1&2 For test purpose, Idea is to check traffic works fine on configuring values Vm's across compute node steps: 1. create 1 VN and launch 2 VMs in same compute node 2. client and server VMs in same compute node 3. configure tcp_mtu_probe to 0, 1, 2 4. send icmp traffic assert if ping fails 5. send tcp traffic assert if tcp traffic fails Pass criteria: 1. If icmp/tcp traffic work fine configuring tcp_mtu_probe values 2025-07-31 05:03:51,070 - DEBUG - Nothing to compare xmpp stats {'10.0.0.27': {'10.20.0.193': '0', '10.20.0.18': '0'}, '10.0.0.57': {'10.20.0.18': '0', '10.20.0.193': '0'}} with 2025-07-31 05:03:51,070 - INFO - Initial checks done. Running the testcase now 2025-07-31 05:03:51,070 - INFO - 2025-07-31 05:03:51,539 - DEBUG - Response for create_network : {'network': {'id': 'b16f7087-63cd-46a6-913d-359796e57d1c', 'name': 'ctest-vn-28174710', 'tenant_id': 'e8822bdeb9be46148a350e1e4b75b1f1', 'project_id': 'e8822bdeb9be46148a350e1e4b75b1f1', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestMtuProbe-15541328', 'ctest-vn-28174710'], 'port_security_enabled': True, 'description': ''}} 2025-07-31 05:03:51,792 - DEBUG - Response for create_subnet : {'subnet': {'id': 'd6902232-871e-4b7c-a862-18b2e7d6e5cb', 'name': '', 'tenant_id': 'e8822bdeb9be46148a350e1e4b75b1f1', 'network_id': 'b16f7087-63cd-46a6-913d-359796e57d1c', 'ip_version': 4, 'cidr': '5.28.8.64/26', 'allocation_pools': [{'start': '5.28.8.66', 'end': '5.28.8.126'}], 'gateway_ip': '5.28.8.65', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '5.28.8.66', 'tags': [], 'project_id': 'e8822bdeb9be46148a350e1e4b75b1f1'}} 2025-07-31 05:03:51,828 - INFO - Created VN ctest-vn-28174710 2025-07-31 05:03:51,843 - DEBUG - VN ctest-vn-28174710 UUID is b16f7087-63cd-46a6-913d-359796e57d1c 2025-07-31 05:03:51,954 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 05:03:51,963 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 05:03:51,976 - DEBUG - Requesting: http://10.0.0.41:8082/project/e8822bde-b9be-4614-8a35-0e1e4b75b1f1 2025-07-31 05:03:52,102 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/b16f7087-63cd-46a6-913d-359796e57d1c 2025-07-31 05:03:52,120 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/b16f7087-63cd-46a6-913d-359796e57d1c 2025-07-31 05:03:52,131 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/7f9efe65-2124-42c5-88ca-39cdf2bd6428 2025-07-31 05:03:52,146 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/7f9efe65-2124-42c5-88ca-39cdf2bd6428 2025-07-31 05:03:52,160 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/9f40d27d-b14f-4fa1-961f-d002b53e2be8 2025-07-31 05:03:52,168 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-31 05:03:52,168 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/b16f7087-63cd-46a6-913d-359796e57d1c 2025-07-31 05:03:52,183 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/7f9efe65-2124-42c5-88ca-39cdf2bd6428 2025-07-31 05:03:52,205 - INFO - Verified VN network id 11 for VN b16f7087-63cd-46a6-913d-359796e57d1c 2025-07-31 05:03:52,205 - INFO - Verifications in API Server for VN ctest-vn-28174710 passed 2025-07-31 05:03:52,205 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/b16f7087-63cd-46a6-913d-359796e57d1c 2025-07-31 05:03:52,217 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/7f9efe65-2124-42c5-88ca-39cdf2bd6428 2025-07-31 05:03:52,228 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/7f9efe65-2124-42c5-88ca-39cdf2bd6428 2025-07-31 05:03:52,239 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/9f40d27d-b14f-4fa1-961f-d002b53e2be8 2025-07-31 05:03:52,254 - DEBUG - Control-node 10.0.0.41 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestMtuProbe-15541328', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '12785561593925420710', 'uuid-lslong': '10465580034266856732'}, 'enable': 'true', 'created': '2025-07-31T05:03:51', 'last-modified': '2025-07-31T05:03:51', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.563463'} 2025-07-31 05:03:52,261 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-31 05:03:52,271 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestMtuProbe-15541328', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '12785561593925420710', 'uuid-lslong': '10465580034266856732'}, 'enable': 'true', 'created': '2025-07-31T05:03:51', 'last-modified': '2025-07-31T05:03:51', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.581138'} 2025-07-31 05:03:52,278 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-31 05:03:52,286 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestMtuProbe-15541328', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '12785561593925420710', 'uuid-lslong': '10465580034266856732'}, 'enable': 'true', 'created': '2025-07-31T05:03:51', 'last-modified': '2025-07-31T05:03:51', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.599730'} 2025-07-31 05:03:52,293 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-31 05:03:52,294 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn-28174710 passed 2025-07-31 05:03:52,294 - DEBUG - ====Verifying policy data for ctest-vn-28174710 in API_Server ====== 2025-07-31 05:03:52,294 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 05:03:52,302 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 05:03:52,322 - DEBUG - Requesting: http://10.0.0.41:8082/project/e8822bde-b9be-4614-8a35-0e1e4b75b1f1 2025-07-31 05:03:52,468 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/b16f7087-63cd-46a6-913d-359796e57d1c 2025-07-31 05:03:52,482 - DEBUG - =>VN ctest-vn-28174710 has no policy to be verified 2025-07-31 05:03:52,482 - DEBUG - Verifying the vn in opserver 2025-07-31 05:03:52,482 - DEBUG - Verifying the default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 virtual network link through opserver 10.0.0.41 2025-07-31 05:03:52,482 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks 2025-07-31 05:03:52,568 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-network/default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710?flat'} 2025-07-31 05:03:52,568 - DEBUG - Verifying the default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 virtual network link through opserver 10.0.0.38 2025-07-31 05:03:52,568 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-07-31 05:03:52,668 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710?flat'} 2025-07-31 05:03:52,668 - DEBUG - Verifying the default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 virtual network link through opserver 10.0.0.61 2025-07-31 05:03:52,668 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-07-31 05:03:52,781 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710?flat'} 2025-07-31 05:03:52,782 - INFO - Validated that VN default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is found in opserver 2025-07-31 05:03:52,839 - DEBUG - Do not have enough data to verify VN in agent 2025-07-31 05:03:52,855 - DEBUG - VRF ids for VN ctest-vn-28174710: {} 2025-07-31 05:03:52,856 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3732-1', 'an-jenkins-deploy-platform-ansible-os-3732-2']} 2025-07-31 05:03:52,856 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3732-1', 'an-jenkins-deploy-platform-ansible-os-3732-2']} 2025-07-31 05:03:54,827 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3732-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3732-1) 2025-07-31 05:03:54,827 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3732-1', 'an-jenkins-deploy-platform-ansible-os-3732-2']} 2025-07-31 05:03:56,402 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3732-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3732-1) 2025-07-31 05:03:56,402 - INFO - Waiting for VM ctest-TestMtuProbe-15541328-22439080 to be up.. 2025-07-31 05:03:56,509 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-07-31 05:04:01,607 - DEBUG - VM is in ACTIVE state now 2025-07-31 05:04:01,607 - INFO - VM name : ctest-TestMtuProbe-15541328-22439080 2025-07-31 05:04:01,712 - DEBUG - VM ctest-TestMtuProbe-15541328-22439080 ID is 0ea4c3b4-c17f-4dce-959b-178975d4f446 2025-07-31 05:04:01,712 - DEBUG - VM ctest-TestMtuProbe-15541328-22439080 launched on Node an-jenkins-deploy-platform-ansible-os-3732-1 2025-07-31 05:04:01,804 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine/0ea4c3b4-c17f-4dce-959b-178975d4f446 2025-07-31 05:04:01,838 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine-interface/b6eb1f5d-5a71-4b22-bb17-56c1c38c373b 2025-07-31 05:04:05,100 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1024ms') 2025-07-31 05:04:05,101 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 failed! 2025-07-31 05:04:05,128 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is 5.28.8.65 and allocation pool is NOT set 2025-07-31 05:04:09,231 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1030ms') 2025-07-31 05:04:09,231 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 failed! 2025-07-31 05:04:09,262 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is 5.28.8.65 and allocation pool is NOT set 2025-07-31 05:04:13,352 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1024ms') 2025-07-31 05:04:13,353 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 failed! 2025-07-31 05:04:13,379 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is 5.28.8.65 and allocation pool is NOT set 2025-07-31 05:04:17,451 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1001ms') 2025-07-31 05:04:17,451 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 failed! 2025-07-31 05:04:17,479 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is 5.28.8.65 and allocation pool is NOT set 2025-07-31 05:04:21,549 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1002ms') 2025-07-31 05:04:21,549 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 failed! 2025-07-31 05:04:21,566 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is 5.28.8.65 and allocation pool is NOT set 2025-07-31 05:04:25,649 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms') 2025-07-31 05:04:25,649 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 failed! 2025-07-31 05:04:25,682 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is 5.28.8.65 and allocation pool is NOT set 2025-07-31 05:04:29,773 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1020ms') 2025-07-31 05:04:29,773 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 failed! 2025-07-31 05:04:29,799 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is 5.28.8.65 and allocation pool is NOT set 2025-07-31 05:04:33,862 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1002ms') 2025-07-31 05:04:33,863 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 failed! 2025-07-31 05:04:33,889 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is 5.28.8.65 and allocation pool is NOT set 2025-07-31 05:04:37,967 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms') 2025-07-31 05:04:37,967 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 failed! 2025-07-31 05:04:37,984 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is 5.28.8.65 and allocation pool is NOT set 2025-07-31 05:04:42,058 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1009ms') 2025-07-31 05:04:42,059 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 failed! 2025-07-31 05:04:42,094 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is 5.28.8.65 and allocation pool is NOT set 2025-07-31 05:04:46,193 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2025-07-31 05:04:46,193 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 failed! 2025-07-31 05:04:46,222 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is 5.28.8.65 and allocation pool is NOT set 2025-07-31 05:04:48,296 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=6.74 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=3.83 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 3.825/5.281/6.737/1.456 ms') 2025-07-31 05:04:48,296 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 passed 2025-07-31 05:04:48,419 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 05:04:48,419 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-22439080, IP 5.28.8.67, Port 22 2025-07-31 05:04:48,488 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-31 05:04:48,650 - DEBUG - VM ctest-TestMtuProbe-15541328-22439080 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-31 05:04:53,651 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 05:04:53,652 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-22439080, IP 5.28.8.67, Port 22 2025-07-31 05:04:53,723 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-31 05:04:53,910 - DEBUG - VM ctest-TestMtuProbe-15541328-22439080 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-31 05:04:58,911 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 05:04:58,912 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-22439080, IP 5.28.8.67, Port 22 2025-07-31 05:04:59,004 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-31 05:04:59,144 - DEBUG - VM ctest-TestMtuProbe-15541328-22439080 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-31 05:05:04,145 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 05:05:04,145 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-22439080, IP 5.28.8.67, Port 22 2025-07-31 05:05:04,219 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-31 05:05:04,373 - DEBUG - VM ctest-TestMtuProbe-15541328-22439080 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-31 05:05:09,374 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 05:05:09,374 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-22439080, IP 5.28.8.67, Port 22 2025-07-31 05:05:09,447 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-31 05:05:09,654 - DEBUG - VM ctest-TestMtuProbe-15541328-22439080 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-31 05:05:14,655 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 05:05:14,655 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-22439080, IP 5.28.8.67, Port 22 2025-07-31 05:05:14,817 - DEBUG - VM ctest-TestMtuProbe-15541328-22439080 is ready for SSH connections 2025-07-31 05:05:14,817 - INFO - Waiting for VM ctest-TestMtuProbe-15541328-05529872 to be up.. 2025-07-31 05:05:15,021 - DEBUG - VM is in ACTIVE state now 2025-07-31 05:05:15,021 - INFO - VM name : ctest-TestMtuProbe-15541328-05529872 2025-07-31 05:05:15,183 - DEBUG - VM ctest-TestMtuProbe-15541328-05529872 ID is 85fb464b-3624-40a6-95ec-b2d247f7bc67 2025-07-31 05:05:15,183 - DEBUG - VM ctest-TestMtuProbe-15541328-05529872 launched on Node an-jenkins-deploy-platform-ansible-os-3732-1 2025-07-31 05:05:15,351 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine/85fb464b-3624-40a6-95ec-b2d247f7bc67 2025-07-31 05:05:15,378 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine-interface/611939da-1dad-453f-99bf-4814ab89ffe0 2025-07-31 05:05:16,645 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=4.35 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=1.19 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 1.193/2.769/4.345/1.576 ms') 2025-07-31 05:05:16,646 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-TestMtuProbe-15541328-05529872 passed 2025-07-31 05:05:16,795 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 05:05:16,795 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-05529872, IP 5.28.8.68, Port 22 2025-07-31 05:05:16,985 - DEBUG - VM ctest-TestMtuProbe-15541328-05529872 is ready for SSH connections 2025-07-31 05:05:16,985 - DEBUG - Running remote_cmd, Cmd : echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:05:16,985 - DEBUG - echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing 2025-07-31 05:06:11,793 - DEBUG - None 2025-07-31 05:06:11,794 - DEBUG - Running remote_cmd, Cmd : echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:06:11,794 - DEBUG - echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing 2025-07-31 05:06:54,017 - DEBUG - None 2025-07-31 05:06:54,018 - INFO - {'echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing': None} 2025-07-31 05:06:54,018 - INFO - {'echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing': None} 2025-07-31 05:06:54,020 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 05:06:54,020 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 5.28.8.68, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:06:54,020 - DEBUG - ping -s 56 -c 3 -W 1 5.28.8.68 2025-07-31 05:06:56,604 - DEBUG - PING 5.28.8.68 (5.28.8.68) 56(84) bytes of data. 64 bytes from 5.28.8.68: icmp_req=1 ttl=64 time=8.17 ms 64 bytes from 5.28.8.68: icmp_req=2 ttl=64 time=1.01 ms 64 bytes from 5.28.8.68: icmp_req=3 ttl=64 time=1.31 ms --- 5.28.8.68 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.012/3.499/8.170/3.305 ms 2025-07-31 05:06:56,605 - INFO - Ping to IP 5.28.8.68 from VM ctest-TestMtuProbe-15541328-22439080 passed 2025-07-31 05:06:56,605 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 05:06:56,606 - DEBUG - Running remote_cmd, Cmd : rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:06:56,606 - DEBUG - nohup rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile & 2025-07-31 05:06:57,199 - DEBUG - total 132 drwxr-xr-x 11 ubuntu ubuntu 4096 Dec 29 2017 . drwxr-xr-x 3 root root 4096 Jan 4 2013 .. -rw------- 1 ubuntu ubuntu 2689 Dec 29 2017 .bash_history -rw-r--r-- 1 ubuntu ubuntu 220 Apr 3 2012 .bash_logout -rw-r--r-- 1 ubuntu ubuntu 3486 Apr 3 2012 .bashrc drwx------ 2 ubuntu ubuntu 4096 Jan 9 2013 .cache -rwxrwxr-x 1 ubuntu ubuntu 1431 Feb 5 2015 flow_test_pktgen.sh drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fontconfig drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fvwm drwxrwxr-x 3 ubuntu ubuntu 4096 Nov 29 2016 .jenkins -rw------- 1 ubuntu ubuntu 52 Jul 2 2014 .lesshst drwx------ 4 ubuntu ubuntu 4096 Jan 9 2013 .mozilla -rwxrwxr-x 1 ubuntu ubuntu 1378 Jun 27 2014 pktgen_new.sh -rw-r--r-- 1 ubuntu ubuntu 675 Apr 3 2012 .profile drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 scr drwx------ 2 ubuntu ubuntu 4096 Feb 3 2015 .ssh -rw-r--r-- 1 ubuntu ubuntu 0 Jan 9 2013 .sudo_as_admin_successful drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 traffic drwxr-xr-x 4 root root 4096 Jun 27 2014 traffic-1.0 -rw-rw-r-- 1 ubuntu ubuntu 16361 Jan 9 2015 traffic-1.0.tar.gz -rw------- 1 root root 4695 Dec 17 2017 .viminfo -rw------- 1 ubuntu ubuntu 152 Jul 2 2014 .Xauthority -rw------- 1 ubuntu ubuntu 29828 Jan 9 2013 .xsession-errors 2025-07-31 05:06:57,199 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -ll 5201 > testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:06:57,199 - DEBUG - nohup nc -4 -q 2 -w 5 -ll 5201 > testfile & 2025-07-31 05:06:57,647 - DEBUG - None 2025-07-31 05:06:57,648 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 5.28.8.68 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:06:57,648 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 5.28.8.68 5201 < testfile 2025-07-31 05:07:00,320 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.00107571 s, 93.0 kB/s 2025-07-31 05:07:00,320 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:07:00,320 - DEBUG - ls -l testfile 2025-07-31 05:07:00,686 - DEBUG - -rw-r--r-- 1 root root 100 Jul 31 05:06 testfile 2025-07-31 05:07:00,686 - INFO - File transfer verification for file size 100 passed on the VM ctest-TestMtuProbe-15541328-05529872 2025-07-31 05:07:00,686 - DEBUG - Running remote_cmd, Cmd : echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:07:00,686 - DEBUG - echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing 2025-07-31 05:07:01,150 - DEBUG - None 2025-07-31 05:07:01,150 - DEBUG - Running remote_cmd, Cmd : echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:07:01,150 - DEBUG - echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing 2025-07-31 05:07:01,588 - DEBUG - None 2025-07-31 05:07:01,588 - INFO - {'echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing': None} 2025-07-31 05:07:01,588 - INFO - {'echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing': None} 2025-07-31 05:07:01,588 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 05:07:01,588 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 5.28.8.68, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:07:01,588 - DEBUG - ping -s 56 -c 3 -W 1 5.28.8.68 2025-07-31 05:07:04,141 - DEBUG - PING 5.28.8.68 (5.28.8.68) 56(84) bytes of data. 64 bytes from 5.28.8.68: icmp_req=1 ttl=64 time=2.28 ms 64 bytes from 5.28.8.68: icmp_req=2 ttl=64 time=0.881 ms 64 bytes from 5.28.8.68: icmp_req=3 ttl=64 time=0.836 ms --- 5.28.8.68 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 0.836/1.334/2.286/0.673 ms 2025-07-31 05:07:04,141 - INFO - Ping to IP 5.28.8.68 from VM ctest-TestMtuProbe-15541328-22439080 passed 2025-07-31 05:07:04,141 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 05:07:04,141 - DEBUG - Running remote_cmd, Cmd : rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:07:04,141 - DEBUG - nohup rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile & 2025-07-31 05:07:04,774 - DEBUG - total 132 drwxr-xr-x 11 ubuntu ubuntu 4096 Jul 31 05:07 . drwxr-xr-x 3 root root 4096 Jan 4 2013 .. -rw------- 1 ubuntu ubuntu 2689 Dec 29 2017 .bash_history -rw-r--r-- 1 ubuntu ubuntu 220 Apr 3 2012 .bash_logout -rw-r--r-- 1 ubuntu ubuntu 3486 Apr 3 2012 .bashrc drwx------ 2 ubuntu ubuntu 4096 Jan 9 2013 .cache -rwxrwxr-x 1 ubuntu ubuntu 1431 Feb 5 2015 flow_test_pktgen.sh drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fontconfig drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fvwm drwxrwxr-x 3 ubuntu ubuntu 4096 Nov 29 2016 .jenkins -rw------- 1 ubuntu ubuntu 52 Jul 2 2014 .lesshst drwx------ 4 ubuntu ubuntu 4096 Jan 9 2013 .mozilla -rwxrwxr-x 1 ubuntu ubuntu 1378 Jun 27 2014 pktgen_new.sh -rw-r--r-- 1 ubuntu ubuntu 675 Apr 3 2012 .profile drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 scr drwx------ 2 ubuntu ubuntu 4096 Feb 3 2015 .ssh -rw-r--r-- 1 ubuntu ubuntu 0 Jan 9 2013 .sudo_as_admin_successful drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 traffic drwxr-xr-x 4 root root 4096 Jun 27 2014 traffic-1.0 -rw-rw-r-- 1 ubuntu ubuntu 16361 Jan 9 2015 traffic-1.0.tar.gz -rw------- 1 root root 4695 Dec 17 2017 .viminfo -rw------- 1 ubuntu ubuntu 152 Jul 2 2014 .Xauthority -rw------- 1 ubuntu ubuntu 29828 Jan 9 2013 .xsession-errors 2025-07-31 05:07:04,775 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -ll 5201 > testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:07:04,775 - DEBUG - nohup nc -4 -q 2 -w 5 -ll 5201 > testfile & 2025-07-31 05:07:05,238 - DEBUG - None 2025-07-31 05:07:05,239 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10002 5.28.8.68 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:07:05,239 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10002 5.28.8.68 5201 < testfile 2025-07-31 05:07:07,837 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.00117167 s, 85.3 kB/s 2025-07-31 05:07:07,838 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:07:07,838 - DEBUG - ls -l testfile 2025-07-31 05:07:08,180 - DEBUG - -rw-r--r-- 1 root root 100 Jul 31 05:07 testfile 2025-07-31 05:07:08,180 - INFO - File transfer verification for file size 100 passed on the VM ctest-TestMtuProbe-15541328-05529872 2025-07-31 05:07:08,180 - DEBUG - Running remote_cmd, Cmd : echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:07:08,181 - DEBUG - echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing 2025-07-31 05:07:08,616 - DEBUG - None 2025-07-31 05:07:08,617 - DEBUG - Running remote_cmd, Cmd : echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:07:08,617 - DEBUG - echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing 2025-07-31 05:07:09,092 - DEBUG - None 2025-07-31 05:07:09,093 - INFO - {'echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing': None} 2025-07-31 05:07:09,093 - INFO - {'echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing': None} 2025-07-31 05:07:09,093 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 05:07:09,093 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 5.28.8.68, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:07:09,093 - DEBUG - ping -s 56 -c 3 -W 1 5.28.8.68 2025-07-31 05:07:11,570 - DEBUG - PING 5.28.8.68 (5.28.8.68) 56(84) bytes of data. 64 bytes from 5.28.8.68: icmp_req=1 ttl=64 time=3.05 ms 64 bytes from 5.28.8.68: icmp_req=2 ttl=64 time=1.10 ms 64 bytes from 5.28.8.68: icmp_req=3 ttl=64 time=1.13 ms --- 5.28.8.68 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2005ms rtt min/avg/max/mdev = 1.101/1.762/3.053/0.912 ms 2025-07-31 05:07:11,570 - INFO - Ping to IP 5.28.8.68 from VM ctest-TestMtuProbe-15541328-22439080 passed 2025-07-31 05:07:11,571 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-31 05:07:11,571 - DEBUG - Running remote_cmd, Cmd : rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:07:11,571 - DEBUG - nohup rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile & 2025-07-31 05:07:12,153 - DEBUG - total 132 drwxr-xr-x 11 ubuntu ubuntu 4096 Jul 31 05:07 . drwxr-xr-x 3 root root 4096 Jan 4 2013 .. -rw------- 1 ubuntu ubuntu 2689 Dec 29 2017 .bash_history -rw-r--r-- 1 ubuntu ubuntu 220 Apr 3 2012 .bash_logout -rw-r--r-- 1 ubuntu ubuntu 3486 Apr 3 2012 .bashrc drwx------ 2 ubuntu ubuntu 4096 Jan 9 2013 .cache -rwxrwxr-x 1 ubuntu ubuntu 1431 Feb 5 2015 flow_test_pktgen.sh drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fontconfig drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fvwm drwxrwxr-x 3 ubuntu ubuntu 4096 Nov 29 2016 .jenkins -rw------- 1 ubuntu ubuntu 52 Jul 2 2014 .lesshst drwx------ 4 ubuntu ubuntu 4096 Jan 9 2013 .mozilla -rwxrwxr-x 1 ubuntu ubuntu 1378 Jun 27 2014 pktgen_new.sh -rw-r--r-- 1 ubuntu ubuntu 675 Apr 3 2012 .profile drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 scr drwx------ 2 ubuntu ubuntu 4096 Feb 3 2015 .ssh -rw-r--r-- 1 ubuntu ubuntu 0 Jan 9 2013 .sudo_as_admin_successful drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 traffic drwxr-xr-x 4 root root 4096 Jun 27 2014 traffic-1.0 -rw-rw-r-- 1 ubuntu ubuntu 16361 Jan 9 2015 traffic-1.0.tar.gz -rw------- 1 root root 4695 Dec 17 2017 .viminfo -rw------- 1 ubuntu ubuntu 152 Jul 2 2014 .Xauthority -rw------- 1 ubuntu ubuntu 29828 Jan 9 2013 .xsession-errors 2025-07-31 05:07:12,153 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -ll 5201 > testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:07:12,153 - DEBUG - nohup nc -4 -q 2 -w 5 -ll 5201 > testfile & 2025-07-31 05:07:12,667 - DEBUG - None 2025-07-31 05:07:12,668 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 5.28.8.68 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:07:12,668 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 5.28.8.68 5201 < testfile 2025-07-31 05:07:15,240 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.000948353 s, 105 kB/s 2025-07-31 05:07:15,240 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-07-31 05:07:15,241 - DEBUG - ls -l testfile 2025-07-31 05:07:15,576 - DEBUG - -rw-r--r-- 1 root root 100 Jul 31 05:07 testfile 2025-07-31 05:07:15,577 - INFO - File transfer verification for file size 100 passed on the VM ctest-TestMtuProbe-15541328-05529872 2025-07-31 05:07:15,577 - INFO - Deleting VM ctest-TestMtuProbe-15541328-05529872 2025-07-31 05:07:15,713 - INFO - Deleting VM ctest-TestMtuProbe-15541328-22439080 2025-07-31 05:07:15,784 - INFO - Deleting VN ctest-vn-28174710 2025-07-31 05:07:15,836 - DEBUG - VN b16f7087-63cd-46a6-913d-359796e57d1c still in use: Unable to complete operation on network b16f7087-63cd-46a6-913d-359796e57d1c. There are one or more ports still in use on the network. Neutron server returns request_ids: ['req-ff315fdc-ac8f-40cf-bf91-f03935cdf27e'] 2025-07-31 05:07:15,836 - WARNING - Deleting VN ctest-vn-28174710 failed..Will retry 2025-07-31 05:07:18,024 - DEBUG - Response for deleting network () 2025-07-31 05:07:18,024 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/7f9efe65-2124-42c5-88ca-39cdf2bd6428 2025-07-31 05:07:18,033 - DEBUG - Response Code: 404 2025-07-31 05:07:18,033 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 05:07:18,041 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 05:07:18,069 - DEBUG - Requesting: http://10.0.0.41:8082/project/e8822bde-b9be-4614-8a35-0e1e4b75b1f1 2025-07-31 05:07:18,169 - INFO - Validated that VN ctest-vn-28174710 is not found in API Server 2025-07-31 05:07:18,184 - DEBUG - VN ctest-vn-28174710 is not present in Agent 10.0.0.27 2025-07-31 05:07:18,199 - DEBUG - VN ctest-vn-28174710 is not present in Agent 10.0.0.57 2025-07-31 05:07:18,199 - INFO - Validated that VN ctest-vn-28174710 is not in any agent 2025-07-31 05:07:18,251 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn-28174710 info 2025-07-31 05:07:19,598 - DEBUG - No XMPP flaps were noticed during the test 2025-07-31 05:07:19,598 - INFO - END TEST : test_mtu_probe_intra_node : PASSED[0:03:30] 2025-07-31 05:07:19,598 - INFO - -------------------------------------------------------------------------------- 2025-07-31 05:07:20,483 - INFO - Deleted project: ctest-TestMtuProbe-15541328, ID : e8822bde-b9be-4614-8a35-0e1e4b75b1f1