2025-10-02 03:48:41,485 - INFO - Domain Default found not creating 2025-10-02 03:48:41,639 - INFO - Project ctest-TestBasicVMVN0-35143319 not found, creating it 2025-10-02 03:48:42,143 - INFO - Created Project:ctest-TestBasicVMVN0-35143319, ID : 7612bb4f-d0d8-4c4e-9c49-460de03b6971 2025-10-02 03:48:43,915 - INFO - ================================================================================ 2025-10-02 03:48:43,915 - INFO - STARTING TEST : test_control_node_switchover 2025-10-02 03:48:43,915 - INFO - TEST DESCRIPTION : Stop the control node and check peering with agent fallback to other control node. 1. Pick one VN from respource pool which has 2 VM's in it 2. Verify ping between VM's 3. Find active control node in cluster by agent inspect 4. Stop control service on active control node 5. Verify agents are connected to new active control-node using xmpp connections 6. Bring back control service on previous active node 7. Verify ping between VM's again after bringing up control serveice Pass criteria: Step 2,5 and 7 should pass 2025-10-02 03:48:45,171 - DEBUG - Skipping xmpp flap check 2025-10-02 03:48:45,171 - INFO - Initial checks done. Running the testcase now 2025-10-02 03:48:45,171 - INFO - 2025-10-02 03:48:45,807 - DEBUG - Response for create_network : {'network': {'id': '29920874-38d7-4e76-81d9-931fb87a1857', 'name': 'ctest-vn1-94725148', 'tenant_id': '7612bb4fd0d84c4e9c49460de03b6971', 'project_id': '7612bb4fd0d84c4e9c49460de03b6971', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestBasicVMVN0-35143319', 'ctest-vn1-94725148'], 'port_security_enabled': True, 'description': ''}} 2025-10-02 03:48:46,135 - DEBUG - Response for create_subnet : {'subnet': {'id': '28c3a764-171f-4a48-b4bf-8afee24ac65f', 'name': '', 'tenant_id': '7612bb4fd0d84c4e9c49460de03b6971', 'network_id': '29920874-38d7-4e76-81d9-931fb87a1857', 'ip_version': 4, 'cidr': '192.168.1.0/24', 'allocation_pools': [{'start': '192.168.1.2', 'end': '192.168.1.254'}], 'gateway_ip': '192.168.1.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '192.168.1.2', 'tags': [], 'project_id': '7612bb4fd0d84c4e9c49460de03b6971'}} 2025-10-02 03:48:46,159 - INFO - Created VN ctest-vn1-94725148 2025-10-02 03:48:46,171 - DEBUG - VN ctest-vn1-94725148 UUID is 29920874-38d7-4e76-81d9-931fb87a1857 2025-10-02 03:48:46,254 - DEBUG - Requesting: http://10.0.0.56:8082/domains 2025-10-02 03:48:46,552 - DEBUG - Requesting: http://10.0.0.56:8082/domains 2025-10-02 03:48:46,581 - DEBUG - Requesting: http://10.0.0.56:8082/domain/b0543e69-2184-4be1-8e23-ac5840233bc6 2025-10-02 03:48:46,596 - DEBUG - Requesting: http://10.0.0.56:8082/project/7612bb4f-d0d8-4c4e-9c49-460de03b6971 2025-10-02 03:48:46,692 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/29920874-38d7-4e76-81d9-931fb87a1857 2025-10-02 03:48:46,703 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/29920874-38d7-4e76-81d9-931fb87a1857 2025-10-02 03:48:46,711 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/66cade29-e9b2-46d4-8eb4-75fcb4c81bee 2025-10-02 03:48:46,720 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/66cade29-e9b2-46d4-8eb4-75fcb4c81bee 2025-10-02 03:48:46,728 - DEBUG - Requesting: http://10.0.0.56:8082/route-target/8e8f481c-7398-4d6a-9cb6-0c036f63a80e 2025-10-02 03:48:46,735 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-02 03:48:46,735 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/29920874-38d7-4e76-81d9-931fb87a1857 2025-10-02 03:48:46,743 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/66cade29-e9b2-46d4-8eb4-75fcb4c81bee 2025-10-02 03:48:46,760 - INFO - Verified VN network id 13 for VN 29920874-38d7-4e76-81d9-931fb87a1857 2025-10-02 03:48:46,760 - INFO - Verifications in API Server for VN ctest-vn1-94725148 passed 2025-10-02 03:48:46,760 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/29920874-38d7-4e76-81d9-931fb87a1857 2025-10-02 03:48:46,771 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/66cade29-e9b2-46d4-8eb4-75fcb4c81bee 2025-10-02 03:48:46,782 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/66cade29-e9b2-46d4-8eb4-75fcb4c81bee 2025-10-02 03:48:46,790 - DEBUG - Requesting: http://10.0.0.56:8082/route-target/8e8f481c-7398-4d6a-9cb6-0c036f63a80e 2025-10-02 03:48:46,802 - DEBUG - Control-node 10.0.0.56 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-35143319', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '2995465997417664118', 'uuid-lslong': '9356671465287194711'}, 'enable': 'true', 'created': '2025-10-02T03:48:45', 'last-modified': '2025-10-02T03:48:46', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.734425'} 2025-10-02 03:48:46,807 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-02 03:48:46,812 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-35143319', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '2995465997417664118', 'uuid-lslong': '9356671465287194711'}, 'enable': 'true', 'created': '2025-10-02T03:48:45', 'last-modified': '2025-10-02T03:48:46', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.745015'} 2025-10-02 03:48:46,818 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-02 03:48:46,823 - DEBUG - Control-node 10.0.0.49 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-35143319', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '2995465997417664118', 'uuid-lslong': '9356671465287194711'}, 'enable': 'true', 'created': '2025-10-02T03:48:45', 'last-modified': '2025-10-02T03:48:46', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.757963'} 2025-10-02 03:48:46,829 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-02 03:48:46,829 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-94725148 passed 2025-10-02 03:48:46,830 - DEBUG - ====Verifying policy data for ctest-vn1-94725148 in API_Server ====== 2025-10-02 03:48:46,830 - DEBUG - Requesting: http://10.0.0.56:8082/domains 2025-10-02 03:48:46,836 - DEBUG - Requesting: http://10.0.0.56:8082/domain/b0543e69-2184-4be1-8e23-ac5840233bc6 2025-10-02 03:48:46,846 - DEBUG - Requesting: http://10.0.0.56:8082/project/7612bb4f-d0d8-4c4e-9c49-460de03b6971 2025-10-02 03:48:46,942 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/29920874-38d7-4e76-81d9-931fb87a1857 2025-10-02 03:48:46,950 - DEBUG - =>VN ctest-vn1-94725148 has no policy to be verified 2025-10-02 03:48:46,950 - DEBUG - Verifying the vn in opserver 2025-10-02 03:48:46,950 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 virtual network link through opserver 10.0.0.56 2025-10-02 03:48:46,951 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-networks 2025-10-02 03:48:47,246 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-networks 2025-10-02 03:48:47,334 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148', 'href': 'http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148?flat'} 2025-10-02 03:48:47,334 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 virtual network link through opserver 10.0.0.38 2025-10-02 03:48:47,334 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-10-02 03:48:47,655 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-10-02 03:48:47,730 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148?flat'} 2025-10-02 03:48:47,730 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 virtual network link through opserver 10.0.0.49 2025-10-02 03:48:47,730 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-networks 2025-10-02 03:48:48,029 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-networks 2025-10-02 03:48:48,086 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148', 'href': 'http://10.0.0.49:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148?flat'} 2025-10-02 03:48:48,086 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 is found in opserver 2025-10-02 03:48:48,105 - DEBUG - Do not have enough data to verify VN in agent 2025-10-02 03:48:48,114 - DEBUG - VRF ids for VN ctest-vn1-94725148: {} 2025-10-02 03:48:48,171 - DEBUG - Services list from nova: [, , , ] 2025-10-02 03:48:49,204 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4210-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4210-2) 2025-10-02 03:48:50,475 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4210-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4210-1) 2025-10-02 03:48:50,475 - INFO - Waiting for VM ctest-vn1_vm1-69086009 to be up.. 2025-10-02 03:48:50,551 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-10-02 03:48:55,631 - DEBUG - VM is in ACTIVE state now 2025-10-02 03:48:55,631 - INFO - VM name : ctest-vn1_vm1-69086009 2025-10-02 03:48:55,723 - DEBUG - VM ctest-vn1_vm1-69086009 ID is 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 2025-10-02 03:48:55,750 - DEBUG - VM ctest-vn1_vm1-69086009 launched on Node an-jenkins-deploy-platform-ansible-os-4210-2 2025-10-02 03:48:55,831 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/1c7d726c-2305-4d2e-b8bb-a6f4997443a3 2025-10-02 03:48:56,131 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/1c7d726c-2305-4d2e-b8bb-a6f4997443a3 2025-10-02 03:48:56,163 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine-interface/43a7e90f-1471-4002-a318-613e89011a3f 2025-10-02 03:48:59,338 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1008ms') 2025-10-02 03:48:59,338 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-69086009 failed! 2025-10-02 03:48:59,352 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 is 192.168.1.1 and allocation pool is NOT set 2025-10-02 03:49:03,428 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2025-10-02 03:49:03,428 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-69086009 failed! 2025-10-02 03:49:03,444 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 is 192.168.1.1 and allocation pool is NOT set 2025-10-02 03:49:07,527 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2025-10-02 03:49:07,528 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-69086009 failed! 2025-10-02 03:49:07,542 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 is 192.168.1.1 and allocation pool is NOT set 2025-10-02 03:49:11,625 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-10-02 03:49:11,625 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-69086009 failed! 2025-10-02 03:49:11,642 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 is 192.168.1.1 and allocation pool is NOT set 2025-10-02 03:49:15,716 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-10-02 03:49:15,716 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-69086009 failed! 2025-10-02 03:49:15,731 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 is 192.168.1.1 and allocation pool is NOT set 2025-10-02 03:49:19,810 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2025-10-02 03:49:19,811 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-69086009 failed! 2025-10-02 03:49:19,824 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 is 192.168.1.1 and allocation pool is NOT set 2025-10-02 03:49:23,912 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1020ms') 2025-10-02 03:49:23,912 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-69086009 failed! 2025-10-02 03:49:23,927 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 is 192.168.1.1 and allocation pool is NOT set 2025-10-02 03:49:28,006 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2025-10-02 03:49:28,006 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-69086009 failed! 2025-10-02 03:49:28,023 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 is 192.168.1.1 and allocation pool is NOT set 2025-10-02 03:49:30,095 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=10.5 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=8.24 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 8.240/9.376/10.512/1.136 ms') 2025-10-02 03:49:30,095 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-69086009 passed 2025-10-02 03:49:30,166 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:49:30,166 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-69086009, IP 192.168.1.3, Port 22 2025-10-02 03:49:30,234 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-02 03:49:30,330 - DEBUG - VM ctest-vn1_vm1-69086009 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-02 03:49:35,331 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:49:35,331 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-69086009, IP 192.168.1.3, Port 22 2025-10-02 03:49:35,398 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-02 03:49:35,486 - DEBUG - VM ctest-vn1_vm1-69086009 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-02 03:49:40,487 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:49:40,487 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-69086009, IP 192.168.1.3, Port 22 2025-10-02 03:49:40,553 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-02 03:49:40,637 - DEBUG - VM ctest-vn1_vm1-69086009 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-02 03:49:45,637 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:49:45,638 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-69086009, IP 192.168.1.3, Port 22 2025-10-02 03:49:45,705 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-02 03:49:45,793 - DEBUG - VM ctest-vn1_vm1-69086009 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-02 03:49:50,794 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:49:50,795 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-69086009, IP 192.168.1.3, Port 22 2025-10-02 03:49:50,977 - DEBUG - VM ctest-vn1_vm1-69086009 is ready for SSH connections 2025-10-02 03:49:50,977 - INFO - Waiting for VM ctest-vn1_vm2-49855519 to be up.. 2025-10-02 03:49:51,062 - DEBUG - VM is in ACTIVE state now 2025-10-02 03:49:51,062 - INFO - VM name : ctest-vn1_vm2-49855519 2025-10-02 03:49:51,138 - DEBUG - VM ctest-vn1_vm2-49855519 ID is 29752b28-03ff-401a-908b-8ce091302d5f 2025-10-02 03:49:51,138 - DEBUG - VM ctest-vn1_vm2-49855519 launched on Node an-jenkins-deploy-platform-ansible-os-4210-1 2025-10-02 03:49:51,222 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/29752b28-03ff-401a-908b-8ce091302d5f 2025-10-02 03:49:51,232 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine-interface/1049ca0d-0576-4fdd-a425-069f07a521ff 2025-10-02 03:49:52,389 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=6.54 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=1.09 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 1.093/3.818/6.543/2.725 ms') 2025-10-02 03:49:52,389 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm2-49855519 passed 2025-10-02 03:49:52,460 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:49:52,460 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-49855519, IP 192.168.1.4, Port 22 2025-10-02 03:49:52,621 - DEBUG - VM ctest-vn1_vm2-49855519 is ready for SSH connections 2025-10-02 03:49:52,621 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:49:52,621 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-10-02 03:49:52,621 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4 2025-10-02 03:50:15,700 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data. 64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=15.0 ms 64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.55 ms 64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.42 ms --- 192.168.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.427/6.019/15.075/6.403 ms 2025-10-02 03:50:15,700 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-69086009 passed 2025-10-02 03:50:15,700 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:50:15,700 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.33, gateway password: c0ntrail123 2025-10-02 03:50:15,700 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3 2025-10-02 03:50:53,424 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. 64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=9.37 ms 64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.01 ms 64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.27 ms --- 192.168.1.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2005ms rtt min/avg/max/mdev = 1.012/3.886/9.374/3.882 ms 2025-10-02 03:50:53,425 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-49855519 passed 2025-10-02 03:50:53,433 - INFO - Active control node from the Agent 10.0.0.130 is 10.0.0.56 2025-10-02 03:50:53,434 - INFO - Stoping the Control service in 10.0.0.56 2025-10-02 03:50:54,038 - INFO - 10.0.0.56 2025-10-02 03:50:54,038 - INFO - control:inactive 2025-10-02 03:50:54,038 - INFO - Contrail services control are down on nodes 10.0.0.56 2025-10-02 03:50:59,045 - INFO - Active control node from the Agent 10.0.0.130 is 10.0.0.38 2025-10-02 03:50:59,045 - INFO - Starting the Control service in 10.0.0.56 2025-10-02 03:50:59,422 - INFO - 10.0.0.56 2025-10-02 03:50:59,425 - INFO - control:initializing 2025-10-02 03:50:59,425 - DEBUG - defaultdict(, {'10.0.0.56': {'control': {'status': 'initializing', 'description': None}}}) 2025-10-02 03:50:59,425 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-02 03:51:04,534 - INFO - 10.0.0.56 2025-10-02 03:51:04,541 - INFO - control:active 2025-10-02 03:51:04,541 - INFO - Contrail services ['control'] are up on nodes 10.0.0.56 2025-10-02 03:51:09,671 - DEBUG - VM is in ACTIVE state now 2025-10-02 03:51:09,671 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:51:09,671 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/1c7d726c-2305-4d2e-b8bb-a6f4997443a3 2025-10-02 03:51:09,681 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine-interface/43a7e90f-1471-4002-a318-613e89011a3f 2025-10-02 03:51:09,692 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/1c7d726c-2305-4d2e-b8bb-a6f4997443a3 2025-10-02 03:51:09,701 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine-interface/43a7e90f-1471-4002-a318-613e89011a3f 2025-10-02 03:51:09,712 - DEBUG - Requesting: http://10.0.0.49:8082/instance-ip/d233b30e-b152-4935-b958-8995d4167bb3 2025-10-02 03:51:09,723 - DEBUG - Verifying in api server 10.0.0.49 2025-10-02 03:51:09,723 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/1c7d726c-2305-4d2e-b8bb-a6f4997443a3 2025-10-02 03:51:09,732 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine-interface/43a7e90f-1471-4002-a318-613e89011a3f 2025-10-02 03:51:09,744 - DEBUG - Requesting: http://10.0.0.49:8082/instance-ip/d233b30e-b152-4935-b958-8995d4167bb3 2025-10-02 03:51:09,752 - INFO - VM ctest-vn1_vm1-69086009 verfication in all API Servers passed 2025-10-02 03:51:09,772 - DEBUG - VM ctest-vn1_vm1-69086009 Tap interface: {'index': '3', 'name': 'tap43a7e90f-14', 'uuid': '43a7e90f-1471-4002-a318-613e89011a3f', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148:ctest-vn1-94725148', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148', 'vm_uuid': '1c7d726c-2305-4d2e-b8bb-a6f4997443a3', 'vm_name': 'ctest-vn1_vm1-69086009', 'ip_addr': '192.168.1.3', 'mac_addr': '02:43:a7:e9:0f:14', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '165', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-35143319:43a7e90f-1471-4002-a318-613e89011a3f', 'sg_uuid_list': ['44109285-648a-42a8-91b0-26de1a86fc15'], 'static_route_list': None, 'vm_project_uuid': '7612bb4f-d0d8-4c4e-9c49-460de03b6971', 'admin_state': 'Enabled', 'flow_key_idx': '26', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-02 03:51:09,772 - DEBUG - Agent 10.0.0.130 vrf name: default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148:ctest-vn1-94725148 2025-10-02 03:51:09,778 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148:ctest-vn1-94725148', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148', 'table_label': '-1', 'vxlan_id': '13', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.249:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-10-02 03:51:09,861 - DEBUG - VM ctest-vn1_vm1-69086009 : Tap interface tap43a7e90f-14 is set to Active 2025-10-02 03:51:09,861 - DEBUG - Tap interface tap43a7e90f-14 detail : {'index': '3', 'name': 'tap43a7e90f-14', 'uuid': '43a7e90f-1471-4002-a318-613e89011a3f', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148:ctest-vn1-94725148', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148', 'vm_uuid': '1c7d726c-2305-4d2e-b8bb-a6f4997443a3', 'vm_name': 'ctest-vn1_vm1-69086009', 'ip_addr': '192.168.1.3', 'mac_addr': '02:43:a7:e9:0f:14', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '165', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-35143319:43a7e90f-1471-4002-a318-613e89011a3f', 'sg_uuid_list': ['44109285-648a-42a8-91b0-26de1a86fc15'], 'static_route_list': None, 'vm_project_uuid': '7612bb4f-d0d8-4c4e-9c49-460de03b6971', 'admin_state': 'Enabled', 'flow_key_idx': '26', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-02 03:51:09,978 - DEBUG - Starting Layer 2 verification in Agent 2025-10-02 03:51:09,991 - DEBUG - Layer 2 path is seen for VM MAC 02:43:a7:e9:0f:14 in agent 10.0.0.130 2025-10-02 03:51:09,991 - DEBUG - Active layer 2 route in agent is present for VMI tap43a7e90f-14 2025-10-02 03:51:09,991 - DEBUG - L2 label(29) matches bw route table and interface table 2025-10-02 03:51:09,997 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 is found in Agent of node 10.0.0.33 2025-10-02 03:51:10,132 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 is consistent in agent 10.0.0.33 2025-10-02 03:51:10,133 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.33 2025-10-02 03:51:10,133 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 verification for VM ctest-vn1_vm1-69086009 in Agent 10.0.0.33 passed 2025-10-02 03:51:10,253 - DEBUG - Starting all layer 2 verification in agent 10.0.0.33 2025-10-02 03:51:10,268 - DEBUG - Route for VM MAC 02:43:a7:e9:0f:14 is consistent in agent 10.0.0.33 2025-10-02 03:51:10,276 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 is found in Agent of node 10.0.0.130 2025-10-02 03:51:10,407 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 is consistent in agent 10.0.0.130 2025-10-02 03:51:10,408 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.130 2025-10-02 03:51:10,408 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 verification for VM ctest-vn1_vm1-69086009 in Agent 10.0.0.130 passed 2025-10-02 03:51:10,516 - DEBUG - Starting all layer 2 verification in agent 10.0.0.130 2025-10-02 03:51:10,531 - DEBUG - Route for VM MAC 02:43:a7:e9:0f:14 is consistent in agent 10.0.0.130 2025-10-02 03:51:10,531 - INFO - VM ctest-vn1_vm1-69086009 verifications in Compute nodes passed 2025-10-02 03:51:10,765 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.33 2025-10-02 03:51:10,771 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.249, 24; Got 10.20.0.249,24 2025-10-02 03:51:10,771 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.33 2025-10-02 03:51:10,875 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.130 2025-10-02 03:51:10,880 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-10-02 03:51:10,880 - INFO - Validated routes of VM ctest-vn1_vm1-69086009 in all vrouters 2025-10-02 03:51:11,432 - DEBUG - Starting all layer2 verification in 10.0.0.38 Control Node 2025-10-02 03:51:11,556 - DEBUG - Layer2 route found for VM MAC 02:43:a7:e9:0f:14 in Control-node 10.0.0.38 2025-10-02 03:51:11,556 - DEBUG - L2 Label for VM ctest-vn1_vm1-69086009 same between Control-node 10.0.0.38 and Agent, Expected: 29, Seen: 29 2025-10-02 03:51:11,556 - DEBUG - Starting all layer2 verification in 10.0.0.56 Control Node 2025-10-02 03:51:11,678 - DEBUG - Layer2 route found for VM MAC 02:43:a7:e9:0f:14 in Control-node 10.0.0.56 2025-10-02 03:51:11,678 - DEBUG - L2 Label for VM ctest-vn1_vm1-69086009 same between Control-node 10.0.0.56 and Agent, Expected: 29, Seen: 29 2025-10-02 03:51:11,678 - INFO - Verification in Control-nodes for VM ctest-vn1_vm1-69086009 passed 2025-10-02 03:51:11,678 - DEBUG - Verifying the vm in opserver 2025-10-02 03:51:11,679 - DEBUG - Verifying in collector 10.0.0.49 ... 2025-10-02 03:51:11,679 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-machine/1c7d726c-2305-4d2e-b8bb-a6f4997443a3?flat 2025-10-02 03:51:11,694 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-35143319:43a7e90f-1471-4002-a318-613e89011a3f?flat 2025-10-02 03:51:11,710 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-35143319:43a7e90f-1471-4002-a318-613e89011a3f?flat 2025-10-02 03:51:11,767 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-machine/1c7d726c-2305-4d2e-b8bb-a6f4997443a3?flat 2025-10-02 03:51:11,820 - DEBUG - Verifying vm in vn uve 2025-10-02 03:51:11,820 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-35143319:43a7e90f-1471-4002-a318-613e89011a3f?flat 2025-10-02 03:51:11,874 - DEBUG - VM uve shows interface as {'label': 24, 'admin_state': True, 'l2_active': True, 'vm_uuid': '1c7d726c-2305-4d2e-b8bb-a6f4997443a3', 'vhostuser_mode': 0, 'is_health_check_active': True, 'tx_vlan': 65535, 'ip6_address': '::', 'fixed_ip4_list': ['192.168.1.3'], 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148', 'rx_vlan': 65535, 'active': True, 'ip_address': '192.168.1.3', 'uuid': '43a7e90f-1471-4002-a318-613e89011a3f', 'mac_address': '02:43:a7:e9:0f:14', 'port_mirror_enabled': False, 'vn_uuid': '29920874-38d7-4e76-81d9-931fb87a1857', 'ip6_active': False, 'ip4_active': True, 'gateway': '192.168.1.1', '__T': 1759377061170245, 'vm_name': 'ctest-vn1_vm1-69086009'} 2025-10-02 03:51:11,874 - DEBUG - VM uve shows ip address as ['192.168.1.3', '::'] 2025-10-02 03:51:11,874 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 2025-10-02 03:51:11,874 - INFO - Verifying through opserver in 10.0.0.56 2025-10-02 03:51:11,874 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148?flat 2025-10-02 03:51:11,899 - DEBUG - expected vm list 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 2025-10-02 03:51:11,899 - DEBUG - Extracted vm list ['1c7d726c-2305-4d2e-b8bb-a6f4997443a3'] 2025-10-02 03:51:11,899 - INFO - VM 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 is present in default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 2025-10-02 03:51:11,899 - DEBUG - Verifying vm in vrouter uve 2025-10-02 03:51:11,899 - DEBUG - Getting info from collector 10.0.0.49.. 2025-10-02 03:51:11,899 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-machine/1c7d726c-2305-4d2e-b8bb-a6f4997443a3?flat 2025-10-02 03:51:11,954 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4210-2'] 2025-10-02 03:51:11,954 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4210-2:Compute:contrail-vrouter-agent:0?flat 2025-10-02 03:51:11,986 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4210-2?flat 2025-10-02 03:51:12,017 - DEBUG - VM 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 is present in vrouter an-jenkins-deploy-platform-ansible-os-4210-2 uve 2025-10-02 03:51:12,017 - DEBUG - Validated that VM ctest-vn1_vm1-69086009 is in Vrouter an-jenkins-deploy-platform-ansible-os-4210-2 UVE 2025-10-02 03:51:12,017 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-10-02 03:51:12,023 - DEBUG - Expected tap interface of VM uuid 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 is default-domain:ctest-TestBasicVMVN0-35143319:43a7e90f-1471-4002-a318-613e89011a3f 2025-10-02 03:51:12,023 - DEBUG - Expected VN of VM uuid 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 is default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 2025-10-02 03:51:12,023 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4210-2:Compute:contrail-vrouter-agent:0?flat 2025-10-02 03:51:12,054 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4210-2?flat 2025-10-02 03:51:12,128 - DEBUG - VM 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 is present in vrouter an-jenkins-deploy-platform-ansible-os-4210-2 uve 2025-10-02 03:51:12,128 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-35143319:43a7e90f-1471-4002-a318-613e89011a3f of vm 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 is present in vrouter an-jenkins-deploy-platform-ansible-os-4210-2 uve 2025-10-02 03:51:12,128 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 of vm 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 is present in vrouter an-jenkins-deploy-platform-ansible-os-4210-2 uve 2025-10-02 03:51:12,128 - DEBUG - Verifying the 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 virtual network link through opserver 10.0.0.56 2025-10-02 03:51:12,128 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machines 2025-10-02 03:51:12,139 - DEBUG - VM link and name as {'name': '1c7d726c-2305-4d2e-b8bb-a6f4997443a3', 'href': 'http://10.0.0.56:8081/analytics/uves/virtual-machine/1c7d726c-2305-4d2e-b8bb-a6f4997443a3?flat'} 2025-10-02 03:51:12,139 - DEBUG - Verifying the 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 virtual network link through opserver 10.0.0.38 2025-10-02 03:51:12,139 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-10-02 03:51:12,185 - DEBUG - VM link and name as {'name': '1c7d726c-2305-4d2e-b8bb-a6f4997443a3', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-machine/1c7d726c-2305-4d2e-b8bb-a6f4997443a3?flat'} 2025-10-02 03:51:12,185 - DEBUG - Verifying the 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 virtual network link through opserver 10.0.0.49 2025-10-02 03:51:12,185 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-machines 2025-10-02 03:51:12,197 - DEBUG - VM link and name as {'name': '1c7d726c-2305-4d2e-b8bb-a6f4997443a3', 'href': 'http://10.0.0.49:8081/analytics/uves/virtual-machine/1c7d726c-2305-4d2e-b8bb-a6f4997443a3?flat'} 2025-10-02 03:51:12,198 - INFO - VM ctest-vn1_vm1-69086009 validations in Opserver passed 2025-10-02 03:51:12,270 - DEBUG - VM is in ACTIVE state now 2025-10-02 03:51:12,271 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:51:12,271 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/29752b28-03ff-401a-908b-8ce091302d5f 2025-10-02 03:51:12,280 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine-interface/1049ca0d-0576-4fdd-a425-069f07a521ff 2025-10-02 03:51:12,292 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/29752b28-03ff-401a-908b-8ce091302d5f 2025-10-02 03:51:12,299 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine-interface/1049ca0d-0576-4fdd-a425-069f07a521ff 2025-10-02 03:51:12,309 - DEBUG - Requesting: http://10.0.0.49:8082/instance-ip/07435e0d-fef8-4fb7-8d9c-bde2703c051c 2025-10-02 03:51:12,323 - DEBUG - Verifying in api server 10.0.0.49 2025-10-02 03:51:12,323 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/29752b28-03ff-401a-908b-8ce091302d5f 2025-10-02 03:51:12,331 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine-interface/1049ca0d-0576-4fdd-a425-069f07a521ff 2025-10-02 03:51:12,341 - DEBUG - Requesting: http://10.0.0.49:8082/instance-ip/07435e0d-fef8-4fb7-8d9c-bde2703c051c 2025-10-02 03:51:12,348 - INFO - VM ctest-vn1_vm2-49855519 verfication in all API Servers passed 2025-10-02 03:51:12,368 - DEBUG - VM ctest-vn1_vm2-49855519 Tap interface: {'index': '3', 'name': 'tap1049ca0d-05', 'uuid': '1049ca0d-0576-4fdd-a425-069f07a521ff', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148:ctest-vn1-94725148', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148', 'vm_uuid': '29752b28-03ff-401a-908b-8ce091302d5f', 'vm_name': 'ctest-vn1_vm2-49855519', 'ip_addr': '192.168.1.4', 'mac_addr': '02:10:49:ca:0d:05', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '123', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-35143319:1049ca0d-0576-4fdd-a425-069f07a521ff', 'sg_uuid_list': ['44109285-648a-42a8-91b0-26de1a86fc15'], 'static_route_list': None, 'vm_project_uuid': '7612bb4f-d0d8-4c4e-9c49-460de03b6971', 'admin_state': 'Enabled', 'flow_key_idx': '24', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-02 03:51:12,368 - DEBUG - Agent 10.0.0.33 vrf name: default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148:ctest-vn1-94725148 2025-10-02 03:51:12,374 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148:ctest-vn1-94725148', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148', 'table_label': '-1', 'vxlan_id': '13', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.252:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-10-02 03:51:12,453 - DEBUG - VM ctest-vn1_vm2-49855519 : Tap interface tap1049ca0d-05 is set to Active 2025-10-02 03:51:12,453 - DEBUG - Tap interface tap1049ca0d-05 detail : {'index': '3', 'name': 'tap1049ca0d-05', 'uuid': '1049ca0d-0576-4fdd-a425-069f07a521ff', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148:ctest-vn1-94725148', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148', 'vm_uuid': '29752b28-03ff-401a-908b-8ce091302d5f', 'vm_name': 'ctest-vn1_vm2-49855519', 'ip_addr': '192.168.1.4', 'mac_addr': '02:10:49:ca:0d:05', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '123', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-35143319:1049ca0d-0576-4fdd-a425-069f07a521ff', 'sg_uuid_list': ['44109285-648a-42a8-91b0-26de1a86fc15'], 'static_route_list': None, 'vm_project_uuid': '7612bb4f-d0d8-4c4e-9c49-460de03b6971', 'admin_state': 'Enabled', 'flow_key_idx': '24', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-02 03:51:12,563 - DEBUG - Starting Layer 2 verification in Agent 2025-10-02 03:51:12,573 - DEBUG - Layer 2 path is seen for VM MAC 02:10:49:ca:0d:05 in agent 10.0.0.33 2025-10-02 03:51:12,573 - DEBUG - Active layer 2 route in agent is present for VMI tap1049ca0d-05 2025-10-02 03:51:12,573 - DEBUG - L2 label(29) matches bw route table and interface table 2025-10-02 03:51:12,578 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 is found in Agent of node 10.0.0.33 2025-10-02 03:51:12,702 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 is consistent in agent 10.0.0.33 2025-10-02 03:51:12,702 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.33 2025-10-02 03:51:12,702 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 verification for VM ctest-vn1_vm2-49855519 in Agent 10.0.0.33 passed 2025-10-02 03:51:12,824 - DEBUG - Starting all layer 2 verification in agent 10.0.0.33 2025-10-02 03:51:12,835 - DEBUG - Route for VM MAC 02:10:49:ca:0d:05 is consistent in agent 10.0.0.33 2025-10-02 03:51:12,844 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 is found in Agent of node 10.0.0.130 2025-10-02 03:51:12,972 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 is consistent in agent 10.0.0.130 2025-10-02 03:51:12,972 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.130 2025-10-02 03:51:12,973 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 verification for VM ctest-vn1_vm2-49855519 in Agent 10.0.0.130 passed 2025-10-02 03:51:13,088 - DEBUG - Starting all layer 2 verification in agent 10.0.0.130 2025-10-02 03:51:13,098 - DEBUG - Route for VM MAC 02:10:49:ca:0d:05 is consistent in agent 10.0.0.130 2025-10-02 03:51:13,098 - INFO - VM ctest-vn1_vm2-49855519 verifications in Compute nodes passed 2025-10-02 03:51:13,329 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.33 2025-10-02 03:51:13,333 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-10-02 03:51:13,443 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.130 2025-10-02 03:51:13,449 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.252, 25; Got 10.20.0.252,25 2025-10-02 03:51:13,449 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.130 2025-10-02 03:51:13,449 - INFO - Validated routes of VM ctest-vn1_vm2-49855519 in all vrouters 2025-10-02 03:51:13,973 - DEBUG - Starting all layer2 verification in 10.0.0.38 Control Node 2025-10-02 03:51:14,102 - DEBUG - Layer2 route found for VM MAC 02:10:49:ca:0d:05 in Control-node 10.0.0.38 2025-10-02 03:51:14,102 - DEBUG - L2 Label for VM ctest-vn1_vm2-49855519 same between Control-node 10.0.0.38 and Agent, Expected: 29, Seen: 29 2025-10-02 03:51:14,102 - DEBUG - Starting all layer2 verification in 10.0.0.56 Control Node 2025-10-02 03:51:14,228 - DEBUG - Layer2 route found for VM MAC 02:10:49:ca:0d:05 in Control-node 10.0.0.56 2025-10-02 03:51:14,228 - DEBUG - L2 Label for VM ctest-vn1_vm2-49855519 same between Control-node 10.0.0.56 and Agent, Expected: 29, Seen: 29 2025-10-02 03:51:14,228 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-49855519 passed 2025-10-02 03:51:14,228 - DEBUG - Verifying the vm in opserver 2025-10-02 03:51:14,228 - DEBUG - Verifying in collector 10.0.0.49 ... 2025-10-02 03:51:14,228 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-machine/29752b28-03ff-401a-908b-8ce091302d5f?flat 2025-10-02 03:51:14,245 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-35143319:1049ca0d-0576-4fdd-a425-069f07a521ff?flat 2025-10-02 03:51:14,262 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-35143319:1049ca0d-0576-4fdd-a425-069f07a521ff?flat 2025-10-02 03:51:14,279 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-machine/29752b28-03ff-401a-908b-8ce091302d5f?flat 2025-10-02 03:51:14,297 - DEBUG - Verifying vm in vn uve 2025-10-02 03:51:14,297 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-35143319:1049ca0d-0576-4fdd-a425-069f07a521ff?flat 2025-10-02 03:51:14,315 - DEBUG - VM uve shows interface as {'__T': 1759376932517938, 'rx_vlan': 65535, 'active': True, 'label': 25, 'vn_uuid': '29920874-38d7-4e76-81d9-931fb87a1857', 'gateway': '192.168.1.1', 'ip4_active': True, 'vm_name': 'ctest-vn1_vm2-49855519', 'is_health_check_active': True, 'uuid': '1049ca0d-0576-4fdd-a425-069f07a521ff', 'ip6_active': False, 'ip_address': '192.168.1.4', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148', 'l2_active': True, 'admin_state': True, 'vm_uuid': '29752b28-03ff-401a-908b-8ce091302d5f', 'vhostuser_mode': 0, 'port_mirror_enabled': False, 'fixed_ip4_list': ['192.168.1.4'], 'ip6_address': '::', 'tx_vlan': 65535, 'mac_address': '02:10:49:ca:0d:05'} 2025-10-02 03:51:14,315 - DEBUG - VM uve shows ip address as ['192.168.1.4', '::'] 2025-10-02 03:51:14,315 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 2025-10-02 03:51:14,315 - INFO - Verifying through opserver in 10.0.0.56 2025-10-02 03:51:14,315 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148?flat 2025-10-02 03:51:14,342 - DEBUG - expected vm list 29752b28-03ff-401a-908b-8ce091302d5f 2025-10-02 03:51:14,342 - DEBUG - Extracted vm list ['29752b28-03ff-401a-908b-8ce091302d5f'] 2025-10-02 03:51:14,342 - INFO - VM 29752b28-03ff-401a-908b-8ce091302d5f is present in default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 2025-10-02 03:51:14,343 - DEBUG - Verifying vm in vrouter uve 2025-10-02 03:51:14,343 - DEBUG - Getting info from collector 10.0.0.49.. 2025-10-02 03:51:14,343 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-machine/29752b28-03ff-401a-908b-8ce091302d5f?flat 2025-10-02 03:51:14,359 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4210-1'] 2025-10-02 03:51:14,359 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4210-1:Compute:contrail-vrouter-agent:0?flat 2025-10-02 03:51:14,401 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4210-1?flat 2025-10-02 03:51:14,433 - DEBUG - VM 29752b28-03ff-401a-908b-8ce091302d5f is present in vrouter an-jenkins-deploy-platform-ansible-os-4210-1 uve 2025-10-02 03:51:14,433 - DEBUG - Validated that VM ctest-vn1_vm2-49855519 is in Vrouter an-jenkins-deploy-platform-ansible-os-4210-1 UVE 2025-10-02 03:51:14,433 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-10-02 03:51:14,448 - DEBUG - Expected tap interface of VM uuid 29752b28-03ff-401a-908b-8ce091302d5f is default-domain:ctest-TestBasicVMVN0-35143319:1049ca0d-0576-4fdd-a425-069f07a521ff 2025-10-02 03:51:14,448 - DEBUG - Expected VN of VM uuid 29752b28-03ff-401a-908b-8ce091302d5f is default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 2025-10-02 03:51:14,448 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4210-1:Compute:contrail-vrouter-agent:0?flat 2025-10-02 03:51:14,517 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4210-1?flat 2025-10-02 03:51:14,557 - DEBUG - VM 29752b28-03ff-401a-908b-8ce091302d5f is present in vrouter an-jenkins-deploy-platform-ansible-os-4210-1 uve 2025-10-02 03:51:14,557 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-35143319:1049ca0d-0576-4fdd-a425-069f07a521ff of vm 29752b28-03ff-401a-908b-8ce091302d5f is present in vrouter an-jenkins-deploy-platform-ansible-os-4210-1 uve 2025-10-02 03:51:14,557 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 of vm 29752b28-03ff-401a-908b-8ce091302d5f is present in vrouter an-jenkins-deploy-platform-ansible-os-4210-1 uve 2025-10-02 03:51:14,558 - DEBUG - Verifying the 29752b28-03ff-401a-908b-8ce091302d5f virtual network link through opserver 10.0.0.56 2025-10-02 03:51:14,558 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machines 2025-10-02 03:51:14,573 - DEBUG - VM link and name as {'name': '29752b28-03ff-401a-908b-8ce091302d5f', 'href': 'http://10.0.0.56:8081/analytics/uves/virtual-machine/29752b28-03ff-401a-908b-8ce091302d5f?flat'} 2025-10-02 03:51:14,573 - DEBUG - Verifying the 29752b28-03ff-401a-908b-8ce091302d5f virtual network link through opserver 10.0.0.38 2025-10-02 03:51:14,573 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-10-02 03:51:14,586 - DEBUG - VM link and name as {'name': '29752b28-03ff-401a-908b-8ce091302d5f', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-machine/29752b28-03ff-401a-908b-8ce091302d5f?flat'} 2025-10-02 03:51:14,586 - DEBUG - Verifying the 29752b28-03ff-401a-908b-8ce091302d5f virtual network link through opserver 10.0.0.49 2025-10-02 03:51:14,587 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-machines 2025-10-02 03:51:14,600 - DEBUG - VM link and name as {'name': '29752b28-03ff-401a-908b-8ce091302d5f', 'href': 'http://10.0.0.49:8081/analytics/uves/virtual-machine/29752b28-03ff-401a-908b-8ce091302d5f?flat'} 2025-10-02 03:51:14,600 - INFO - VM ctest-vn1_vm2-49855519 validations in Opserver passed 2025-10-02 03:51:14,600 - INFO - Checking the ping between the VM again 2025-10-02 03:51:14,600 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:51:14,601 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-10-02 03:51:14,601 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4 2025-10-02 03:51:17,099 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data. 64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=4.16 ms 64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.40 ms 64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=0.991 ms --- 192.168.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 0.991/2.185/4.164/1.409 ms 2025-10-02 03:51:17,099 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-69086009 passed 2025-10-02 03:51:17,100 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:51:17,100 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.33, gateway password: c0ntrail123 2025-10-02 03:51:17,100 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3 2025-10-02 03:51:19,546 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. 64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=2.70 ms 64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.80 ms 64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.00 ms --- 192.168.1.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 1.000/1.835/2.703/0.695 ms 2025-10-02 03:51:19,546 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-49855519 passed 2025-10-02 03:51:19,546 - INFO - Deleting VM ctest-vn1_vm2-49855519 2025-10-02 03:51:19,613 - DEBUG - Verifying in api server 10.0.0.49 2025-10-02 03:51:19,613 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/29752b28-03ff-401a-908b-8ce091302d5f 2025-10-02 03:51:19,621 - DEBUG - VM ID 29752b28-03ff-401a-908b-8ce091302d5f of VM ctest-vn1_vm2-49855519 is still found in API Server 2025-10-02 03:51:21,622 - DEBUG - Verifying in api server 10.0.0.49 2025-10-02 03:51:21,623 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/29752b28-03ff-401a-908b-8ce091302d5f 2025-10-02 03:51:21,630 - DEBUG - Response Code: 404 2025-10-02 03:51:21,631 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/29752b28-03ff-401a-908b-8ce091302d5f 2025-10-02 03:51:21,638 - DEBUG - Response Code: 404 2025-10-02 03:51:21,638 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/29752b28-03ff-401a-908b-8ce091302d5f 2025-10-02 03:51:21,643 - DEBUG - Response Code: 404 2025-10-02 03:51:21,644 - INFO - VM ctest-vn1_vm2-49855519 is fully removed in API-Server 2025-10-02 03:51:21,794 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-10-02 03:51:21,824 - DEBUG - Validated that vrouter 10.0.0.33 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 2025-10-02 03:51:21,929 - DEBUG - VM route 192.168.1.4 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 of compute 10.0.0.130 2025-10-02 03:51:21,932 - DEBUG - Validated that vrouter 10.0.0.130 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 2025-10-02 03:51:21,932 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 2025-10-02 03:51:21,932 - INFO - VM ctest-vn1_vm2-49855519 is removed in Compute, and routes are removed in all compute nodes 2025-10-02 03:51:22,179 - INFO - Routes for VM ctest-vn1_vm2-49855519 is removed in all control-nodes 2025-10-02 03:51:22,179 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4210-1:Compute:contrail-vrouter-agent:0?flat 2025-10-02 03:51:22,222 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4210-1?flat 2025-10-02 03:51:22,250 - DEBUG - VM 29752b28-03ff-401a-908b-8ce091302d5f is still present in vrouter an-jenkins-deploy-platform-ansible-os-4210-1 uve 2025-10-02 03:51:22,250 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-02 03:51:22,250 - DEBUG - interface for vm 29752b28-03ff-401a-908b-8ce091302d5f deleted from vrouter uve ... 2025-10-02 03:51:22,250 - DEBUG - Deleted interface not in error interface list ... 2025-10-02 03:51:25,251 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4210-1:Compute:contrail-vrouter-agent:0?flat 2025-10-02 03:51:25,301 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4210-1?flat 2025-10-02 03:51:25,353 - DEBUG - VM 29752b28-03ff-401a-908b-8ce091302d5f is not present in vrouter an-jenkins-deploy-platform-ansible-os-4210-1 uve 2025-10-02 03:51:25,353 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-02 03:51:25,353 - DEBUG - interface for vm 29752b28-03ff-401a-908b-8ce091302d5f deleted from vrouter uve ... 2025-10-02 03:51:25,353 - DEBUG - Deleted interface not in error interface list ... 2025-10-02 03:51:25,353 - DEBUG - Validated that VM 29752b28-03ff-401a-908b-8ce091302d5f is removed in Vrouter UVE 2025-10-02 03:51:25,353 - DEBUG - Verifying through opserver in 10.0.0.56 2025-10-02 03:51:25,353 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148?flat 2025-10-02 03:51:25,381 - DEBUG - 29752b28-03ff-401a-908b-8ce091302d5f BM not in default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 uve 2025-10-02 03:51:25,382 - DEBUG - Validated that VM 29752b28-03ff-401a-908b-8ce091302d5f is not present in the VN UVE 2025-10-02 03:51:25,382 - DEBUG - Verifying the 29752b28-03ff-401a-908b-8ce091302d5f virtual network link through opserver 10.0.0.56 2025-10-02 03:51:25,382 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machines 2025-10-02 03:51:25,396 - INFO - 29752b28-03ff-401a-908b-8ce091302d5f vm uve deleted from opserver 2025-10-02 03:51:25,396 - DEBUG - Verifying the 29752b28-03ff-401a-908b-8ce091302d5f virtual network link through opserver 10.0.0.38 2025-10-02 03:51:25,397 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-10-02 03:51:25,408 - INFO - 29752b28-03ff-401a-908b-8ce091302d5f vm uve deleted from opserver 2025-10-02 03:51:25,409 - DEBUG - Verifying the 29752b28-03ff-401a-908b-8ce091302d5f virtual network link through opserver 10.0.0.49 2025-10-02 03:51:25,409 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-machines 2025-10-02 03:51:25,420 - INFO - 29752b28-03ff-401a-908b-8ce091302d5f vm uve deleted from opserver 2025-10-02 03:51:25,421 - INFO - Validated that VM 29752b28-03ff-401a-908b-8ce091302d5f is removed from Opserver 2025-10-02 03:51:25,421 - INFO - Deleting VM ctest-vn1_vm1-69086009 2025-10-02 03:51:25,532 - DEBUG - Verifying in api server 10.0.0.49 2025-10-02 03:51:25,532 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/1c7d726c-2305-4d2e-b8bb-a6f4997443a3 2025-10-02 03:51:25,541 - DEBUG - VM ID 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 of VM ctest-vn1_vm1-69086009 is still found in API Server 2025-10-02 03:51:27,542 - DEBUG - Verifying in api server 10.0.0.49 2025-10-02 03:51:27,542 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/1c7d726c-2305-4d2e-b8bb-a6f4997443a3 2025-10-02 03:51:27,549 - DEBUG - Response Code: 404 2025-10-02 03:51:27,549 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/1c7d726c-2305-4d2e-b8bb-a6f4997443a3 2025-10-02 03:51:27,555 - DEBUG - Response Code: 404 2025-10-02 03:51:27,555 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/1c7d726c-2305-4d2e-b8bb-a6f4997443a3 2025-10-02 03:51:27,560 - DEBUG - Response Code: 404 2025-10-02 03:51:27,560 - INFO - VM ctest-vn1_vm1-69086009 is fully removed in API-Server 2025-10-02 03:51:27,713 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-10-02 03:51:27,756 - DEBUG - Validated that vrouter 10.0.0.33 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 2025-10-02 03:51:27,765 - DEBUG - Validated that vrouter 10.0.0.130 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 2025-10-02 03:51:27,765 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 2025-10-02 03:51:27,766 - INFO - VM ctest-vn1_vm1-69086009 is removed in Compute, and routes are removed in all compute nodes 2025-10-02 03:51:27,993 - INFO - Routes for VM ctest-vn1_vm1-69086009 is removed in all control-nodes 2025-10-02 03:51:27,993 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4210-2:Compute:contrail-vrouter-agent:0?flat 2025-10-02 03:51:28,028 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4210-2?flat 2025-10-02 03:51:28,057 - DEBUG - VM 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4210-2 uve 2025-10-02 03:51:28,057 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-02 03:51:28,058 - DEBUG - interface for vm 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 deleted from vrouter uve ... 2025-10-02 03:51:28,058 - DEBUG - Deleted interface not in error interface list ... 2025-10-02 03:51:31,058 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4210-2:Compute:contrail-vrouter-agent:0?flat 2025-10-02 03:51:31,094 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4210-2?flat 2025-10-02 03:51:31,124 - DEBUG - VM 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4210-2 uve 2025-10-02 03:51:31,124 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-02 03:51:31,124 - DEBUG - interface for vm 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 deleted from vrouter uve ... 2025-10-02 03:51:31,124 - DEBUG - Deleted interface not in error interface list ... 2025-10-02 03:51:34,126 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4210-2:Compute:contrail-vrouter-agent:0?flat 2025-10-02 03:51:34,157 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4210-2?flat 2025-10-02 03:51:34,186 - DEBUG - VM 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 is not present in vrouter an-jenkins-deploy-platform-ansible-os-4210-2 uve 2025-10-02 03:51:34,186 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-02 03:51:34,186 - DEBUG - interface for vm 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 deleted from vrouter uve ... 2025-10-02 03:51:34,186 - DEBUG - Deleted interface not in error interface list ... 2025-10-02 03:51:34,186 - DEBUG - Validated that VM 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 is removed in Vrouter UVE 2025-10-02 03:51:34,186 - DEBUG - Verifying through opserver in 10.0.0.56 2025-10-02 03:51:34,186 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148?flat 2025-10-02 03:51:34,209 - DEBUG - 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 BM not in default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-94725148 uve 2025-10-02 03:51:34,209 - DEBUG - Validated that VM 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 is not present in the VN UVE 2025-10-02 03:51:34,209 - DEBUG - Verifying the 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 virtual network link through opserver 10.0.0.56 2025-10-02 03:51:34,210 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machines 2025-10-02 03:51:34,222 - INFO - 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 vm uve deleted from opserver 2025-10-02 03:51:34,222 - DEBUG - Verifying the 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 virtual network link through opserver 10.0.0.38 2025-10-02 03:51:34,222 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-10-02 03:51:34,234 - INFO - 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 vm uve deleted from opserver 2025-10-02 03:51:34,234 - DEBUG - Verifying the 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 virtual network link through opserver 10.0.0.49 2025-10-02 03:51:34,234 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-machines 2025-10-02 03:51:34,246 - INFO - 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 vm uve deleted from opserver 2025-10-02 03:51:34,246 - INFO - Validated that VM 1c7d726c-2305-4d2e-b8bb-a6f4997443a3 is removed from Opserver 2025-10-02 03:51:34,246 - INFO - Deleting VN ctest-vn1-94725148 2025-10-02 03:51:34,454 - DEBUG - Response for deleting network () 2025-10-02 03:51:34,454 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/66cade29-e9b2-46d4-8eb4-75fcb4c81bee 2025-10-02 03:51:34,460 - DEBUG - Response Code: 404 2025-10-02 03:51:34,460 - DEBUG - Requesting: http://10.0.0.56:8082/domains 2025-10-02 03:51:34,466 - DEBUG - Requesting: http://10.0.0.56:8082/domain/b0543e69-2184-4be1-8e23-ac5840233bc6 2025-10-02 03:51:34,514 - DEBUG - Requesting: http://10.0.0.56:8082/project/7612bb4f-d0d8-4c4e-9c49-460de03b6971 2025-10-02 03:51:34,602 - INFO - Validated that VN ctest-vn1-94725148 is not found in API Server 2025-10-02 03:51:34,612 - DEBUG - VN ctest-vn1-94725148 is not present in Agent 10.0.0.33 2025-10-02 03:51:34,621 - DEBUG - VN ctest-vn1-94725148 is not present in Agent 10.0.0.130 2025-10-02 03:51:34,621 - INFO - Validated that VN ctest-vn1-94725148 is not in any agent 2025-10-02 03:51:34,652 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-94725148 info 2025-10-02 03:51:35,903 - DEBUG - Skipping xmpp flap check 2025-10-02 03:51:35,903 - INFO - END TEST : test_control_node_switchover : PASSED[0:02:52] 2025-10-02 03:51:35,903 - INFO - -------------------------------------------------------------------------------- 2025-10-02 03:51:35,906 - INFO - ================================================================================ 2025-10-02 03:51:35,906 - INFO - STARTING TEST : test_process_restart_in_policy_between_vns 2025-10-02 03:51:35,906 - INFO - TEST DESCRIPTION : Test to validate that with policy having rule to check icmp fwding between VMs on different VNs , ping between VMs should pass with process restarts 1. Pick 2 VN's from resource pool which has one VM each 2. Create policy with icmp allow rule between those VN's and bind it networks 3. Ping from one VM to another VM 4. Restart process 'vrouter' and 'control' on setup 5. Ping again between VM's after process restart Pass criteria: Step 2,3,4 and 5 should pass 2025-10-02 03:51:37,147 - DEBUG - Skipping xmpp flap check 2025-10-02 03:51:37,147 - INFO - Initial checks done. Running the testcase now 2025-10-02 03:51:37,147 - INFO - 2025-10-02 03:51:37,153 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-10-02 03:51:37,442 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-10-02 03:51:37,998 - INFO - Created VN ctest-vn1-80182616, UUID :1c862540-f7c7-471b-b33a-77cef607922f 2025-10-02 03:51:38,310 - DEBUG - Requesting: http://10.0.0.56:8082/domains 2025-10-02 03:51:38,316 - DEBUG - Requesting: http://10.0.0.56:8082/domain/b0543e69-2184-4be1-8e23-ac5840233bc6 2025-10-02 03:51:38,326 - DEBUG - Requesting: http://10.0.0.56:8082/project/7612bb4f-d0d8-4c4e-9c49-460de03b6971 2025-10-02 03:51:38,446 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/1c862540-f7c7-471b-b33a-77cef607922f 2025-10-02 03:51:38,459 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/1c862540-f7c7-471b-b33a-77cef607922f 2025-10-02 03:51:38,468 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/bff0383e-27bf-4de3-a2c9-15fd9d24118f 2025-10-02 03:51:38,477 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/bff0383e-27bf-4de3-a2c9-15fd9d24118f 2025-10-02 03:51:38,484 - DEBUG - Requesting: http://10.0.0.56:8082/route-target/8b8faaea-2f0f-4ca2-ab2f-34142c532392 2025-10-02 03:51:38,491 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-02 03:51:38,492 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/1c862540-f7c7-471b-b33a-77cef607922f 2025-10-02 03:51:38,501 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/bff0383e-27bf-4de3-a2c9-15fd9d24118f 2025-10-02 03:51:38,516 - INFO - Verified VN network id 13 for VN 1c862540-f7c7-471b-b33a-77cef607922f 2025-10-02 03:51:38,516 - INFO - Verifications in API Server for VN ctest-vn1-80182616 passed 2025-10-02 03:51:38,516 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/1c862540-f7c7-471b-b33a-77cef607922f 2025-10-02 03:51:38,528 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/bff0383e-27bf-4de3-a2c9-15fd9d24118f 2025-10-02 03:51:38,536 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/bff0383e-27bf-4de3-a2c9-15fd9d24118f 2025-10-02 03:51:38,543 - DEBUG - Requesting: http://10.0.0.56:8082/route-target/8b8faaea-2f0f-4ca2-ab2f-34142c532392 2025-10-02 03:51:38,555 - DEBUG - Control-node 10.0.0.56 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-35143319', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '2055371240906376987', 'uuid-lslong': '12914766612212978223'}, 'enable': 'true', 'created': '2025-10-02T03:51:37', 'last-modified': '2025-10-02T03:51:38', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.418510'} 2025-10-02 03:51:38,561 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-02 03:51:38,566 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-35143319', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '2055371240906376987', 'uuid-lslong': '12914766612212978223'}, 'enable': 'true', 'created': '2025-10-02T03:51:37', 'last-modified': '2025-10-02T03:51:38', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.428121'} 2025-10-02 03:51:38,572 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-02 03:51:38,579 - DEBUG - Control-node 10.0.0.49 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-35143319', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '2055371240906376987', 'uuid-lslong': '12914766612212978223'}, 'enable': 'true', 'created': '2025-10-02T03:51:37', 'last-modified': '2025-10-02T03:51:38', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.439796'} 2025-10-02 03:51:38,587 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-02 03:51:38,587 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-80182616 passed 2025-10-02 03:51:38,587 - DEBUG - ====Verifying policy data for ctest-vn1-80182616 in API_Server ====== 2025-10-02 03:51:38,587 - DEBUG - Requesting: http://10.0.0.56:8082/domains 2025-10-02 03:51:38,593 - DEBUG - Requesting: http://10.0.0.56:8082/domain/b0543e69-2184-4be1-8e23-ac5840233bc6 2025-10-02 03:51:38,605 - DEBUG - Requesting: http://10.0.0.56:8082/project/7612bb4f-d0d8-4c4e-9c49-460de03b6971 2025-10-02 03:51:38,699 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/1c862540-f7c7-471b-b33a-77cef607922f 2025-10-02 03:51:38,707 - DEBUG - =>VN ctest-vn1-80182616 has no policy to be verified 2025-10-02 03:51:38,708 - DEBUG - Verifying the vn in opserver 2025-10-02 03:51:38,708 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 virtual network link through opserver 10.0.0.56 2025-10-02 03:51:38,708 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-networks 2025-10-02 03:51:38,718 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616', 'href': 'http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616?flat'} 2025-10-02 03:51:38,718 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 virtual network link through opserver 10.0.0.38 2025-10-02 03:51:38,718 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-10-02 03:51:38,730 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616?flat'} 2025-10-02 03:51:38,730 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 virtual network link through opserver 10.0.0.49 2025-10-02 03:51:38,730 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-networks 2025-10-02 03:51:38,746 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616', 'href': 'http://10.0.0.49:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616?flat'} 2025-10-02 03:51:38,746 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 is found in opserver 2025-10-02 03:51:38,765 - DEBUG - Do not have enough data to verify VN in agent 2025-10-02 03:51:38,773 - DEBUG - VRF ids for VN ctest-vn1-80182616: {} 2025-10-02 03:51:39,418 - INFO - Created VN ctest-vn2-20688640, UUID :13750373-7837-4fb1-ab88-a4e0114a9102 2025-10-02 03:51:39,738 - DEBUG - Requesting: http://10.0.0.56:8082/domains 2025-10-02 03:51:39,745 - DEBUG - Requesting: http://10.0.0.56:8082/domain/b0543e69-2184-4be1-8e23-ac5840233bc6 2025-10-02 03:51:39,758 - DEBUG - Requesting: http://10.0.0.56:8082/project/7612bb4f-d0d8-4c4e-9c49-460de03b6971 2025-10-02 03:51:39,849 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/13750373-7837-4fb1-ab88-a4e0114a9102 2025-10-02 03:51:39,865 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/13750373-7837-4fb1-ab88-a4e0114a9102 2025-10-02 03:51:39,874 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/63da8ad1-305a-4d44-b7b6-3496eaa3bdc6 2025-10-02 03:51:39,883 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/63da8ad1-305a-4d44-b7b6-3496eaa3bdc6 2025-10-02 03:51:39,889 - DEBUG - Requesting: http://10.0.0.56:8082/route-target/a826a643-6b0e-4f98-9076-76fe9db42fb7 2025-10-02 03:51:39,895 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-10-02 03:51:39,895 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/13750373-7837-4fb1-ab88-a4e0114a9102 2025-10-02 03:51:39,902 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/63da8ad1-305a-4d44-b7b6-3496eaa3bdc6 2025-10-02 03:51:39,916 - INFO - Verified VN network id 14 for VN 13750373-7837-4fb1-ab88-a4e0114a9102 2025-10-02 03:51:39,916 - INFO - Verifications in API Server for VN ctest-vn2-20688640 passed 2025-10-02 03:51:39,916 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/13750373-7837-4fb1-ab88-a4e0114a9102 2025-10-02 03:51:39,927 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/63da8ad1-305a-4d44-b7b6-3496eaa3bdc6 2025-10-02 03:51:39,934 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/63da8ad1-305a-4d44-b7b6-3496eaa3bdc6 2025-10-02 03:51:39,943 - DEBUG - Requesting: http://10.0.0.56:8082/route-target/a826a643-6b0e-4f98-9076-76fe9db42fb7 2025-10-02 03:51:39,955 - DEBUG - Control-node 10.0.0.56 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-35143319', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '1402030653468790705', 'uuid-lslong': '12360310459588055298'}, 'enable': 'true', 'created': '2025-10-02T03:51:39', 'last-modified': '2025-10-02T03:51:39', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.389856'} 2025-10-02 03:51:39,962 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-10-02 03:51:39,967 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-35143319', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '1402030653468790705', 'uuid-lslong': '12360310459588055298'}, 'enable': 'true', 'created': '2025-10-02T03:51:39', 'last-modified': '2025-10-02T03:51:39', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.402660'} 2025-10-02 03:51:39,973 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-10-02 03:51:39,977 - DEBUG - Control-node 10.0.0.49 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-35143319', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '1402030653468790705', 'uuid-lslong': '12360310459588055298'}, 'enable': 'true', 'created': '2025-10-02T03:51:39', 'last-modified': '2025-10-02T03:51:39', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.412981'} 2025-10-02 03:51:39,983 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-10-02 03:51:39,983 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn2-20688640 passed 2025-10-02 03:51:39,983 - DEBUG - ====Verifying policy data for ctest-vn2-20688640 in API_Server ====== 2025-10-02 03:51:39,983 - DEBUG - Requesting: http://10.0.0.56:8082/domains 2025-10-02 03:51:39,989 - DEBUG - Requesting: http://10.0.0.56:8082/domain/b0543e69-2184-4be1-8e23-ac5840233bc6 2025-10-02 03:51:39,999 - DEBUG - Requesting: http://10.0.0.56:8082/project/7612bb4f-d0d8-4c4e-9c49-460de03b6971 2025-10-02 03:51:40,089 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/13750373-7837-4fb1-ab88-a4e0114a9102 2025-10-02 03:51:40,099 - DEBUG - =>VN ctest-vn2-20688640 has no policy to be verified 2025-10-02 03:51:40,099 - DEBUG - Verifying the vn in opserver 2025-10-02 03:51:40,099 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640 virtual network link through opserver 10.0.0.56 2025-10-02 03:51:40,100 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-networks 2025-10-02 03:51:40,111 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640', 'href': 'http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640?flat'} 2025-10-02 03:51:40,111 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640 virtual network link through opserver 10.0.0.38 2025-10-02 03:51:40,111 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-10-02 03:51:40,124 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640?flat'} 2025-10-02 03:51:40,124 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640 virtual network link through opserver 10.0.0.49 2025-10-02 03:51:40,124 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-networks 2025-10-02 03:51:40,136 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640', 'href': 'http://10.0.0.49:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640?flat'} 2025-10-02 03:51:40,137 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640 is found in opserver 2025-10-02 03:51:40,156 - DEBUG - Do not have enough data to verify VN in agent 2025-10-02 03:51:40,165 - DEBUG - VRF ids for VN ctest-vn2-20688640: {} 2025-10-02 03:51:41,630 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4210-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4210-2) 2025-10-02 03:51:42,924 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4210-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4210-1) 2025-10-02 03:51:42,924 - INFO - Waiting for VM ctest-vn1_vm1-70599077 to be up.. 2025-10-02 03:51:43,008 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-10-02 03:51:48,081 - DEBUG - VM is in ACTIVE state now 2025-10-02 03:51:48,081 - INFO - VM name : ctest-vn1_vm1-70599077 2025-10-02 03:51:48,151 - DEBUG - VM ctest-vn1_vm1-70599077 ID is 17b8f325-e4a0-4d89-a694-9aefc9763967 2025-10-02 03:51:48,151 - DEBUG - VM ctest-vn1_vm1-70599077 launched on Node an-jenkins-deploy-platform-ansible-os-4210-2 2025-10-02 03:51:48,221 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/17b8f325-e4a0-4d89-a694-9aefc9763967 2025-10-02 03:51:48,230 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine-interface/95e9f70b-123f-4da8-8047-f214ea63ebd4 2025-10-02 03:51:51,400 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-10-02 03:51:51,400 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-70599077 failed! 2025-10-02 03:51:51,415 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 is 192.168.1.254 and allocation pool is NOT set 2025-10-02 03:51:55,493 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-10-02 03:51:55,493 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-70599077 failed! 2025-10-02 03:51:55,506 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 is 192.168.1.254 and allocation pool is NOT set 2025-10-02 03:51:59,590 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2025-10-02 03:51:59,590 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-70599077 failed! 2025-10-02 03:51:59,603 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 is 192.168.1.254 and allocation pool is NOT set 2025-10-02 03:52:03,690 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2025-10-02 03:52:03,690 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-70599077 failed! 2025-10-02 03:52:03,707 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 is 192.168.1.254 and allocation pool is NOT set 2025-10-02 03:52:07,779 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms') 2025-10-02 03:52:07,780 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-70599077 failed! 2025-10-02 03:52:07,793 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 is 192.168.1.254 and allocation pool is NOT set 2025-10-02 03:52:11,876 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2025-10-02 03:52:11,876 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-70599077 failed! 2025-10-02 03:52:11,891 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 is 192.168.1.254 and allocation pool is NOT set 2025-10-02 03:52:15,974 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2025-10-02 03:52:15,974 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-70599077 failed! 2025-10-02 03:52:15,987 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 is 192.168.1.254 and allocation pool is NOT set 2025-10-02 03:52:20,071 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2025-10-02 03:52:20,071 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-70599077 failed! 2025-10-02 03:52:20,086 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 is 192.168.1.254 and allocation pool is NOT set 2025-10-02 03:52:24,162 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=13.4 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 1 received, 50% packet loss, time 1019ms\r\nrtt min/avg/max/mdev = 13.373/13.373/13.373/0.000 ms') 2025-10-02 03:52:24,162 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-70599077 passed 2025-10-02 03:52:24,231 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:52:24,231 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-70599077, IP 192.168.1.252, Port 22 2025-10-02 03:52:24,297 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-02 03:52:24,383 - DEBUG - VM ctest-vn1_vm1-70599077 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-02 03:52:29,384 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:52:29,384 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-70599077, IP 192.168.1.252, Port 22 2025-10-02 03:52:29,449 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-02 03:52:29,525 - DEBUG - VM ctest-vn1_vm1-70599077 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-02 03:52:34,526 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:52:34,526 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-70599077, IP 192.168.1.252, Port 22 2025-10-02 03:52:34,593 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-02 03:52:34,672 - DEBUG - VM ctest-vn1_vm1-70599077 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-02 03:52:39,673 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:52:39,673 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-70599077, IP 192.168.1.252, Port 22 2025-10-02 03:52:39,741 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-02 03:52:39,826 - DEBUG - VM ctest-vn1_vm1-70599077 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-02 03:52:44,827 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:52:44,827 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-70599077, IP 192.168.1.252, Port 22 2025-10-02 03:52:44,993 - DEBUG - VM ctest-vn1_vm1-70599077 is ready for SSH connections 2025-10-02 03:52:44,993 - INFO - Waiting for VM ctest-vn2_vm1-84787369 to be up.. 2025-10-02 03:52:45,074 - DEBUG - VM is in ACTIVE state now 2025-10-02 03:52:45,074 - INFO - VM name : ctest-vn2_vm1-84787369 2025-10-02 03:52:45,144 - DEBUG - VM ctest-vn2_vm1-84787369 ID is 590eb253-1423-438c-b514-ea6a24b70ac3 2025-10-02 03:52:45,144 - DEBUG - VM ctest-vn2_vm1-84787369 launched on Node an-jenkins-deploy-platform-ansible-os-4210-1 2025-10-02 03:52:45,216 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/590eb253-1423-438c-b514-ea6a24b70ac3 2025-10-02 03:52:45,231 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine-interface/5e1fbfcb-8857-4bb0-8079-050606f132f3 2025-10-02 03:52:46,402 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=3.90 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=4.93 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 3.900/4.414/4.928/0.514 ms') 2025-10-02 03:52:46,402 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn2_vm1-84787369 passed 2025-10-02 03:52:46,471 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:52:46,471 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-84787369, IP 192.168.2.252, Port 22 2025-10-02 03:52:46,636 - DEBUG - VM ctest-vn2_vm1-84787369 is ready for SSH connections 2025-10-02 03:52:46,636 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:52:46,636 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.252, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-10-02 03:52:46,636 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.252 2025-10-02 03:53:08,470 - DEBUG - PING 192.168.2.252 (192.168.2.252) 56(84) bytes of data. 64 bytes from 192.168.2.252: icmp_req=1 ttl=63 time=8.29 ms 64 bytes from 192.168.2.252: icmp_req=2 ttl=63 time=1.20 ms 64 bytes from 192.168.2.252: icmp_req=3 ttl=63 time=1.38 ms --- 192.168.2.252 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 1.203/3.628/8.295/3.301 ms 2025-10-02 03:53:08,470 - INFO - Ping to IP 192.168.2.252 from VM ctest-vn1_vm1-70599077 passed 2025-10-02 03:53:11,895 - INFO - 10.0.0.33 2025-10-02 03:53:11,905 - INFO - agent:initializing 2025-10-02 03:53:11,905 - DEBUG - defaultdict(, {'10.0.0.33': {'agent': {'status': 'initializing', 'description': None}}}) 2025-10-02 03:53:11,906 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-02 03:53:16,992 - INFO - 10.0.0.33 2025-10-02 03:53:16,997 - INFO - agent:active 2025-10-02 03:53:16,997 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.33 2025-10-02 03:53:20,409 - INFO - 10.0.0.130 2025-10-02 03:53:20,414 - INFO - agent:initializing 2025-10-02 03:53:20,414 - DEBUG - defaultdict(, {'10.0.0.130': {'agent': {'status': 'initializing', 'description': None}}}) 2025-10-02 03:53:20,414 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-02 03:53:25,506 - INFO - 10.0.0.130 2025-10-02 03:53:25,512 - INFO - agent:active 2025-10-02 03:53:25,512 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.130 2025-10-02 03:53:25,981 - INFO - 10.0.0.56 2025-10-02 03:53:25,985 - INFO - control:initializing 2025-10-02 03:53:25,985 - DEBUG - defaultdict(, {'10.0.0.56': {'control': {'status': 'initializing', 'description': None}}}) 2025-10-02 03:53:25,985 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-02 03:53:31,097 - INFO - 10.0.0.56 2025-10-02 03:53:31,102 - INFO - control:active 2025-10-02 03:53:31,102 - INFO - Contrail services ['control'] are up on nodes 10.0.0.56 2025-10-02 03:53:31,547 - INFO - 10.0.0.38 2025-10-02 03:53:31,550 - INFO - control:initializing 2025-10-02 03:53:31,551 - DEBUG - defaultdict(, {'10.0.0.38': {'control': {'status': 'initializing', 'description': None}}}) 2025-10-02 03:53:31,551 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-02 03:53:36,653 - INFO - 10.0.0.38 2025-10-02 03:53:36,659 - INFO - control:active 2025-10-02 03:53:36,659 - INFO - Contrail services ['control'] are up on nodes 10.0.0.38 2025-10-02 03:53:37,144 - INFO - 10.0.0.49 2025-10-02 03:53:37,148 - INFO - control:initializing 2025-10-02 03:53:37,148 - DEBUG - defaultdict(, {'10.0.0.49': {'control': {'status': 'initializing', 'description': None}}}) 2025-10-02 03:53:37,148 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-02 03:53:42,251 - INFO - 10.0.0.49 2025-10-02 03:53:42,257 - INFO - control:active 2025-10-02 03:53:42,258 - INFO - Contrail services ['control'] are up on nodes 10.0.0.49 2025-10-02 03:53:43,022 - INFO - 10.0.0.56 2025-10-02 03:53:43,025 - INFO - api-server:initializing 2025-10-02 03:53:43,025 - DEBUG - defaultdict(, {'10.0.0.56': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-02 03:53:43,025 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-02 03:53:48,139 - INFO - 10.0.0.56 2025-10-02 03:53:48,142 - INFO - api-server:initializing 2025-10-02 03:53:48,142 - DEBUG - defaultdict(, {'10.0.0.56': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-02 03:53:48,142 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-10-02 03:53:53,259 - INFO - 10.0.0.56 2025-10-02 03:53:53,268 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2025-10-02 03:53:53,269 - DEBUG - defaultdict(, {'10.0.0.56': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2025-10-02 03:53:53,269 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-10-02 03:53:58,379 - INFO - 10.0.0.56 2025-10-02 03:53:58,383 - INFO - api-server:active 2025-10-02 03:53:58,383 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.56 2025-10-02 03:53:59,089 - INFO - 10.0.0.38 2025-10-02 03:53:59,093 - INFO - api-server:initializing 2025-10-02 03:53:59,094 - DEBUG - defaultdict(, {'10.0.0.38': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-02 03:53:59,094 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-02 03:54:04,193 - INFO - 10.0.0.38 2025-10-02 03:54:04,197 - INFO - api-server:initializing 2025-10-02 03:54:04,198 - DEBUG - defaultdict(, {'10.0.0.38': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-02 03:54:04,198 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-10-02 03:54:09,307 - INFO - 10.0.0.38 2025-10-02 03:54:09,315 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2025-10-02 03:54:09,315 - DEBUG - defaultdict(, {'10.0.0.38': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2025-10-02 03:54:09,315 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-10-02 03:54:14,413 - INFO - 10.0.0.38 2025-10-02 03:54:14,418 - INFO - api-server:active 2025-10-02 03:54:14,418 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.38 2025-10-02 03:54:15,102 - INFO - 10.0.0.49 2025-10-02 03:54:15,106 - INFO - api-server:initializing 2025-10-02 03:54:15,106 - DEBUG - defaultdict(, {'10.0.0.49': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-02 03:54:15,106 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-02 03:54:20,215 - INFO - 10.0.0.49 2025-10-02 03:54:20,220 - INFO - api-server:initializing 2025-10-02 03:54:20,220 - DEBUG - defaultdict(, {'10.0.0.49': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-02 03:54:20,220 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-10-02 03:54:25,317 - INFO - 10.0.0.49 2025-10-02 03:54:25,322 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2025-10-02 03:54:25,322 - DEBUG - defaultdict(, {'10.0.0.49': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2025-10-02 03:54:25,322 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-10-02 03:54:30,421 - INFO - 10.0.0.49 2025-10-02 03:54:30,426 - INFO - api-server:active 2025-10-02 03:54:30,426 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.49 2025-10-02 03:54:30,574 - INFO - 10.0.0.56 2025-10-02 03:54:30,574 - INFO - config 2025-10-02 03:54:30,580 - INFO - config-nodemgr:active 2025-10-02 03:54:30,583 - INFO - api-server:active 2025-10-02 03:54:30,585 - INFO - schema:backup 2025-10-02 03:54:30,589 - INFO - svc-monitor:active 2025-10-02 03:54:30,592 - INFO - device-manager:backup 2025-10-02 03:54:30,592 - INFO - config-database 2025-10-02 03:54:30,593 - INFO - config-cassandra:active 2025-10-02 03:54:30,593 - INFO - config-zookeeper:active 2025-10-02 03:54:30,593 - INFO - config-rabbitmq:active 2025-10-02 03:54:30,593 - INFO - control 2025-10-02 03:54:30,596 - INFO - control-nodemgr:active 2025-10-02 03:54:30,602 - INFO - control:active 2025-10-02 03:54:30,602 - INFO - named:active 2025-10-02 03:54:30,608 - INFO - dns:active 2025-10-02 03:54:30,608 - INFO - analytics 2025-10-02 03:54:30,613 - INFO - analytics-nodemgr:active 2025-10-02 03:54:30,616 - INFO - analytics-api:active 2025-10-02 03:54:30,622 - INFO - collector:active 2025-10-02 03:54:30,622 - INFO - analytics-database 2025-10-02 03:54:30,623 - INFO - analytics-cassandra:active 2025-10-02 03:54:30,627 - INFO - analyticsdb-nodemgr:active 2025-10-02 03:54:30,632 - INFO - query-engine:active 2025-10-02 03:54:30,632 - INFO - webui 2025-10-02 03:54:30,632 - INFO - webui:active 2025-10-02 03:54:30,632 - INFO - webui-middleware:active 2025-10-02 03:54:30,632 - INFO - redis:active 2025-10-02 03:54:30,632 - INFO - analytics_snmp 2025-10-02 03:54:30,636 - INFO - snmp-collector:active 2025-10-02 03:54:30,636 - INFO - snmp-topology:active 2025-10-02 03:54:30,636 - INFO - analytics_alarm 2025-10-02 03:54:30,636 - INFO - alarmgen:active 2025-10-02 03:54:30,733 - INFO - 10.0.0.38 2025-10-02 03:54:30,733 - INFO - config 2025-10-02 03:54:30,738 - INFO - config-nodemgr:active 2025-10-02 03:54:30,742 - INFO - api-server:active 2025-10-02 03:54:30,746 - INFO - schema:active 2025-10-02 03:54:30,749 - INFO - svc-monitor:backup 2025-10-02 03:54:30,752 - INFO - device-manager:backup 2025-10-02 03:54:30,752 - INFO - config-database 2025-10-02 03:54:30,752 - INFO - config-cassandra:active 2025-10-02 03:54:30,752 - INFO - config-zookeeper:active 2025-10-02 03:54:30,752 - INFO - config-rabbitmq:active 2025-10-02 03:54:30,752 - INFO - control 2025-10-02 03:54:30,758 - INFO - control-nodemgr:active 2025-10-02 03:54:30,762 - INFO - control:active 2025-10-02 03:54:30,762 - INFO - named:active 2025-10-02 03:54:30,769 - INFO - dns:active 2025-10-02 03:54:30,770 - INFO - analytics 2025-10-02 03:54:30,774 - INFO - analytics-nodemgr:active 2025-10-02 03:54:30,778 - INFO - analytics-api:active 2025-10-02 03:54:30,783 - INFO - collector:active 2025-10-02 03:54:30,783 - INFO - analytics-database 2025-10-02 03:54:30,783 - INFO - analytics-cassandra:active 2025-10-02 03:54:30,788 - INFO - analyticsdb-nodemgr:active 2025-10-02 03:54:30,795 - INFO - query-engine:active 2025-10-02 03:54:30,795 - INFO - webui 2025-10-02 03:54:30,795 - INFO - webui:active 2025-10-02 03:54:30,795 - INFO - webui-middleware:active 2025-10-02 03:54:30,796 - INFO - redis:active 2025-10-02 03:54:30,796 - INFO - analytics_snmp 2025-10-02 03:54:30,799 - INFO - snmp-collector:active 2025-10-02 03:54:30,799 - INFO - snmp-topology:active 2025-10-02 03:54:30,799 - INFO - analytics_alarm 2025-10-02 03:54:30,799 - INFO - alarmgen:active 2025-10-02 03:54:30,886 - INFO - 10.0.0.49 2025-10-02 03:54:30,886 - INFO - config 2025-10-02 03:54:30,891 - INFO - config-nodemgr:active 2025-10-02 03:54:30,896 - INFO - api-server:active 2025-10-02 03:54:30,898 - INFO - schema:backup 2025-10-02 03:54:30,901 - INFO - svc-monitor:backup 2025-10-02 03:54:30,905 - INFO - device-manager:active 2025-10-02 03:54:30,905 - INFO - config-database 2025-10-02 03:54:30,905 - INFO - config-cassandra:active 2025-10-02 03:54:30,905 - INFO - config-zookeeper:active 2025-10-02 03:54:30,905 - INFO - config-rabbitmq:active 2025-10-02 03:54:30,905 - INFO - control 2025-10-02 03:54:30,909 - INFO - control-nodemgr:active 2025-10-02 03:54:30,914 - INFO - control:active 2025-10-02 03:54:30,915 - INFO - named:active 2025-10-02 03:54:30,919 - INFO - dns:active 2025-10-02 03:54:30,919 - INFO - analytics 2025-10-02 03:54:30,924 - INFO - analytics-nodemgr:active 2025-10-02 03:54:30,928 - INFO - analytics-api:active 2025-10-02 03:54:30,933 - INFO - collector:active 2025-10-02 03:54:30,933 - INFO - analytics-database 2025-10-02 03:54:30,933 - INFO - analytics-cassandra:active 2025-10-02 03:54:30,938 - INFO - analyticsdb-nodemgr:active 2025-10-02 03:54:30,942 - INFO - query-engine:active 2025-10-02 03:54:30,942 - INFO - webui 2025-10-02 03:54:30,943 - INFO - webui:active 2025-10-02 03:54:30,943 - INFO - webui-middleware:active 2025-10-02 03:54:30,943 - INFO - redis:active 2025-10-02 03:54:30,943 - INFO - analytics_snmp 2025-10-02 03:54:30,947 - INFO - snmp-collector:active 2025-10-02 03:54:30,948 - INFO - snmp-topology:active 2025-10-02 03:54:30,948 - INFO - analytics_alarm 2025-10-02 03:54:30,948 - INFO - alarmgen:active 2025-10-02 03:54:31,035 - INFO - 10.0.0.33 2025-10-02 03:54:31,035 - INFO - vrouter 2025-10-02 03:54:31,040 - INFO - vrouter-nodemgr:active 2025-10-02 03:54:31,046 - INFO - agent:active 2025-10-02 03:54:31,134 - INFO - 10.0.0.130 2025-10-02 03:54:31,134 - INFO - vrouter 2025-10-02 03:54:31,139 - INFO - vrouter-nodemgr:active 2025-10-02 03:54:31,144 - INFO - agent:active 2025-10-02 03:54:31,144 - INFO - All the contrail services are up on all nodes 2025-10-02 03:54:31,144 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4210-1?flat 2025-10-02 03:54:31,533 - INFO - an-jenkins-deploy-platform-ansible-os-4210-1:contrail-vrouter-agent module connection to ['10.20.0.25:5269'] servers UP 2025-10-02 03:54:31,533 - INFO - an-jenkins-deploy-platform-ansible-os-4210-1 connected to xmpp 10.20.0.25 2025-10-02 03:54:31,533 - INFO - an-jenkins-deploy-platform-ansible-os-4210-1:contrail-vrouter-agent module connection to ['10.20.0.254:5269'] servers UP 2025-10-02 03:54:31,533 - INFO - an-jenkins-deploy-platform-ansible-os-4210-1 connected to xmpp 10.20.0.254 2025-10-02 03:54:31,533 - WARNING - an-jenkins-deploy-platform-ansible-os-4210-1:contrail-vrouter-agent module connection to ['10.20.0.14:5269'] servers NOT UP 2025-10-02 03:54:31,533 - INFO - an-jenkins-deploy-platform-ansible-os-4210-1:contrail-vrouter-agent module connection to ['10.20.0.25:53'] servers UP 2025-10-02 03:54:31,533 - INFO - an-jenkins-deploy-platform-ansible-os-4210-1 connected to dns 10.20.0.25 2025-10-02 03:54:31,533 - INFO - an-jenkins-deploy-platform-ansible-os-4210-1:contrail-vrouter-agent module connection to ['10.20.0.254:53'] servers UP 2025-10-02 03:54:31,534 - INFO - an-jenkins-deploy-platform-ansible-os-4210-1 connected to dns 10.20.0.254 2025-10-02 03:54:31,534 - WARNING - an-jenkins-deploy-platform-ansible-os-4210-1:contrail-vrouter-agent module connection to ['10.20.0.14:53'] servers NOT UP 2025-10-02 03:54:31,534 - WARNING - an-jenkins-deploy-platform-ansible-os-4210-1:contrail-vrouter-agent module connection to ['10.0.0.56:8086'] servers NOT UP 2025-10-02 03:54:31,534 - WARNING - an-jenkins-deploy-platform-ansible-os-4210-1:contrail-vrouter-agent module connection to ['10.0.0.38:8086'] servers NOT UP 2025-10-02 03:54:31,534 - INFO - an-jenkins-deploy-platform-ansible-os-4210-1:contrail-vrouter-agent module connection to ['10.0.0.49:8086'] servers UP 2025-10-02 03:54:31,534 - INFO - an-jenkins-deploy-platform-ansible-os-4210-1 connected to collector 10.0.0.49 2025-10-02 03:54:31,534 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4210-1?flat 2025-10-02 03:54:31,900 - INFO - cn-jenkins-deploy-platform-ansible-os-4210-1:contrail-control module connection to 10.0.0.56:9041 servers UP 2025-10-02 03:54:31,900 - INFO - cn-jenkins-deploy-platform-ansible-os-4210-1:contrail-control module connection to 10.0.0.56:5673 servers UP 2025-10-02 03:54:31,900 - INFO - cn-jenkins-deploy-platform-ansible-os-4210-1:contrail-control module connection to ['10.0.0.56:8086'] servers UP 2025-10-02 03:54:31,900 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4210-2?flat 2025-10-02 03:54:31,954 - INFO - cn-jenkins-deploy-platform-ansible-os-4210-2:contrail-control module connection to 10.0.0.56:9041 servers UP 2025-10-02 03:54:31,954 - INFO - cn-jenkins-deploy-platform-ansible-os-4210-2:contrail-control module connection to 10.0.0.56:5673 servers UP 2025-10-02 03:54:31,954 - INFO - cn-jenkins-deploy-platform-ansible-os-4210-2:contrail-control module connection to ['10.0.0.56:8086'] servers UP 2025-10-02 03:54:31,954 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4210-3?flat 2025-10-02 03:54:31,974 - INFO - cn-jenkins-deploy-platform-ansible-os-4210-3:contrail-control module connection to 10.0.0.56:9041 servers UP 2025-10-02 03:54:31,974 - INFO - cn-jenkins-deploy-platform-ansible-os-4210-3:contrail-control module connection to 10.0.0.56:5673 servers UP 2025-10-02 03:54:31,974 - INFO - cn-jenkins-deploy-platform-ansible-os-4210-3:contrail-control module connection to ['10.0.0.56:8086'] servers UP 2025-10-02 03:54:31,975 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4210-1.?flat 2025-10-02 03:54:31,993 - DEBUG - contrail-api is running 2025-10-02 03:54:31,993 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4210-2.?flat 2025-10-02 03:54:32,012 - DEBUG - contrail-api is running 2025-10-02 03:54:32,012 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4210-3.?flat 2025-10-02 03:54:32,033 - DEBUG - contrail-api is running 2025-10-02 03:54:32,318 - INFO - Created VN ctest-vn3-32047234, UUID :9465c050-639b-43f4-9bcf-eea575d47ab4 2025-10-02 03:54:32,712 - DEBUG - Requesting: http://10.0.0.56:8082/domains 2025-10-02 03:54:32,737 - DEBUG - Requesting: http://10.0.0.56:8082/domain/b0543e69-2184-4be1-8e23-ac5840233bc6 2025-10-02 03:54:32,753 - DEBUG - Requesting: http://10.0.0.56:8082/project/7612bb4f-d0d8-4c4e-9c49-460de03b6971 2025-10-02 03:54:32,872 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/1c862540-f7c7-471b-b33a-77cef607922f 2025-10-02 03:54:32,886 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/1c862540-f7c7-471b-b33a-77cef607922f 2025-10-02 03:54:32,896 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/bff0383e-27bf-4de3-a2c9-15fd9d24118f 2025-10-02 03:54:32,903 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/bff0383e-27bf-4de3-a2c9-15fd9d24118f 2025-10-02 03:54:32,911 - DEBUG - Requesting: http://10.0.0.56:8082/route-target/8b8faaea-2f0f-4ca2-ab2f-34142c532392 2025-10-02 03:54:32,936 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-02 03:54:32,936 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/1c862540-f7c7-471b-b33a-77cef607922f 2025-10-02 03:54:32,949 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/bff0383e-27bf-4de3-a2c9-15fd9d24118f 2025-10-02 03:54:32,964 - INFO - Verified VN network id 13 for VN 1c862540-f7c7-471b-b33a-77cef607922f 2025-10-02 03:54:32,965 - INFO - Verifications in API Server for VN ctest-vn1-80182616 passed 2025-10-02 03:54:32,965 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/1c862540-f7c7-471b-b33a-77cef607922f 2025-10-02 03:54:32,975 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/bff0383e-27bf-4de3-a2c9-15fd9d24118f 2025-10-02 03:54:32,981 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/bff0383e-27bf-4de3-a2c9-15fd9d24118f 2025-10-02 03:54:32,987 - DEBUG - Requesting: http://10.0.0.56:8082/route-target/8b8faaea-2f0f-4ca2-ab2f-34142c532392 2025-10-02 03:54:32,996 - DEBUG - Control-node 10.0.0.56 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-35143319', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '2055371240906376987', 'uuid-lslong': '12914766612212978223'}, 'enable': 'true', 'created': '2025-10-02T03:51:37', 'last-modified': '2025-10-02T03:51:38', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:06.704580'} 2025-10-02 03:54:33,000 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-02 03:54:33,008 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-35143319', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '2055371240906376987', 'uuid-lslong': '12914766612212978223'}, 'enable': 'true', 'created': '2025-10-02T03:51:37', 'last-modified': '2025-10-02T03:51:38', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:01.069834'} 2025-10-02 03:54:33,015 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-02 03:54:33,021 - DEBUG - Control-node 10.0.0.49 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-35143319', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '2055371240906376987', 'uuid-lslong': '12914766612212978223'}, 'enable': 'true', 'created': '2025-10-02T03:51:37', 'last-modified': '2025-10-02T03:51:38', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:55.386048'} 2025-10-02 03:54:33,027 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-02 03:54:33,027 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-80182616 passed 2025-10-02 03:54:33,028 - DEBUG - ====Verifying policy data for ctest-vn1-80182616 in API_Server ====== 2025-10-02 03:54:33,028 - DEBUG - Requesting: http://10.0.0.56:8082/domains 2025-10-02 03:54:33,034 - DEBUG - Requesting: http://10.0.0.56:8082/domain/b0543e69-2184-4be1-8e23-ac5840233bc6 2025-10-02 03:54:33,045 - DEBUG - Requesting: http://10.0.0.56:8082/project/7612bb4f-d0d8-4c4e-9c49-460de03b6971 2025-10-02 03:54:33,132 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/1c862540-f7c7-471b-b33a-77cef607922f 2025-10-02 03:54:33,146 - DEBUG - ==>Verifying data for policy with id: cbe14f88-bb2d-49d6-bdd1-b4a5a417bfb0, fqn: ['default-domain', 'ctest-TestBasicVMVN0-35143319', 'policy1'] 2025-10-02 03:54:33,146 - INFO - VN ctest-vn1-80182616 Policy verification: verify_vn_policy_in_api_server, status: True 2025-10-02 03:54:33,146 - DEBUG - Verifying the vn in opserver 2025-10-02 03:54:33,146 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 virtual network link through opserver 10.0.0.56 2025-10-02 03:54:33,146 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-networks 2025-10-02 03:54:33,162 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616', 'href': 'http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616?flat'} 2025-10-02 03:54:33,162 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 virtual network link through opserver 10.0.0.38 2025-10-02 03:54:33,162 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-10-02 03:54:33,219 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616?flat'} 2025-10-02 03:54:33,219 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 virtual network link through opserver 10.0.0.49 2025-10-02 03:54:33,219 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-networks 2025-10-02 03:54:33,276 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616', 'href': 'http://10.0.0.49:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616?flat'} 2025-10-02 03:54:33,276 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 is found in opserver 2025-10-02 03:54:33,286 - DEBUG - VRF ids for VN ctest-vn1-80182616: {'10.0.0.130': '2'} 2025-10-02 03:54:34,463 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4210-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4210-2) 2025-10-02 03:54:34,517 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-10-02 03:54:39,602 - DEBUG - VM is in ACTIVE state now 2025-10-02 03:54:39,602 - INFO - VM name : ctest-vn1_vm2-75628550 2025-10-02 03:54:39,683 - DEBUG - VM ctest-vn1_vm2-75628550 ID is 5050a030-f368-49d2-8eab-52b7bdaf41e5 2025-10-02 03:54:39,771 - DEBUG - VM is in ACTIVE state now 2025-10-02 03:54:39,771 - DEBUG - VM ctest-vn1_vm2-75628550 launched on Node an-jenkins-deploy-platform-ansible-os-4210-2 2025-10-02 03:54:39,859 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/5050a030-f368-49d2-8eab-52b7bdaf41e5 2025-10-02 03:54:39,894 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/5050a030-f368-49d2-8eab-52b7bdaf41e5 2025-10-02 03:54:39,902 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine-interface/7543745a-c217-4a34-8449-4ddf8cb62263 2025-10-02 03:54:39,912 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/5050a030-f368-49d2-8eab-52b7bdaf41e5 2025-10-02 03:54:39,920 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine-interface/7543745a-c217-4a34-8449-4ddf8cb62263 2025-10-02 03:54:39,930 - DEBUG - Requesting: http://10.0.0.49:8082/instance-ip/d5053f16-b9c2-4b40-a0c9-15fb45f3a801 2025-10-02 03:54:39,939 - DEBUG - Verifying in api server 10.0.0.49 2025-10-02 03:54:39,939 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/5050a030-f368-49d2-8eab-52b7bdaf41e5 2025-10-02 03:54:39,948 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine-interface/7543745a-c217-4a34-8449-4ddf8cb62263 2025-10-02 03:54:39,958 - DEBUG - Requesting: http://10.0.0.49:8082/instance-ip/d5053f16-b9c2-4b40-a0c9-15fb45f3a801 2025-10-02 03:54:39,969 - INFO - VM ctest-vn1_vm2-75628550 verfication in all API Servers passed 2025-10-02 03:54:39,993 - DEBUG - VM ctest-vn1_vm2-75628550 Tap interface: {'index': '4', 'name': 'tap7543745a-c2', 'uuid': '7543745a-c217-4a34-8449-4ddf8cb62263', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616:ctest-vn1-80182616', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616', 'vm_uuid': '5050a030-f368-49d2-8eab-52b7bdaf41e5', 'vm_name': 'ctest-vn1_vm2-75628550', 'ip_addr': '192.168.1.251', 'mac_addr': '02:75:43:74:5a:c2', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '169', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-35143319:7543745a-c217-4a34-8449-4ddf8cb62263', 'sg_uuid_list': ['44109285-648a-42a8-91b0-26de1a86fc15'], 'static_route_list': None, 'vm_project_uuid': '7612bb4f-d0d8-4c4e-9c49-460de03b6971', 'admin_state': 'Enabled', 'flow_key_idx': '32', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-02 03:54:39,993 - DEBUG - Agent 10.0.0.130 vrf name: default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616:ctest-vn1-80182616 2025-10-02 03:54:40,000 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616:ctest-vn1-80182616', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616', 'table_label': '-1', 'vxlan_id': '13', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.249:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-10-02 03:54:40,085 - DEBUG - VM ctest-vn1_vm2-75628550 : Tap interface tap7543745a-c2 is set to Active 2025-10-02 03:54:40,085 - DEBUG - Tap interface tap7543745a-c2 detail : {'index': '4', 'name': 'tap7543745a-c2', 'uuid': '7543745a-c217-4a34-8449-4ddf8cb62263', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616:ctest-vn1-80182616', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616', 'vm_uuid': '5050a030-f368-49d2-8eab-52b7bdaf41e5', 'vm_name': 'ctest-vn1_vm2-75628550', 'ip_addr': '192.168.1.251', 'mac_addr': '02:75:43:74:5a:c2', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '169', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-35143319:7543745a-c217-4a34-8449-4ddf8cb62263', 'sg_uuid_list': ['44109285-648a-42a8-91b0-26de1a86fc15'], 'static_route_list': None, 'vm_project_uuid': '7612bb4f-d0d8-4c4e-9c49-460de03b6971', 'admin_state': 'Enabled', 'flow_key_idx': '32', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-02 03:54:40,203 - DEBUG - Starting Layer 2 verification in Agent 2025-10-02 03:54:40,220 - DEBUG - Layer 2 path is seen for VM MAC 02:75:43:74:5a:c2 in agent 10.0.0.130 2025-10-02 03:54:40,222 - DEBUG - Active layer 2 route in agent is present for VMI tap7543745a-c2 2025-10-02 03:54:40,222 - DEBUG - L2 label(34) matches bw route table and interface table 2025-10-02 03:54:40,238 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 is found in Agent of node 10.0.0.130 2025-10-02 03:54:40,329 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 is consistent in agent 10.0.0.130 2025-10-02 03:54:40,329 - DEBUG - Route for VM IP ['192.168.1.251'] is consistent in agent 10.0.0.130 2025-10-02 03:54:40,329 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 verification for VM ctest-vn1_vm2-75628550 in Agent 10.0.0.130 passed 2025-10-02 03:54:40,443 - DEBUG - Starting all layer 2 verification in agent 10.0.0.130 2025-10-02 03:54:40,455 - DEBUG - Route for VM MAC 02:75:43:74:5a:c2 is consistent in agent 10.0.0.130 2025-10-02 03:54:43,657 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2025-10-02 03:54:43,657 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-75628550 failed! 2025-10-02 03:54:43,671 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 is 192.168.1.254 and allocation pool is NOT set 2025-10-02 03:54:47,753 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1020ms') 2025-10-02 03:54:47,754 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-75628550 failed! 2025-10-02 03:54:47,766 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 is 192.168.1.254 and allocation pool is NOT set 2025-10-02 03:54:51,847 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2025-10-02 03:54:51,847 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-75628550 failed! 2025-10-02 03:54:51,861 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 is 192.168.1.254 and allocation pool is NOT set 2025-10-02 03:54:55,941 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-10-02 03:54:55,941 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-75628550 failed! 2025-10-02 03:54:55,959 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 is 192.168.1.254 and allocation pool is NOT set 2025-10-02 03:55:00,041 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1008ms') 2025-10-02 03:55:00,041 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-75628550 failed! 2025-10-02 03:55:00,057 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 is 192.168.1.254 and allocation pool is NOT set 2025-10-02 03:55:04,141 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1016ms') 2025-10-02 03:55:04,141 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-75628550 failed! 2025-10-02 03:55:04,160 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 is 192.168.1.254 and allocation pool is NOT set 2025-10-02 03:55:08,266 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1030ms') 2025-10-02 03:55:08,267 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-75628550 failed! 2025-10-02 03:55:08,281 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 is 192.168.1.254 and allocation pool is NOT set 2025-10-02 03:55:12,367 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-10-02 03:55:12,367 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-75628550 failed! 2025-10-02 03:55:12,382 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 is 192.168.1.254 and allocation pool is NOT set 2025-10-02 03:55:16,452 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=17.1 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 1 received, 50% packet loss, time 1008ms\r\nrtt min/avg/max/mdev = 17.101/17.101/17.101/0.000 ms') 2025-10-02 03:55:16,453 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-75628550 passed 2025-10-02 03:55:16,453 - INFO - VM ctest-vn1_vm2-75628550 verifications in Compute nodes passed 2025-10-02 03:55:16,667 - DEBUG - Validated VM route 192.168.1.251 in vrouter of 10.0.0.130 2025-10-02 03:55:16,671 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-10-02 03:55:16,671 - INFO - Validated routes of VM ctest-vn1_vm2-75628550 in all vrouters 2025-10-02 03:55:17,101 - DEBUG - Starting all layer2 verification in 10.0.0.38 Control Node 2025-10-02 03:55:17,233 - DEBUG - Layer2 route found for VM MAC 02:75:43:74:5a:c2 in Control-node 10.0.0.38 2025-10-02 03:55:17,233 - DEBUG - L2 Label for VM ctest-vn1_vm2-75628550 same between Control-node 10.0.0.38 and Agent, Expected: 34, Seen: 34 2025-10-02 03:55:17,234 - DEBUG - Starting all layer2 verification in 10.0.0.49 Control Node 2025-10-02 03:55:17,358 - DEBUG - Layer2 route found for VM MAC 02:75:43:74:5a:c2 in Control-node 10.0.0.49 2025-10-02 03:55:17,358 - DEBUG - L2 Label for VM ctest-vn1_vm2-75628550 same between Control-node 10.0.0.49 and Agent, Expected: 34, Seen: 34 2025-10-02 03:55:17,358 - DEBUG - Starting all layer2 verification in 10.0.0.56 Control Node 2025-10-02 03:55:17,489 - DEBUG - Layer2 route found for VM MAC 02:75:43:74:5a:c2 in Control-node 10.0.0.56 2025-10-02 03:55:17,489 - DEBUG - L2 Label for VM ctest-vn1_vm2-75628550 same between Control-node 10.0.0.56 and Agent, Expected: 34, Seen: 34 2025-10-02 03:55:17,489 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-75628550 passed 2025-10-02 03:55:17,489 - DEBUG - Verifying the vm in opserver 2025-10-02 03:55:17,489 - DEBUG - Verifying in collector 10.0.0.49 ... 2025-10-02 03:55:17,489 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-machine/5050a030-f368-49d2-8eab-52b7bdaf41e5?flat 2025-10-02 03:55:17,549 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-35143319:7543745a-c217-4a34-8449-4ddf8cb62263?flat 2025-10-02 03:55:17,565 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-35143319:7543745a-c217-4a34-8449-4ddf8cb62263?flat 2025-10-02 03:55:17,587 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-machine/5050a030-f368-49d2-8eab-52b7bdaf41e5?flat 2025-10-02 03:55:17,601 - DEBUG - Verifying vm in vn uve 2025-10-02 03:55:17,601 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-35143319:7543745a-c217-4a34-8449-4ddf8cb62263?flat 2025-10-02 03:55:17,621 - DEBUG - VM uve shows interface as {'__T': 1759377291070175, 'ip6_active': False, 'rx_vlan': 65535, 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616', 'ip4_active': True, 'vm_name': 'ctest-vn1_vm2-75628550', 'port_mirror_enabled': False, 'is_health_check_active': True, 'active': True, 'label': 30, 'vhostuser_mode': 0, 'ip_address': '192.168.1.251', 'tx_vlan': 65535, 'admin_state': True, 'l2_active': True, 'mac_address': '02:75:43:74:5a:c2', 'vm_uuid': '5050a030-f368-49d2-8eab-52b7bdaf41e5', 'vn_uuid': '1c862540-f7c7-471b-b33a-77cef607922f', 'ip6_address': '::', 'uuid': '7543745a-c217-4a34-8449-4ddf8cb62263', 'fixed_ip4_list': ['192.168.1.251'], 'gateway': '192.168.1.254'} 2025-10-02 03:55:17,621 - DEBUG - VM uve shows ip address as ['192.168.1.251', '::'] 2025-10-02 03:55:17,621 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 2025-10-02 03:55:17,621 - INFO - Verifying through opserver in 10.0.0.56 2025-10-02 03:55:17,621 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616?flat 2025-10-02 03:55:17,641 - DEBUG - expected vm list 5050a030-f368-49d2-8eab-52b7bdaf41e5 2025-10-02 03:55:17,641 - DEBUG - Extracted vm list ['5050a030-f368-49d2-8eab-52b7bdaf41e5'] 2025-10-02 03:55:17,641 - INFO - VM 5050a030-f368-49d2-8eab-52b7bdaf41e5 is present in default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 2025-10-02 03:55:17,641 - DEBUG - Verifying vm in vrouter uve 2025-10-02 03:55:17,641 - DEBUG - Getting info from collector 10.0.0.49.. 2025-10-02 03:55:17,641 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-machine/5050a030-f368-49d2-8eab-52b7bdaf41e5?flat 2025-10-02 03:55:17,697 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4210-2'] 2025-10-02 03:55:17,697 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4210-2:Compute:contrail-vrouter-agent:0?flat 2025-10-02 03:55:17,739 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4210-2?flat 2025-10-02 03:55:17,771 - DEBUG - VM 5050a030-f368-49d2-8eab-52b7bdaf41e5 is present in vrouter an-jenkins-deploy-platform-ansible-os-4210-2 uve 2025-10-02 03:55:17,771 - DEBUG - Validated that VM ctest-vn1_vm2-75628550 is in Vrouter an-jenkins-deploy-platform-ansible-os-4210-2 UVE 2025-10-02 03:55:17,771 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-10-02 03:55:17,779 - DEBUG - Expected tap interface of VM uuid 5050a030-f368-49d2-8eab-52b7bdaf41e5 is default-domain:ctest-TestBasicVMVN0-35143319:7543745a-c217-4a34-8449-4ddf8cb62263 2025-10-02 03:55:17,779 - DEBUG - Expected VN of VM uuid 5050a030-f368-49d2-8eab-52b7bdaf41e5 is default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 2025-10-02 03:55:17,780 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4210-2:Compute:contrail-vrouter-agent:0?flat 2025-10-02 03:55:17,820 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4210-2?flat 2025-10-02 03:55:17,857 - DEBUG - VM 5050a030-f368-49d2-8eab-52b7bdaf41e5 is present in vrouter an-jenkins-deploy-platform-ansible-os-4210-2 uve 2025-10-02 03:55:17,857 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-35143319:7543745a-c217-4a34-8449-4ddf8cb62263 of vm 5050a030-f368-49d2-8eab-52b7bdaf41e5 is present in vrouter an-jenkins-deploy-platform-ansible-os-4210-2 uve 2025-10-02 03:55:17,857 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 of vm 5050a030-f368-49d2-8eab-52b7bdaf41e5 is present in vrouter an-jenkins-deploy-platform-ansible-os-4210-2 uve 2025-10-02 03:55:17,857 - DEBUG - Verifying the 5050a030-f368-49d2-8eab-52b7bdaf41e5 virtual network link through opserver 10.0.0.56 2025-10-02 03:55:17,858 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machines 2025-10-02 03:55:17,871 - DEBUG - VM link and name as {'name': '5050a030-f368-49d2-8eab-52b7bdaf41e5', 'href': 'http://10.0.0.56:8081/analytics/uves/virtual-machine/5050a030-f368-49d2-8eab-52b7bdaf41e5?flat'} 2025-10-02 03:55:17,871 - DEBUG - Verifying the 5050a030-f368-49d2-8eab-52b7bdaf41e5 virtual network link through opserver 10.0.0.38 2025-10-02 03:55:17,871 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-10-02 03:55:17,944 - DEBUG - VM link and name as {'name': '5050a030-f368-49d2-8eab-52b7bdaf41e5', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-machine/5050a030-f368-49d2-8eab-52b7bdaf41e5?flat'} 2025-10-02 03:55:17,944 - DEBUG - Verifying the 5050a030-f368-49d2-8eab-52b7bdaf41e5 virtual network link through opserver 10.0.0.49 2025-10-02 03:55:17,944 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-machines 2025-10-02 03:55:17,958 - DEBUG - VM link and name as {'name': '5050a030-f368-49d2-8eab-52b7bdaf41e5', 'href': 'http://10.0.0.49:8081/analytics/uves/virtual-machine/5050a030-f368-49d2-8eab-52b7bdaf41e5?flat'} 2025-10-02 03:55:17,958 - INFO - VM ctest-vn1_vm2-75628550 validations in Opserver passed 2025-10-02 03:55:19,188 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4210-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4210-1) 2025-10-02 03:55:19,239 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-10-02 03:55:24,327 - DEBUG - VM is in ACTIVE state now 2025-10-02 03:55:24,328 - INFO - VM name : ctest-vn2_vm2-07593574 2025-10-02 03:55:24,407 - DEBUG - VM ctest-vn2_vm2-07593574 ID is 416cc01f-f406-4067-bed8-6ccb1b3afc1d 2025-10-02 03:55:24,496 - DEBUG - VM is in ACTIVE state now 2025-10-02 03:55:24,497 - DEBUG - VM ctest-vn2_vm2-07593574 launched on Node an-jenkins-deploy-platform-ansible-os-4210-1 2025-10-02 03:55:24,599 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/416cc01f-f406-4067-bed8-6ccb1b3afc1d 2025-10-02 03:55:24,608 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/416cc01f-f406-4067-bed8-6ccb1b3afc1d 2025-10-02 03:55:24,616 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine-interface/03c176dd-3b7b-4678-82be-9c2b633b9d53 2025-10-02 03:55:24,627 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/416cc01f-f406-4067-bed8-6ccb1b3afc1d 2025-10-02 03:55:24,636 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine-interface/03c176dd-3b7b-4678-82be-9c2b633b9d53 2025-10-02 03:55:24,647 - DEBUG - Requesting: http://10.0.0.49:8082/instance-ip/56257738-658e-4572-b81d-8c5ff1b14110 2025-10-02 03:55:24,656 - DEBUG - Verifying in api server 10.0.0.49 2025-10-02 03:55:24,656 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/416cc01f-f406-4067-bed8-6ccb1b3afc1d 2025-10-02 03:55:24,664 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine-interface/03c176dd-3b7b-4678-82be-9c2b633b9d53 2025-10-02 03:55:24,677 - DEBUG - Requesting: http://10.0.0.49:8082/instance-ip/56257738-658e-4572-b81d-8c5ff1b14110 2025-10-02 03:55:24,687 - INFO - VM ctest-vn2_vm2-07593574 verfication in all API Servers passed 2025-10-02 03:55:24,702 - DEBUG - VM ctest-vn2_vm2-07593574 Tap interface: {'index': '4', 'name': 'tap03c176dd-3b', 'uuid': '03c176dd-3b7b-4678-82be-9c2b633b9d53', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640:ctest-vn2-20688640', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640', 'vm_uuid': '416cc01f-f406-4067-bed8-6ccb1b3afc1d', 'vm_name': 'ctest-vn2_vm2-07593574', 'ip_addr': '192.168.2.251', 'mac_addr': '02:03:c1:76:dd:3b', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '127', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-35143319:03c176dd-3b7b-4678-82be-9c2b633b9d53', 'sg_uuid_list': ['44109285-648a-42a8-91b0-26de1a86fc15'], 'static_route_list': None, 'vm_project_uuid': '7612bb4f-d0d8-4c4e-9c49-460de03b6971', 'admin_state': 'Enabled', 'flow_key_idx': '30', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-02 03:55:24,702 - DEBUG - Agent 10.0.0.33 vrf name: default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640:ctest-vn2-20688640 2025-10-02 03:55:24,707 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640:ctest-vn2-20688640', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640', 'table_label': '-1', 'vxlan_id': '14', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.252:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-10-02 03:55:24,793 - DEBUG - VM ctest-vn2_vm2-07593574 : Tap interface tap03c176dd-3b is set to Active 2025-10-02 03:55:24,793 - DEBUG - Tap interface tap03c176dd-3b detail : {'index': '4', 'name': 'tap03c176dd-3b', 'uuid': '03c176dd-3b7b-4678-82be-9c2b633b9d53', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640:ctest-vn2-20688640', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640', 'vm_uuid': '416cc01f-f406-4067-bed8-6ccb1b3afc1d', 'vm_name': 'ctest-vn2_vm2-07593574', 'ip_addr': '192.168.2.251', 'mac_addr': '02:03:c1:76:dd:3b', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '127', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-35143319:03c176dd-3b7b-4678-82be-9c2b633b9d53', 'sg_uuid_list': ['44109285-648a-42a8-91b0-26de1a86fc15'], 'static_route_list': None, 'vm_project_uuid': '7612bb4f-d0d8-4c4e-9c49-460de03b6971', 'admin_state': 'Enabled', 'flow_key_idx': '30', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-02 03:55:24,911 - DEBUG - Starting Layer 2 verification in Agent 2025-10-02 03:55:24,921 - DEBUG - Layer 2 path is seen for VM MAC 02:03:c1:76:dd:3b in agent 10.0.0.33 2025-10-02 03:55:24,922 - DEBUG - Active layer 2 route in agent is present for VMI tap03c176dd-3b 2025-10-02 03:55:24,922 - DEBUG - L2 label(34) matches bw route table and interface table 2025-10-02 03:55:24,928 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640 is found in Agent of node 10.0.0.33 2025-10-02 03:55:25,064 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640 is consistent in agent 10.0.0.33 2025-10-02 03:55:25,064 - DEBUG - Route for VM IP ['192.168.2.251'] is consistent in agent 10.0.0.33 2025-10-02 03:55:25,064 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640 verification for VM ctest-vn2_vm2-07593574 in Agent 10.0.0.33 passed 2025-10-02 03:55:25,183 - DEBUG - Starting all layer 2 verification in agent 10.0.0.33 2025-10-02 03:55:25,199 - DEBUG - Route for VM MAC 02:03:c1:76:dd:3b is consistent in agent 10.0.0.33 2025-10-02 03:55:28,411 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1032ms') 2025-10-02 03:55:28,411 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-07593574 failed! 2025-10-02 03:55:28,429 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640 is 192.168.2.254 and allocation pool is NOT set 2025-10-02 03:55:32,521 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1025ms') 2025-10-02 03:55:32,521 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-07593574 failed! 2025-10-02 03:55:32,535 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640 is 192.168.2.254 and allocation pool is NOT set 2025-10-02 03:55:36,610 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2025-10-02 03:55:36,610 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-07593574 failed! 2025-10-02 03:55:36,623 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640 is 192.168.2.254 and allocation pool is NOT set 2025-10-02 03:55:40,710 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1021ms') 2025-10-02 03:55:40,710 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-07593574 failed! 2025-10-02 03:55:40,728 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640 is 192.168.2.254 and allocation pool is NOT set 2025-10-02 03:55:44,805 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2025-10-02 03:55:44,805 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-07593574 failed! 2025-10-02 03:55:44,820 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640 is 192.168.2.254 and allocation pool is NOT set 2025-10-02 03:55:48,899 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2025-10-02 03:55:48,900 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-07593574 failed! 2025-10-02 03:55:48,914 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640 is 192.168.2.254 and allocation pool is NOT set 2025-10-02 03:55:52,991 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-10-02 03:55:52,991 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-07593574 failed! 2025-10-02 03:55:53,004 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640 is 192.168.2.254 and allocation pool is NOT set 2025-10-02 03:55:57,089 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=5.68 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 1 received, 50% packet loss, time 1021ms\r\nrtt min/avg/max/mdev = 5.675/5.675/5.675/0.000 ms') 2025-10-02 03:55:57,089 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-07593574 passed 2025-10-02 03:55:57,089 - INFO - VM ctest-vn2_vm2-07593574 verifications in Compute nodes passed 2025-10-02 03:55:57,289 - DEBUG - Validated VM route 192.168.2.251 in vrouter of 10.0.0.33 2025-10-02 03:55:57,295 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-10-02 03:55:57,295 - INFO - Validated routes of VM ctest-vn2_vm2-07593574 in all vrouters 2025-10-02 03:55:57,730 - DEBUG - Starting all layer2 verification in 10.0.0.38 Control Node 2025-10-02 03:55:57,860 - DEBUG - Layer2 route found for VM MAC 02:03:c1:76:dd:3b in Control-node 10.0.0.38 2025-10-02 03:55:57,860 - DEBUG - L2 Label for VM ctest-vn2_vm2-07593574 same between Control-node 10.0.0.38 and Agent, Expected: 34, Seen: 34 2025-10-02 03:55:57,860 - DEBUG - Starting all layer2 verification in 10.0.0.49 Control Node 2025-10-02 03:55:57,996 - DEBUG - Layer2 route found for VM MAC 02:03:c1:76:dd:3b in Control-node 10.0.0.49 2025-10-02 03:55:57,996 - DEBUG - L2 Label for VM ctest-vn2_vm2-07593574 same between Control-node 10.0.0.49 and Agent, Expected: 34, Seen: 34 2025-10-02 03:55:57,996 - DEBUG - Starting all layer2 verification in 10.0.0.56 Control Node 2025-10-02 03:55:58,131 - DEBUG - Layer2 route found for VM MAC 02:03:c1:76:dd:3b in Control-node 10.0.0.56 2025-10-02 03:55:58,131 - DEBUG - L2 Label for VM ctest-vn2_vm2-07593574 same between Control-node 10.0.0.56 and Agent, Expected: 34, Seen: 34 2025-10-02 03:55:58,131 - INFO - Verification in Control-nodes for VM ctest-vn2_vm2-07593574 passed 2025-10-02 03:55:58,131 - DEBUG - Verifying the vm in opserver 2025-10-02 03:55:58,131 - DEBUG - Verifying in collector 10.0.0.49 ... 2025-10-02 03:55:58,131 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-machine/416cc01f-f406-4067-bed8-6ccb1b3afc1d?flat 2025-10-02 03:55:58,149 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-35143319:03c176dd-3b7b-4678-82be-9c2b633b9d53?flat 2025-10-02 03:55:58,169 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-35143319:03c176dd-3b7b-4678-82be-9c2b633b9d53?flat 2025-10-02 03:55:58,187 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-machine/416cc01f-f406-4067-bed8-6ccb1b3afc1d?flat 2025-10-02 03:55:58,204 - DEBUG - Verifying vm in vn uve 2025-10-02 03:55:58,204 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-35143319:03c176dd-3b7b-4678-82be-9c2b633b9d53?flat 2025-10-02 03:55:58,223 - DEBUG - VM uve shows interface as {'__T': 1759377342486008, 'ip6_active': False, 'rx_vlan': 65535, 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640', 'ip4_active': True, 'vm_name': 'ctest-vn2_vm2-07593574', 'port_mirror_enabled': False, 'is_health_check_active': True, 'active': True, 'label': 30, 'vhostuser_mode': 0, 'ip_address': '192.168.2.251', 'tx_vlan': 65535, 'admin_state': True, 'l2_active': True, 'mac_address': '02:03:c1:76:dd:3b', 'vm_uuid': '416cc01f-f406-4067-bed8-6ccb1b3afc1d', 'vn_uuid': '13750373-7837-4fb1-ab88-a4e0114a9102', 'ip6_address': '::', 'uuid': '03c176dd-3b7b-4678-82be-9c2b633b9d53', 'fixed_ip4_list': ['192.168.2.251'], 'gateway': '192.168.2.254'} 2025-10-02 03:55:58,224 - DEBUG - VM uve shows ip address as ['192.168.2.251', '::'] 2025-10-02 03:55:58,224 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640 2025-10-02 03:55:58,224 - INFO - Verifying through opserver in 10.0.0.56 2025-10-02 03:55:58,224 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640?flat 2025-10-02 03:55:58,246 - DEBUG - expected vm list 416cc01f-f406-4067-bed8-6ccb1b3afc1d 2025-10-02 03:55:58,246 - DEBUG - Extracted vm list ['416cc01f-f406-4067-bed8-6ccb1b3afc1d'] 2025-10-02 03:55:58,246 - INFO - VM 416cc01f-f406-4067-bed8-6ccb1b3afc1d is present in default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640 2025-10-02 03:55:58,246 - DEBUG - Verifying vm in vrouter uve 2025-10-02 03:55:58,246 - DEBUG - Getting info from collector 10.0.0.49.. 2025-10-02 03:55:58,246 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-machine/416cc01f-f406-4067-bed8-6ccb1b3afc1d?flat 2025-10-02 03:55:58,262 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4210-1'] 2025-10-02 03:55:58,262 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4210-1:Compute:contrail-vrouter-agent:0?flat 2025-10-02 03:55:58,316 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4210-1?flat 2025-10-02 03:55:58,344 - DEBUG - VM 416cc01f-f406-4067-bed8-6ccb1b3afc1d is present in vrouter an-jenkins-deploy-platform-ansible-os-4210-1 uve 2025-10-02 03:55:58,344 - DEBUG - Validated that VM ctest-vn2_vm2-07593574 is in Vrouter an-jenkins-deploy-platform-ansible-os-4210-1 UVE 2025-10-02 03:55:58,345 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-10-02 03:55:58,352 - DEBUG - Expected tap interface of VM uuid 416cc01f-f406-4067-bed8-6ccb1b3afc1d is default-domain:ctest-TestBasicVMVN0-35143319:03c176dd-3b7b-4678-82be-9c2b633b9d53 2025-10-02 03:55:58,352 - DEBUG - Expected VN of VM uuid 416cc01f-f406-4067-bed8-6ccb1b3afc1d is default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640 2025-10-02 03:55:58,352 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4210-1:Compute:contrail-vrouter-agent:0?flat 2025-10-02 03:55:58,407 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4210-1?flat 2025-10-02 03:55:58,437 - DEBUG - VM 416cc01f-f406-4067-bed8-6ccb1b3afc1d is present in vrouter an-jenkins-deploy-platform-ansible-os-4210-1 uve 2025-10-02 03:55:58,437 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-35143319:03c176dd-3b7b-4678-82be-9c2b633b9d53 of vm 416cc01f-f406-4067-bed8-6ccb1b3afc1d is present in vrouter an-jenkins-deploy-platform-ansible-os-4210-1 uve 2025-10-02 03:55:58,437 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640 of vm 416cc01f-f406-4067-bed8-6ccb1b3afc1d is present in vrouter an-jenkins-deploy-platform-ansible-os-4210-1 uve 2025-10-02 03:55:58,437 - DEBUG - Verifying the 416cc01f-f406-4067-bed8-6ccb1b3afc1d virtual network link through opserver 10.0.0.56 2025-10-02 03:55:58,437 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machines 2025-10-02 03:55:58,450 - DEBUG - VM link and name as {'name': '416cc01f-f406-4067-bed8-6ccb1b3afc1d', 'href': 'http://10.0.0.56:8081/analytics/uves/virtual-machine/416cc01f-f406-4067-bed8-6ccb1b3afc1d?flat'} 2025-10-02 03:55:58,450 - DEBUG - Verifying the 416cc01f-f406-4067-bed8-6ccb1b3afc1d virtual network link through opserver 10.0.0.38 2025-10-02 03:55:58,450 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-10-02 03:55:58,462 - DEBUG - VM link and name as {'name': '416cc01f-f406-4067-bed8-6ccb1b3afc1d', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-machine/416cc01f-f406-4067-bed8-6ccb1b3afc1d?flat'} 2025-10-02 03:55:58,462 - DEBUG - Verifying the 416cc01f-f406-4067-bed8-6ccb1b3afc1d virtual network link through opserver 10.0.0.49 2025-10-02 03:55:58,462 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-machines 2025-10-02 03:55:58,475 - DEBUG - VM link and name as {'name': '416cc01f-f406-4067-bed8-6ccb1b3afc1d', 'href': 'http://10.0.0.49:8081/analytics/uves/virtual-machine/416cc01f-f406-4067-bed8-6ccb1b3afc1d?flat'} 2025-10-02 03:55:58,475 - INFO - VM ctest-vn2_vm2-07593574 validations in Opserver passed 2025-10-02 03:55:58,476 - INFO - Waiting for VM ctest-vn1_vm2-75628550 to be up.. 2025-10-02 03:55:58,476 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:55:58,476 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/5050a030-f368-49d2-8eab-52b7bdaf41e5 2025-10-02 03:55:58,485 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine-interface/7543745a-c217-4a34-8449-4ddf8cb62263 2025-10-02 03:55:59,655 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=3.67 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.433 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 0.433/2.051/3.669/1.618 ms') 2025-10-02 03:55:59,655 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-75628550 passed 2025-10-02 03:55:59,728 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:55:59,728 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-75628550, IP 192.168.1.251, Port 22 2025-10-02 03:55:59,902 - DEBUG - VM ctest-vn1_vm2-75628550 is ready for SSH connections 2025-10-02 03:55:59,902 - INFO - Waiting for VM ctest-vn2_vm2-07593574 to be up.. 2025-10-02 03:55:59,902 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:55:59,902 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/416cc01f-f406-4067-bed8-6ccb1b3afc1d 2025-10-02 03:55:59,913 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine-interface/03c176dd-3b7b-4678-82be-9c2b633b9d53 2025-10-02 03:56:01,073 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=1.31 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.462 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.462/0.885/1.309/0.423 ms') 2025-10-02 03:56:01,073 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-07593574 passed 2025-10-02 03:56:01,156 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:56:01,156 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-07593574, IP 192.168.2.251, Port 22 2025-10-02 03:56:01,213 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-02 03:56:01,309 - DEBUG - VM ctest-vn2_vm2-07593574 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-02 03:56:06,310 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:56:06,310 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-07593574, IP 192.168.2.251, Port 22 2025-10-02 03:56:06,365 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-02 03:56:06,442 - DEBUG - VM ctest-vn2_vm2-07593574 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-02 03:56:11,443 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:56:11,443 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-07593574, IP 192.168.2.251, Port 22 2025-10-02 03:56:11,512 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-02 03:56:11,613 - DEBUG - VM ctest-vn2_vm2-07593574 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-02 03:56:16,613 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:56:16,613 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-07593574, IP 192.168.2.251, Port 22 2025-10-02 03:56:16,770 - DEBUG - VM ctest-vn2_vm2-07593574 is ready for SSH connections 2025-10-02 03:56:16,770 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-02 03:56:16,770 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.251, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-10-02 03:56:16,770 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.251 2025-10-02 03:56:58,322 - DEBUG - PING 192.168.2.251 (192.168.2.251) 56(84) bytes of data. 64 bytes from 192.168.2.251: icmp_req=1 ttl=63 time=9.65 ms 64 bytes from 192.168.2.251: icmp_req=2 ttl=63 time=1.12 ms 64 bytes from 192.168.2.251: icmp_req=3 ttl=63 time=0.982 ms --- 192.168.2.251 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 0.982/3.921/9.652/4.052 ms 2025-10-02 03:56:58,322 - INFO - Ping to IP 192.168.2.251 from VM ctest-vn1_vm2-75628550 passed 2025-10-02 03:56:58,322 - INFO - Deleting VM ctest-vn2_vm2-07593574 2025-10-02 03:56:58,425 - DEBUG - Verifying in api server 10.0.0.49 2025-10-02 03:56:58,425 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/416cc01f-f406-4067-bed8-6ccb1b3afc1d 2025-10-02 03:56:58,435 - DEBUG - VM ID 416cc01f-f406-4067-bed8-6ccb1b3afc1d of VM ctest-vn2_vm2-07593574 is still found in API Server 2025-10-02 03:57:00,436 - DEBUG - Verifying in api server 10.0.0.49 2025-10-02 03:57:00,436 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/416cc01f-f406-4067-bed8-6ccb1b3afc1d 2025-10-02 03:57:00,443 - DEBUG - Response Code: 404 2025-10-02 03:57:00,443 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/416cc01f-f406-4067-bed8-6ccb1b3afc1d 2025-10-02 03:57:00,449 - DEBUG - Response Code: 404 2025-10-02 03:57:00,449 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/416cc01f-f406-4067-bed8-6ccb1b3afc1d 2025-10-02 03:57:00,455 - DEBUG - Response Code: 404 2025-10-02 03:57:00,455 - INFO - VM ctest-vn2_vm2-07593574 is fully removed in API-Server 2025-10-02 03:57:00,605 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-10-02 03:57:00,758 - DEBUG - VM route 192.168.2.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640 of compute 10.0.0.33 2025-10-02 03:57:00,763 - DEBUG - Validated that vrouter 10.0.0.33 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640 2025-10-02 03:57:00,763 - DEBUG - Validated that vrouter 10.0.0.130 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640 2025-10-02 03:57:00,763 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640 2025-10-02 03:57:00,763 - INFO - VM ctest-vn2_vm2-07593574 is removed in Compute, and routes are removed in all compute nodes 2025-10-02 03:57:01,004 - INFO - Routes for VM ctest-vn2_vm2-07593574 is removed in all control-nodes 2025-10-02 03:57:01,004 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4210-1:Compute:contrail-vrouter-agent:0?flat 2025-10-02 03:57:01,058 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4210-1?flat 2025-10-02 03:57:01,091 - DEBUG - VM 416cc01f-f406-4067-bed8-6ccb1b3afc1d is still present in vrouter an-jenkins-deploy-platform-ansible-os-4210-1 uve 2025-10-02 03:57:01,091 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-02 03:57:01,091 - DEBUG - interface for vm 416cc01f-f406-4067-bed8-6ccb1b3afc1d deleted from vrouter uve ... 2025-10-02 03:57:01,091 - DEBUG - Deleted interface not in error interface list ... 2025-10-02 03:57:04,092 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4210-1:Compute:contrail-vrouter-agent:0?flat 2025-10-02 03:57:04,146 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4210-1?flat 2025-10-02 03:57:04,177 - DEBUG - VM 416cc01f-f406-4067-bed8-6ccb1b3afc1d is still present in vrouter an-jenkins-deploy-platform-ansible-os-4210-1 uve 2025-10-02 03:57:04,177 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-02 03:57:04,177 - DEBUG - interface for vm 416cc01f-f406-4067-bed8-6ccb1b3afc1d deleted from vrouter uve ... 2025-10-02 03:57:04,177 - DEBUG - Deleted interface not in error interface list ... 2025-10-02 03:57:07,178 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4210-1:Compute:contrail-vrouter-agent:0?flat 2025-10-02 03:57:07,248 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4210-1?flat 2025-10-02 03:57:07,277 - DEBUG - VM 416cc01f-f406-4067-bed8-6ccb1b3afc1d is still present in vrouter an-jenkins-deploy-platform-ansible-os-4210-1 uve 2025-10-02 03:57:07,277 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-02 03:57:07,277 - DEBUG - interface for vm 416cc01f-f406-4067-bed8-6ccb1b3afc1d deleted from vrouter uve ... 2025-10-02 03:57:07,277 - DEBUG - Deleted interface not in error interface list ... 2025-10-02 03:57:10,278 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4210-1:Compute:contrail-vrouter-agent:0?flat 2025-10-02 03:57:10,344 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4210-1?flat 2025-10-02 03:57:10,376 - DEBUG - VM 416cc01f-f406-4067-bed8-6ccb1b3afc1d is still present in vrouter an-jenkins-deploy-platform-ansible-os-4210-1 uve 2025-10-02 03:57:10,376 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-02 03:57:10,376 - DEBUG - interface for vm 416cc01f-f406-4067-bed8-6ccb1b3afc1d deleted from vrouter uve ... 2025-10-02 03:57:10,376 - DEBUG - Deleted interface not in error interface list ... 2025-10-02 03:57:13,377 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4210-1:Compute:contrail-vrouter-agent:0?flat 2025-10-02 03:57:13,429 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4210-1?flat 2025-10-02 03:57:13,457 - DEBUG - VM 416cc01f-f406-4067-bed8-6ccb1b3afc1d is not present in vrouter an-jenkins-deploy-platform-ansible-os-4210-1 uve 2025-10-02 03:57:13,457 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-02 03:57:13,457 - DEBUG - interface for vm 416cc01f-f406-4067-bed8-6ccb1b3afc1d deleted from vrouter uve ... 2025-10-02 03:57:13,457 - DEBUG - Deleted interface not in error interface list ... 2025-10-02 03:57:13,457 - DEBUG - Validated that VM 416cc01f-f406-4067-bed8-6ccb1b3afc1d is removed in Vrouter UVE 2025-10-02 03:57:13,457 - DEBUG - Verifying through opserver in 10.0.0.56 2025-10-02 03:57:13,457 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640?flat 2025-10-02 03:57:13,476 - DEBUG - 416cc01f-f406-4067-bed8-6ccb1b3afc1d BM not in default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn2-20688640 uve 2025-10-02 03:57:13,476 - DEBUG - Validated that VM 416cc01f-f406-4067-bed8-6ccb1b3afc1d is not present in the VN UVE 2025-10-02 03:57:13,476 - DEBUG - Verifying the 416cc01f-f406-4067-bed8-6ccb1b3afc1d virtual network link through opserver 10.0.0.56 2025-10-02 03:57:13,476 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machines 2025-10-02 03:57:13,487 - INFO - 416cc01f-f406-4067-bed8-6ccb1b3afc1d vm uve deleted from opserver 2025-10-02 03:57:13,488 - DEBUG - Verifying the 416cc01f-f406-4067-bed8-6ccb1b3afc1d virtual network link through opserver 10.0.0.38 2025-10-02 03:57:13,488 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-10-02 03:57:13,541 - INFO - 416cc01f-f406-4067-bed8-6ccb1b3afc1d vm uve deleted from opserver 2025-10-02 03:57:13,541 - DEBUG - Verifying the 416cc01f-f406-4067-bed8-6ccb1b3afc1d virtual network link through opserver 10.0.0.49 2025-10-02 03:57:13,541 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-machines 2025-10-02 03:57:13,554 - INFO - 416cc01f-f406-4067-bed8-6ccb1b3afc1d vm uve deleted from opserver 2025-10-02 03:57:13,554 - INFO - Validated that VM 416cc01f-f406-4067-bed8-6ccb1b3afc1d is removed from Opserver 2025-10-02 03:57:13,554 - INFO - Deleting VM ctest-vn1_vm2-75628550 2025-10-02 03:57:13,621 - DEBUG - Verifying in api server 10.0.0.49 2025-10-02 03:57:13,621 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/5050a030-f368-49d2-8eab-52b7bdaf41e5 2025-10-02 03:57:13,630 - DEBUG - VM ID 5050a030-f368-49d2-8eab-52b7bdaf41e5 of VM ctest-vn1_vm2-75628550 is still found in API Server 2025-10-02 03:57:15,632 - DEBUG - Verifying in api server 10.0.0.49 2025-10-02 03:57:15,632 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/5050a030-f368-49d2-8eab-52b7bdaf41e5 2025-10-02 03:57:15,639 - DEBUG - Response Code: 404 2025-10-02 03:57:15,639 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/5050a030-f368-49d2-8eab-52b7bdaf41e5 2025-10-02 03:57:15,646 - DEBUG - Response Code: 404 2025-10-02 03:57:15,646 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-machine/5050a030-f368-49d2-8eab-52b7bdaf41e5 2025-10-02 03:57:15,653 - DEBUG - Response Code: 404 2025-10-02 03:57:15,653 - INFO - VM ctest-vn1_vm2-75628550 is fully removed in API-Server 2025-10-02 03:57:15,821 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-10-02 03:57:15,854 - DEBUG - Validated that vrouter 10.0.0.33 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 2025-10-02 03:57:15,991 - DEBUG - VM route 192.168.1.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 of compute 10.0.0.130 2025-10-02 03:57:15,998 - DEBUG - Validated that vrouter 10.0.0.130 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 2025-10-02 03:57:15,999 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 2025-10-02 03:57:15,999 - INFO - VM ctest-vn1_vm2-75628550 is removed in Compute, and routes are removed in all compute nodes 2025-10-02 03:57:16,229 - INFO - Routes for VM ctest-vn1_vm2-75628550 is removed in all control-nodes 2025-10-02 03:57:16,229 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4210-2:Compute:contrail-vrouter-agent:0?flat 2025-10-02 03:57:16,277 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4210-2?flat 2025-10-02 03:57:16,311 - DEBUG - VM 5050a030-f368-49d2-8eab-52b7bdaf41e5 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4210-2 uve 2025-10-02 03:57:16,311 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-02 03:57:16,311 - DEBUG - interface for vm 5050a030-f368-49d2-8eab-52b7bdaf41e5 deleted from vrouter uve ... 2025-10-02 03:57:16,311 - DEBUG - Deleted interface not in error interface list ... 2025-10-02 03:57:19,312 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4210-2:Compute:contrail-vrouter-agent:0?flat 2025-10-02 03:57:19,360 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4210-2?flat 2025-10-02 03:57:19,388 - DEBUG - VM 5050a030-f368-49d2-8eab-52b7bdaf41e5 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4210-2 uve 2025-10-02 03:57:19,388 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-02 03:57:19,389 - DEBUG - interface for vm 5050a030-f368-49d2-8eab-52b7bdaf41e5 deleted from vrouter uve ... 2025-10-02 03:57:19,389 - DEBUG - Deleted interface not in error interface list ... 2025-10-02 03:57:22,389 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4210-2:Compute:contrail-vrouter-agent:0?flat 2025-10-02 03:57:22,433 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4210-2?flat 2025-10-02 03:57:22,469 - DEBUG - VM 5050a030-f368-49d2-8eab-52b7bdaf41e5 is not present in vrouter an-jenkins-deploy-platform-ansible-os-4210-2 uve 2025-10-02 03:57:22,469 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-02 03:57:22,469 - DEBUG - interface for vm 5050a030-f368-49d2-8eab-52b7bdaf41e5 deleted from vrouter uve ... 2025-10-02 03:57:22,469 - DEBUG - Deleted interface not in error interface list ... 2025-10-02 03:57:22,469 - DEBUG - Validated that VM 5050a030-f368-49d2-8eab-52b7bdaf41e5 is removed in Vrouter UVE 2025-10-02 03:57:22,469 - DEBUG - Verifying through opserver in 10.0.0.56 2025-10-02 03:57:22,469 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616?flat 2025-10-02 03:57:22,489 - DEBUG - 5050a030-f368-49d2-8eab-52b7bdaf41e5 BM not in default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 uve 2025-10-02 03:57:22,489 - DEBUG - Validated that VM 5050a030-f368-49d2-8eab-52b7bdaf41e5 is not present in the VN UVE 2025-10-02 03:57:22,489 - DEBUG - Verifying the 5050a030-f368-49d2-8eab-52b7bdaf41e5 virtual network link through opserver 10.0.0.56 2025-10-02 03:57:22,489 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machines 2025-10-02 03:57:22,500 - INFO - 5050a030-f368-49d2-8eab-52b7bdaf41e5 vm uve deleted from opserver 2025-10-02 03:57:22,500 - DEBUG - Verifying the 5050a030-f368-49d2-8eab-52b7bdaf41e5 virtual network link through opserver 10.0.0.38 2025-10-02 03:57:22,500 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-10-02 03:57:22,515 - INFO - 5050a030-f368-49d2-8eab-52b7bdaf41e5 vm uve deleted from opserver 2025-10-02 03:57:22,515 - DEBUG - Verifying the 5050a030-f368-49d2-8eab-52b7bdaf41e5 virtual network link through opserver 10.0.0.49 2025-10-02 03:57:22,515 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-machines 2025-10-02 03:57:22,528 - INFO - 5050a030-f368-49d2-8eab-52b7bdaf41e5 vm uve deleted from opserver 2025-10-02 03:57:22,528 - INFO - Validated that VM 5050a030-f368-49d2-8eab-52b7bdaf41e5 is removed from Opserver 2025-10-02 03:57:22,529 - INFO - Deleting VN ctest-vn3-32047234 2025-10-02 03:57:22,645 - INFO - Deleting VM ctest-vn2_vm1-84787369 2025-10-02 03:57:22,714 - INFO - Deleting VM ctest-vn1_vm1-70599077 2025-10-02 03:57:22,955 - INFO - Deleting VN ctest-vn2-20688640 2025-10-02 03:57:23,010 - DEBUG - RefsExistError Delete when resource still referred: ['http://10.0.0.49:8082/virtual-machine-interface/5e1fbfcb-8857-4bb0-8079-050606f132f3'] while deleting VN ctest-vn2-20688640..Will retry 2025-10-02 03:57:25,179 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/63da8ad1-305a-4d44-b7b6-3496eaa3bdc6 2025-10-02 03:57:25,184 - DEBUG - Response Code: 404 2025-10-02 03:57:25,184 - DEBUG - Requesting: http://10.0.0.56:8082/domains 2025-10-02 03:57:25,196 - DEBUG - Requesting: http://10.0.0.56:8082/domain/b0543e69-2184-4be1-8e23-ac5840233bc6 2025-10-02 03:57:25,210 - DEBUG - Requesting: http://10.0.0.56:8082/project/7612bb4f-d0d8-4c4e-9c49-460de03b6971 2025-10-02 03:57:25,293 - INFO - Validated that VN ctest-vn2-20688640 is not found in API Server 2025-10-02 03:57:25,304 - DEBUG - VN ctest-vn2-20688640 is not present in Agent 10.0.0.33 2025-10-02 03:57:25,323 - DEBUG - VN ctest-vn2-20688640 is not present in Agent 10.0.0.130 2025-10-02 03:57:25,323 - INFO - Validated that VN ctest-vn2-20688640 is not in any agent 2025-10-02 03:57:25,356 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn2-20688640 info 2025-10-02 03:57:25,493 - INFO - Deleting VN ctest-vn1-80182616 2025-10-02 03:57:25,607 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/bff0383e-27bf-4de3-a2c9-15fd9d24118f 2025-10-02 03:57:25,613 - DEBUG - Response Code: 404 2025-10-02 03:57:25,613 - DEBUG - Requesting: http://10.0.0.56:8082/domains 2025-10-02 03:57:25,619 - DEBUG - Requesting: http://10.0.0.56:8082/domain/b0543e69-2184-4be1-8e23-ac5840233bc6 2025-10-02 03:57:25,637 - DEBUG - Requesting: http://10.0.0.56:8082/project/7612bb4f-d0d8-4c4e-9c49-460de03b6971 2025-10-02 03:57:25,727 - INFO - Validated that VN ctest-vn1-80182616 is not found in API Server 2025-10-02 03:57:25,738 - DEBUG - VN ctest-vn1-80182616 is not present in Agent 10.0.0.33 2025-10-02 03:57:25,748 - DEBUG - VN ctest-vn1-80182616 is not present in Agent 10.0.0.130 2025-10-02 03:57:25,748 - INFO - Validated that VN ctest-vn1-80182616 is not in any agent 2025-10-02 03:57:25,754 - DEBUG - VRF 2 is not seen in agent 10.0.0.130 2025-10-02 03:57:25,759 - DEBUG - Vrouter 10.0.0.130 does not have vrf 2 for VN ctest-vn1-80182616 2025-10-02 03:57:25,759 - INFO - Validated that all vrouters do not have the route table for VN default-domain:ctest-TestBasicVMVN0-35143319:ctest-vn1-80182616 2025-10-02 03:57:25,778 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-80182616 info 2025-10-02 03:57:25,818 - INFO - Deleted policy policy2 2025-10-02 03:57:25,855 - INFO - Deleted policy policy1 2025-10-02 03:57:27,128 - DEBUG - Skipping xmpp flap check 2025-10-02 03:57:27,128 - INFO - END TEST : test_process_restart_in_policy_between_vns : PASSED[0:05:52] 2025-10-02 03:57:27,128 - INFO - -------------------------------------------------------------------------------- 2025-10-02 03:57:27,131 - INFO - ================================================================================ 2025-10-02 03:57:27,131 - INFO - STARTING TEST : test_underlay_broadcast_traffic_handling 2025-10-02 03:57:27,131 - INFO - TEST DESCRIPTION : Test the underlay brocast traffic handling by vrouter. (Bug-1545229). 1. Send broadcast traffic from one compute node. 2. Other compute in same subnet should receive that traffic. 3. Receiving compute should treat this traffic as underlay. 4. Compute should not replicate the packet and send the copy back. Pass criteria: Step 3-4 should pass Maintainer : chhandak@juniper.net 2025-10-02 03:57:28,416 - DEBUG - Skipping xmpp flap check 2025-10-02 03:57:28,416 - INFO - Initial checks done. Running the testcase now 2025-10-02 03:57:28,416 - INFO - 2025-10-02 03:57:28,813 - INFO - Agent interface name: ens6 2025-10-02 03:57:28,893 - DEBUG - Executing command: sudo tcpdump -nni ens6 -U host 0.0.0.0 -w /tmp/ens6_ctest-random-47830828.pcap 2025-10-02 03:57:34,280 - INFO - Pinging broacast address 0.0.0.0 from compute 10.20.0.252 2025-10-02 03:57:48,605 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-10-02 03:57:50,742 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-47830828.pcap | grep -c length 2025-10-02 03:57:50,756 - DEBUG - STDOUT: 10 2025-10-02 03:57:50,756 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-47830828.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-10-02 03:57:50,756 - INFO - 10 packets are found in tcpdump output as expected 2025-10-02 03:57:50,757 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-10-02 03:57:52,805 - INFO - Packet count matched: Compute 10.0.0.130 has receive only 10 packet from source IP 0.0.0.0. No duplicate packet seen 2025-10-02 03:57:54,068 - DEBUG - Skipping xmpp flap check 2025-10-02 03:57:54,068 - INFO - END TEST : test_underlay_broadcast_traffic_handling : PASSED[0:00:27] 2025-10-02 03:57:54,068 - INFO - -------------------------------------------------------------------------------- 2025-10-02 03:57:54,810 - INFO - Deleted project: ctest-TestBasicVMVN0-35143319, ID : 7612bb4f-d0d8-4c4e-9c49-460de03b6971