2026-03-23 04:25:24,328 - INFO - Domain Default found not creating 2026-03-23 04:25:24,554 - INFO - Project ctest-TestBasicVMVN0-88137413 not found, creating it 2026-03-23 04:25:25,148 - INFO - Created Project:ctest-TestBasicVMVN0-88137413, ID : c818fea2-bb8c-4dc4-8c5c-a693e3a97287 2026-03-23 04:25:26,904 - INFO - ================================================================================ 2026-03-23 04:25:26,904 - INFO - STARTING TEST : test_control_node_switchover 2026-03-23 04:25:26,904 - INFO - TEST DESCRIPTION : Stop the control node and check peering with agent fallback to other control node. 1. Pick one VN from respource pool which has 2 VM's in it 2. Verify ping between VM's 3. Find active control node in cluster by agent inspect 4. Stop control service on active control node 5. Verify agents are connected to new active control-node using xmpp connections 6. Bring back control service on previous active node 7. Verify ping between VM's again after bringing up control serveice Pass criteria: Step 2,5 and 7 should pass 2026-03-23 04:25:28,194 - DEBUG - Skipping xmpp flap check 2026-03-23 04:25:28,194 - INFO - Initial checks done. Running the testcase now 2026-03-23 04:25:28,194 - INFO - 2026-03-23 04:25:28,966 - DEBUG - Response for create_network : {'network': {'id': '85170818-fdb6-4ffe-97b2-b69ec42fae4e', 'name': 'ctest-vn1-82380926', 'tenant_id': 'c818fea2bb8c4dc48c5ca693e3a97287', 'project_id': 'c818fea2bb8c4dc48c5ca693e3a97287', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestBasicVMVN0-88137413', 'ctest-vn1-82380926'], 'port_security_enabled': True, 'description': ''}} 2026-03-23 04:25:29,315 - DEBUG - Response for create_subnet : {'subnet': {'id': 'a4c94787-89bf-4ff7-a681-442a52292d22', 'name': '', 'tenant_id': 'c818fea2bb8c4dc48c5ca693e3a97287', 'network_id': '85170818-fdb6-4ffe-97b2-b69ec42fae4e', 'ip_version': 4, 'cidr': '192.168.1.0/24', 'allocation_pools': [{'start': '192.168.1.2', 'end': '192.168.1.254'}], 'gateway_ip': '192.168.1.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '192.168.1.2', 'tags': [], 'project_id': 'c818fea2bb8c4dc48c5ca693e3a97287'}} 2026-03-23 04:25:29,343 - INFO - Created VN ctest-vn1-82380926 2026-03-23 04:25:29,358 - DEBUG - VN ctest-vn1-82380926 UUID is 85170818-fdb6-4ffe-97b2-b69ec42fae4e 2026-03-23 04:25:29,444 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-03-23 04:25:29,776 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-03-23 04:25:29,817 - DEBUG - Requesting: http://10.0.0.27:8082/domain/54adab80-1c58-45eb-8fc6-565ee3803789 2026-03-23 04:25:29,835 - DEBUG - Requesting: http://10.0.0.27:8082/project/c818fea2-bb8c-4dc4-8c5c-a693e3a97287 2026-03-23 04:25:29,952 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/85170818-fdb6-4ffe-97b2-b69ec42fae4e 2026-03-23 04:25:29,965 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/85170818-fdb6-4ffe-97b2-b69ec42fae4e 2026-03-23 04:25:29,976 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/17973f9b-c6aa-4a8f-98c9-9e4edefec96f 2026-03-23 04:25:29,988 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/17973f9b-c6aa-4a8f-98c9-9e4edefec96f 2026-03-23 04:25:29,998 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/1534cdd2-b0ad-4b8a-abb1-f4c2d8727cc5 2026-03-23 04:25:30,008 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-03-23 04:25:30,008 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/85170818-fdb6-4ffe-97b2-b69ec42fae4e 2026-03-23 04:25:30,019 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/17973f9b-c6aa-4a8f-98c9-9e4edefec96f 2026-03-23 04:25:30,038 - INFO - Verified VN network id 15 for VN 85170818-fdb6-4ffe-97b2-b69ec42fae4e 2026-03-23 04:25:30,039 - INFO - Verifications in API Server for VN ctest-vn1-82380926 passed 2026-03-23 04:25:30,039 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/85170818-fdb6-4ffe-97b2-b69ec42fae4e 2026-03-23 04:25:30,055 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/17973f9b-c6aa-4a8f-98c9-9e4edefec96f 2026-03-23 04:25:30,064 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/17973f9b-c6aa-4a8f-98c9-9e4edefec96f 2026-03-23 04:25:30,073 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/1534cdd2-b0ad-4b8a-abb1-f4c2d8727cc5 2026-03-23 04:25:30,100 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-88137413', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '9590142834937581566', 'uuid-lslong': '10931000038594162254'}, 'enable': 'true', 'created': '2026-03-23T04:25:28', 'last-modified': '2026-03-23T04:25:29', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.891833'} 2026-03-23 04:25:30,109 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-03-23 04:25:30,116 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-88137413', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '9590142834937581566', 'uuid-lslong': '10931000038594162254'}, 'enable': 'true', 'created': '2026-03-23T04:25:28', 'last-modified': '2026-03-23T04:25:29', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.908924'} 2026-03-23 04:25:30,125 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-03-23 04:25:30,133 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-88137413', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '9590142834937581566', 'uuid-lslong': '10931000038594162254'}, 'enable': 'true', 'created': '2026-03-23T04:25:28', 'last-modified': '2026-03-23T04:25:29', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.918646'} 2026-03-23 04:25:30,141 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-03-23 04:25:30,141 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-82380926 passed 2026-03-23 04:25:30,141 - DEBUG - ====Verifying policy data for ctest-vn1-82380926 in API_Server ====== 2026-03-23 04:25:30,141 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-03-23 04:25:30,152 - DEBUG - Requesting: http://10.0.0.27:8082/domain/54adab80-1c58-45eb-8fc6-565ee3803789 2026-03-23 04:25:30,171 - DEBUG - Requesting: http://10.0.0.27:8082/project/c818fea2-bb8c-4dc4-8c5c-a693e3a97287 2026-03-23 04:25:30,272 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/85170818-fdb6-4ffe-97b2-b69ec42fae4e 2026-03-23 04:25:30,284 - DEBUG - =>VN ctest-vn1-82380926 has no policy to be verified 2026-03-23 04:25:30,285 - DEBUG - Verifying the vn in opserver 2026-03-23 04:25:30,285 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 virtual network link through opserver 10.0.0.27 2026-03-23 04:25:30,285 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2026-03-23 04:25:30,605 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2026-03-23 04:25:30,697 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926?flat'} 2026-03-23 04:25:30,698 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 virtual network link through opserver 10.0.0.50 2026-03-23 04:25:30,698 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2026-03-23 04:25:31,029 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2026-03-23 04:25:31,105 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926?flat'} 2026-03-23 04:25:31,105 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 virtual network link through opserver 10.0.0.23 2026-03-23 04:25:31,105 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2026-03-23 04:25:31,427 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2026-03-23 04:25:31,516 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926?flat'} 2026-03-23 04:25:31,516 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 is found in opserver 2026-03-23 04:25:31,544 - DEBUG - Do not have enough data to verify VN in agent 2026-03-23 04:25:31,558 - DEBUG - VRF ids for VN ctest-vn1-82380926: {} 2026-03-23 04:25:31,636 - DEBUG - Services list from nova: [, , , ] 2026-03-23 04:25:32,975 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-5514-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-5514-2) 2026-03-23 04:25:34,577 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-5514-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-5514-1) 2026-03-23 04:25:34,577 - INFO - Waiting for VM ctest-vn1_vm1-42110935 to be up.. 2026-03-23 04:25:34,722 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2026-03-23 04:25:39,832 - DEBUG - VM is in ACTIVE state now 2026-03-23 04:25:39,832 - INFO - VM name : ctest-vn1_vm1-42110935 2026-03-23 04:25:39,935 - DEBUG - VM ctest-vn1_vm1-42110935 ID is 8810c406-dbb4-49e2-9adb-065042ff5d8b 2026-03-23 04:25:39,964 - DEBUG - VM ctest-vn1_vm1-42110935 launched on Node an-jenkins-deploy-platform-ansible-os-5514-2 2026-03-23 04:25:40,084 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/8810c406-dbb4-49e2-9adb-065042ff5d8b 2026-03-23 04:25:40,446 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/8810c406-dbb4-49e2-9adb-065042ff5d8b 2026-03-23 04:25:40,501 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/4ac51bf2-0efe-4e37-8b6d-38010782e1d3 2026-03-23 04:25:43,689 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1006ms') 2026-03-23 04:25:43,689 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-42110935 failed! 2026-03-23 04:25:43,703 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 is 192.168.1.1 and allocation pool is NOT set 2026-03-23 04:25:47,789 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2026-03-23 04:25:47,789 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-42110935 failed! 2026-03-23 04:25:47,805 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 is 192.168.1.1 and allocation pool is NOT set 2026-03-23 04:25:51,883 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms') 2026-03-23 04:25:51,883 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-42110935 failed! 2026-03-23 04:25:51,901 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 is 192.168.1.1 and allocation pool is NOT set 2026-03-23 04:25:55,984 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms') 2026-03-23 04:25:55,984 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-42110935 failed! 2026-03-23 04:25:56,001 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 is 192.168.1.1 and allocation pool is NOT set 2026-03-23 04:26:00,075 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1002ms') 2026-03-23 04:26:00,075 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-42110935 failed! 2026-03-23 04:26:00,090 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 is 192.168.1.1 and allocation pool is NOT set 2026-03-23 04:26:04,176 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2026-03-23 04:26:04,176 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-42110935 failed! 2026-03-23 04:26:04,193 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 is 192.168.1.1 and allocation pool is NOT set 2026-03-23 04:26:08,264 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1003ms') 2026-03-23 04:26:08,264 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-42110935 failed! 2026-03-23 04:26:08,280 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 is 192.168.1.1 and allocation pool is NOT set 2026-03-23 04:26:12,360 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2026-03-23 04:26:12,360 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-42110935 failed! 2026-03-23 04:26:12,375 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 is 192.168.1.1 and allocation pool is NOT set 2026-03-23 04:26:16,456 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2026-03-23 04:26:16,456 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-42110935 failed! 2026-03-23 04:26:16,471 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 is 192.168.1.1 and allocation pool is NOT set 2026-03-23 04:26:18,543 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=6.05 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=3.20 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 3.197/4.624/6.052/1.427 ms') 2026-03-23 04:26:18,543 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-42110935 passed 2026-03-23 04:26:18,612 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:26:18,612 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-42110935, IP 192.168.1.3, Port 22 2026-03-23 04:26:18,680 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-03-23 04:26:18,792 - DEBUG - VM ctest-vn1_vm1-42110935 is NOT ready for SSH connections, VM status: ACTIVE 2026-03-23 04:26:23,794 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:26:23,794 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-42110935, IP 192.168.1.3, Port 22 2026-03-23 04:26:23,864 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-03-23 04:26:23,960 - DEBUG - VM ctest-vn1_vm1-42110935 is NOT ready for SSH connections, VM status: ACTIVE 2026-03-23 04:26:28,961 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:26:28,961 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-42110935, IP 192.168.1.3, Port 22 2026-03-23 04:26:29,028 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-03-23 04:26:29,123 - DEBUG - VM ctest-vn1_vm1-42110935 is NOT ready for SSH connections, VM status: ACTIVE 2026-03-23 04:26:34,124 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:26:34,125 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-42110935, IP 192.168.1.3, Port 22 2026-03-23 04:26:34,192 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-03-23 04:26:34,298 - DEBUG - VM ctest-vn1_vm1-42110935 is NOT ready for SSH connections, VM status: ACTIVE 2026-03-23 04:26:39,298 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:26:39,299 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-42110935, IP 192.168.1.3, Port 22 2026-03-23 04:26:39,471 - DEBUG - VM ctest-vn1_vm1-42110935 is ready for SSH connections 2026-03-23 04:26:39,472 - INFO - Waiting for VM ctest-vn1_vm2-75830156 to be up.. 2026-03-23 04:26:39,574 - DEBUG - VM is in ACTIVE state now 2026-03-23 04:26:39,574 - INFO - VM name : ctest-vn1_vm2-75830156 2026-03-23 04:26:39,683 - DEBUG - VM ctest-vn1_vm2-75830156 ID is 66b61710-fb4a-475e-891a-d151c66dbd84 2026-03-23 04:26:39,683 - DEBUG - VM ctest-vn1_vm2-75830156 launched on Node an-jenkins-deploy-platform-ansible-os-5514-1 2026-03-23 04:26:39,786 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/66b61710-fb4a-475e-891a-d151c66dbd84 2026-03-23 04:26:39,799 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/d546366b-40c3-431a-b492-e54db2b3eadc 2026-03-23 04:26:43,076 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=181 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.542 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.542/90.603/180.664/90.061 ms') 2026-03-23 04:26:43,076 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm2-75830156 passed 2026-03-23 04:26:43,144 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:26:43,144 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-75830156, IP 192.168.1.4, Port 22 2026-03-23 04:26:47,358 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-03-23 04:26:47,458 - DEBUG - VM ctest-vn1_vm2-75830156 is NOT ready for SSH connections, VM status: ACTIVE 2026-03-23 04:26:52,458 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:26:52,458 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-75830156, IP 192.168.1.4, Port 22 2026-03-23 04:26:52,919 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-03-23 04:26:53,024 - DEBUG - VM ctest-vn1_vm2-75830156 is NOT ready for SSH connections, VM status: ACTIVE 2026-03-23 04:26:58,024 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:26:58,025 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-75830156, IP 192.168.1.4, Port 22 2026-03-23 04:26:58,085 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-03-23 04:26:58,185 - DEBUG - VM ctest-vn1_vm2-75830156 is NOT ready for SSH connections, VM status: ACTIVE 2026-03-23 04:27:03,186 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:27:03,186 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-75830156, IP 192.168.1.4, Port 22 2026-03-23 04:27:03,339 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-03-23 04:27:03,449 - DEBUG - VM ctest-vn1_vm2-75830156 is NOT ready for SSH connections, VM status: ACTIVE 2026-03-23 04:27:08,450 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:27:08,450 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-75830156, IP 192.168.1.4, Port 22 2026-03-23 04:27:08,789 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-03-23 04:27:08,887 - DEBUG - VM ctest-vn1_vm2-75830156 is NOT ready for SSH connections, VM status: ACTIVE 2026-03-23 04:27:13,887 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:27:13,887 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-75830156, IP 192.168.1.4, Port 22 2026-03-23 04:27:13,985 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-03-23 04:27:14,133 - DEBUG - VM ctest-vn1_vm2-75830156 is NOT ready for SSH connections, VM status: ACTIVE 2026-03-23 04:27:19,134 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:27:19,134 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-75830156, IP 192.168.1.4, Port 22 2026-03-23 04:27:19,203 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-03-23 04:27:19,291 - DEBUG - VM ctest-vn1_vm2-75830156 is NOT ready for SSH connections, VM status: ACTIVE 2026-03-23 04:27:24,292 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:27:24,292 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-75830156, IP 192.168.1.4, Port 22 2026-03-23 04:27:24,463 - DEBUG - VM ctest-vn1_vm2-75830156 is ready for SSH connections 2026-03-23 04:27:24,463 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:27:24,463 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.65, gateway password: c0ntrail123 2026-03-23 04:27:24,463 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4 2026-03-23 04:28:06,245 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data. 64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=11.1 ms 64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.42 ms 64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.38 ms --- 192.168.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.383/4.658/11.171/4.605 ms 2026-03-23 04:28:06,245 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-42110935 passed 2026-03-23 04:28:06,245 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:28:06,245 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.249, gateway password: c0ntrail123 2026-03-23 04:28:06,245 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3 2026-03-23 04:28:54,674 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. 64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=8.96 ms 64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.58 ms 64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.27 ms --- 192.168.1.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.270/3.937/8.960/3.554 ms 2026-03-23 04:28:54,674 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-75830156 passed 2026-03-23 04:28:54,682 - INFO - Active control node from the Agent 10.0.0.65 is 10.0.0.27 2026-03-23 04:28:54,683 - INFO - Stoping the Control service in 10.0.0.27 2026-03-23 04:28:55,371 - INFO - 10.0.0.27 2026-03-23 04:28:55,371 - INFO - control:inactive 2026-03-23 04:28:55,371 - INFO - Contrail services control are down on nodes 10.0.0.27 2026-03-23 04:29:00,380 - INFO - Active control node from the Agent 10.0.0.65 is 10.0.0.23 2026-03-23 04:29:00,380 - INFO - Starting the Control service in 10.0.0.27 2026-03-23 04:29:00,892 - INFO - 10.0.0.27 2026-03-23 04:29:00,897 - INFO - control:initializing 2026-03-23 04:29:00,897 - DEBUG - defaultdict(, {'10.0.0.27': {'control': {'status': 'initializing', 'description': None}}}) 2026-03-23 04:29:00,897 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-03-23 04:29:06,020 - INFO - 10.0.0.27 2026-03-23 04:29:06,027 - INFO - control:active 2026-03-23 04:29:06,028 - INFO - Contrail services ['control'] are up on nodes 10.0.0.27 2026-03-23 04:29:11,147 - DEBUG - VM is in ACTIVE state now 2026-03-23 04:29:11,148 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:29:11,148 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/8810c406-dbb4-49e2-9adb-065042ff5d8b 2026-03-23 04:29:11,155 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/4ac51bf2-0efe-4e37-8b6d-38010782e1d3 2026-03-23 04:29:11,166 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/8810c406-dbb4-49e2-9adb-065042ff5d8b 2026-03-23 04:29:11,175 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/4ac51bf2-0efe-4e37-8b6d-38010782e1d3 2026-03-23 04:29:11,186 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/9e1a4576-151f-4d58-86a4-78c97d909df7 2026-03-23 04:29:11,196 - DEBUG - Verifying in api server 10.0.0.23 2026-03-23 04:29:11,196 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/8810c406-dbb4-49e2-9adb-065042ff5d8b 2026-03-23 04:29:11,204 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/4ac51bf2-0efe-4e37-8b6d-38010782e1d3 2026-03-23 04:29:11,215 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/9e1a4576-151f-4d58-86a4-78c97d909df7 2026-03-23 04:29:11,224 - INFO - VM ctest-vn1_vm1-42110935 verfication in all API Servers passed 2026-03-23 04:29:11,247 - DEBUG - VM ctest-vn1_vm1-42110935 Tap interface: {'index': '3', 'name': 'tap4ac51bf2-0e', 'uuid': '4ac51bf2-0efe-4e37-8b6d-38010782e1d3', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926:ctest-vn1-82380926', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926', 'vm_uuid': '8810c406-dbb4-49e2-9adb-065042ff5d8b', 'vm_name': 'ctest-vn1_vm1-42110935', 'ip_addr': '192.168.1.3', 'mac_addr': '02:4a:c5:1b:f2:0e', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '161', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-88137413:4ac51bf2-0efe-4e37-8b6d-38010782e1d3', 'sg_uuid_list': ['42966352-7ed2-4211-baec-acac6f7d89b2'], 'static_route_list': None, 'vm_project_uuid': 'c818fea2-bb8c-4dc4-8c5c-a693e3a97287', 'admin_state': 'Enabled', 'flow_key_idx': '30', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-03-23 04:29:11,247 - DEBUG - Agent 10.0.0.65 vrf name: default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926:ctest-vn1-82380926 2026-03-23 04:29:11,254 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926:ctest-vn1-82380926', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926', 'table_label': '-1', 'vxlan_id': '15', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.201:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2026-03-23 04:29:11,337 - DEBUG - VM ctest-vn1_vm1-42110935 : Tap interface tap4ac51bf2-0e is set to Active 2026-03-23 04:29:11,338 - DEBUG - Tap interface tap4ac51bf2-0e detail : {'index': '3', 'name': 'tap4ac51bf2-0e', 'uuid': '4ac51bf2-0efe-4e37-8b6d-38010782e1d3', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926:ctest-vn1-82380926', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926', 'vm_uuid': '8810c406-dbb4-49e2-9adb-065042ff5d8b', 'vm_name': 'ctest-vn1_vm1-42110935', 'ip_addr': '192.168.1.3', 'mac_addr': '02:4a:c5:1b:f2:0e', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '161', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-88137413:4ac51bf2-0efe-4e37-8b6d-38010782e1d3', 'sg_uuid_list': ['42966352-7ed2-4211-baec-acac6f7d89b2'], 'static_route_list': None, 'vm_project_uuid': 'c818fea2-bb8c-4dc4-8c5c-a693e3a97287', 'admin_state': 'Enabled', 'flow_key_idx': '30', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-03-23 04:29:11,452 - DEBUG - Starting Layer 2 verification in Agent 2026-03-23 04:29:11,466 - DEBUG - Layer 2 path is seen for VM MAC 02:4a:c5:1b:f2:0e in agent 10.0.0.65 2026-03-23 04:29:11,466 - DEBUG - Active layer 2 route in agent is present for VMI tap4ac51bf2-0e 2026-03-23 04:29:11,466 - DEBUG - L2 label(29) matches bw route table and interface table 2026-03-23 04:29:11,473 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 is found in Agent of node 10.0.0.249 2026-03-23 04:29:11,607 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 is consistent in agent 10.0.0.249 2026-03-23 04:29:11,607 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.249 2026-03-23 04:29:11,607 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 verification for VM ctest-vn1_vm1-42110935 in Agent 10.0.0.249 passed 2026-03-23 04:29:11,725 - DEBUG - Starting all layer 2 verification in agent 10.0.0.249 2026-03-23 04:29:11,738 - DEBUG - Route for VM MAC 02:4a:c5:1b:f2:0e is consistent in agent 10.0.0.249 2026-03-23 04:29:11,745 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 is found in Agent of node 10.0.0.65 2026-03-23 04:29:11,885 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 is consistent in agent 10.0.0.65 2026-03-23 04:29:11,885 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.65 2026-03-23 04:29:11,885 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 verification for VM ctest-vn1_vm1-42110935 in Agent 10.0.0.65 passed 2026-03-23 04:29:12,005 - DEBUG - Starting all layer 2 verification in agent 10.0.0.65 2026-03-23 04:29:12,020 - DEBUG - Route for VM MAC 02:4a:c5:1b:f2:0e is consistent in agent 10.0.0.65 2026-03-23 04:29:12,021 - INFO - VM ctest-vn1_vm1-42110935 verifications in Compute nodes passed 2026-03-23 04:29:12,289 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.249 2026-03-23 04:29:12,294 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.201, 24; Got 10.20.0.201,24 2026-03-23 04:29:12,294 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.249 2026-03-23 04:29:12,414 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.65 2026-03-23 04:29:12,420 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2026-03-23 04:29:12,420 - INFO - Validated routes of VM ctest-vn1_vm1-42110935 in all vrouters 2026-03-23 04:29:12,916 - DEBUG - Starting all layer2 verification in 10.0.0.50 Control Node 2026-03-23 04:29:13,047 - DEBUG - Layer2 route found for VM MAC 02:4a:c5:1b:f2:0e in Control-node 10.0.0.50 2026-03-23 04:29:13,047 - DEBUG - L2 Label for VM ctest-vn1_vm1-42110935 same between Control-node 10.0.0.50 and Agent, Expected: 29, Seen: 29 2026-03-23 04:29:13,047 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2026-03-23 04:29:13,178 - DEBUG - Layer2 route found for VM MAC 02:4a:c5:1b:f2:0e in Control-node 10.0.0.27 2026-03-23 04:29:13,179 - DEBUG - L2 Label for VM ctest-vn1_vm1-42110935 same between Control-node 10.0.0.27 and Agent, Expected: 29, Seen: 29 2026-03-23 04:29:13,179 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2026-03-23 04:29:13,305 - DEBUG - Layer2 route found for VM MAC 02:4a:c5:1b:f2:0e in Control-node 10.0.0.23 2026-03-23 04:29:13,305 - DEBUG - L2 Label for VM ctest-vn1_vm1-42110935 same between Control-node 10.0.0.23 and Agent, Expected: 29, Seen: 29 2026-03-23 04:29:13,305 - INFO - Verification in Control-nodes for VM ctest-vn1_vm1-42110935 passed 2026-03-23 04:29:13,305 - DEBUG - Verifying the vm in opserver 2026-03-23 04:29:13,305 - DEBUG - Verifying in collector 10.0.0.23 ... 2026-03-23 04:29:13,305 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/8810c406-dbb4-49e2-9adb-065042ff5d8b?flat 2026-03-23 04:29:13,324 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-88137413:4ac51bf2-0efe-4e37-8b6d-38010782e1d3?flat 2026-03-23 04:29:13,346 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-88137413:4ac51bf2-0efe-4e37-8b6d-38010782e1d3?flat 2026-03-23 04:29:13,429 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/8810c406-dbb4-49e2-9adb-065042ff5d8b?flat 2026-03-23 04:29:13,512 - DEBUG - Verifying vm in vn uve 2026-03-23 04:29:13,512 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-88137413:4ac51bf2-0efe-4e37-8b6d-38010782e1d3?flat 2026-03-23 04:29:13,612 - DEBUG - VM uve shows interface as {'fixed_ip4_list': ['192.168.1.3'], 'gateway': '192.168.1.1', 'ip6_address': '::', 'vn_uuid': '85170818-fdb6-4ffe-97b2-b69ec42fae4e', 'vhostuser_mode': 0, 'vm_name': 'ctest-vn1_vm1-42110935', 'rx_vlan': 65535, 'ip6_active': False, 'is_health_check_active': True, 'mac_address': '02:4a:c5:1b:f2:0e', 'port_mirror_enabled': False, 'label': 24, 'admin_state': True, 'active': True, 'vm_uuid': '8810c406-dbb4-49e2-9adb-065042ff5d8b', 'ip4_active': True, 'ip_address': '192.168.1.3', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926', 'tx_vlan': 65535, '__T': 1774239950923059, 'uuid': '4ac51bf2-0efe-4e37-8b6d-38010782e1d3', 'l2_active': True} 2026-03-23 04:29:13,612 - DEBUG - VM uve shows ip address as ['192.168.1.3', '::'] 2026-03-23 04:29:13,613 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 2026-03-23 04:29:13,613 - INFO - Verifying through opserver in 10.0.0.27 2026-03-23 04:29:13,613 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926?flat 2026-03-23 04:29:13,668 - DEBUG - expected vm list 8810c406-dbb4-49e2-9adb-065042ff5d8b 2026-03-23 04:29:13,669 - DEBUG - Extracted vm list ['8810c406-dbb4-49e2-9adb-065042ff5d8b'] 2026-03-23 04:29:13,669 - INFO - VM 8810c406-dbb4-49e2-9adb-065042ff5d8b is present in default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 2026-03-23 04:29:13,669 - DEBUG - Verifying vm in vrouter uve 2026-03-23 04:29:13,669 - DEBUG - Getting info from collector 10.0.0.23.. 2026-03-23 04:29:13,670 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/8810c406-dbb4-49e2-9adb-065042ff5d8b?flat 2026-03-23 04:29:13,768 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-5514-2'] 2026-03-23 04:29:13,768 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-2:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:29:13,820 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-2?flat 2026-03-23 04:29:13,858 - DEBUG - VM 8810c406-dbb4-49e2-9adb-065042ff5d8b is present in vrouter an-jenkins-deploy-platform-ansible-os-5514-2 uve 2026-03-23 04:29:13,858 - DEBUG - Validated that VM ctest-vn1_vm1-42110935 is in Vrouter an-jenkins-deploy-platform-ansible-os-5514-2 UVE 2026-03-23 04:29:13,858 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2026-03-23 04:29:13,868 - DEBUG - Expected tap interface of VM uuid 8810c406-dbb4-49e2-9adb-065042ff5d8b is default-domain:ctest-TestBasicVMVN0-88137413:4ac51bf2-0efe-4e37-8b6d-38010782e1d3 2026-03-23 04:29:13,868 - DEBUG - Expected VN of VM uuid 8810c406-dbb4-49e2-9adb-065042ff5d8b is default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 2026-03-23 04:29:13,868 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-2:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:29:13,917 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-2?flat 2026-03-23 04:29:14,008 - DEBUG - VM 8810c406-dbb4-49e2-9adb-065042ff5d8b is present in vrouter an-jenkins-deploy-platform-ansible-os-5514-2 uve 2026-03-23 04:29:14,008 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-88137413:4ac51bf2-0efe-4e37-8b6d-38010782e1d3 of vm 8810c406-dbb4-49e2-9adb-065042ff5d8b is present in vrouter an-jenkins-deploy-platform-ansible-os-5514-2 uve 2026-03-23 04:29:14,008 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 of vm 8810c406-dbb4-49e2-9adb-065042ff5d8b is present in vrouter an-jenkins-deploy-platform-ansible-os-5514-2 uve 2026-03-23 04:29:14,008 - DEBUG - Verifying the 8810c406-dbb4-49e2-9adb-065042ff5d8b virtual network link through opserver 10.0.0.27 2026-03-23 04:29:14,008 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-03-23 04:29:14,024 - DEBUG - VM link and name as {'name': '8810c406-dbb4-49e2-9adb-065042ff5d8b', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/8810c406-dbb4-49e2-9adb-065042ff5d8b?flat'} 2026-03-23 04:29:14,024 - DEBUG - Verifying the 8810c406-dbb4-49e2-9adb-065042ff5d8b virtual network link through opserver 10.0.0.50 2026-03-23 04:29:14,024 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2026-03-23 04:29:14,107 - DEBUG - VM link and name as {'name': '8810c406-dbb4-49e2-9adb-065042ff5d8b', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-machine/8810c406-dbb4-49e2-9adb-065042ff5d8b?flat'} 2026-03-23 04:29:14,107 - DEBUG - Verifying the 8810c406-dbb4-49e2-9adb-065042ff5d8b virtual network link through opserver 10.0.0.23 2026-03-23 04:29:14,107 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-03-23 04:29:14,121 - DEBUG - VM link and name as {'name': '8810c406-dbb4-49e2-9adb-065042ff5d8b', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/8810c406-dbb4-49e2-9adb-065042ff5d8b?flat'} 2026-03-23 04:29:14,121 - INFO - VM ctest-vn1_vm1-42110935 validations in Opserver passed 2026-03-23 04:29:14,255 - DEBUG - VM is in ACTIVE state now 2026-03-23 04:29:14,255 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:29:14,255 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/66b61710-fb4a-475e-891a-d151c66dbd84 2026-03-23 04:29:14,264 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/d546366b-40c3-431a-b492-e54db2b3eadc 2026-03-23 04:29:14,276 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/66b61710-fb4a-475e-891a-d151c66dbd84 2026-03-23 04:29:14,284 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/d546366b-40c3-431a-b492-e54db2b3eadc 2026-03-23 04:29:14,294 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/744076d8-40d3-40e7-9322-839b712a2abf 2026-03-23 04:29:14,305 - DEBUG - Verifying in api server 10.0.0.23 2026-03-23 04:29:14,305 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/66b61710-fb4a-475e-891a-d151c66dbd84 2026-03-23 04:29:14,314 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/d546366b-40c3-431a-b492-e54db2b3eadc 2026-03-23 04:29:14,325 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/744076d8-40d3-40e7-9322-839b712a2abf 2026-03-23 04:29:14,337 - INFO - VM ctest-vn1_vm2-75830156 verfication in all API Servers passed 2026-03-23 04:29:14,358 - DEBUG - VM ctest-vn1_vm2-75830156 Tap interface: {'index': '3', 'name': 'tapd546366b-40', 'uuid': 'd546366b-40c3-431a-b492-e54db2b3eadc', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926:ctest-vn1-82380926', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926', 'vm_uuid': '66b61710-fb4a-475e-891a-d151c66dbd84', 'vm_name': 'ctest-vn1_vm2-75830156', 'ip_addr': '192.168.1.4', 'mac_addr': '02:d5:46:36:6b:40', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '127', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-88137413:d546366b-40c3-431a-b492-e54db2b3eadc', 'sg_uuid_list': ['42966352-7ed2-4211-baec-acac6f7d89b2'], 'static_route_list': None, 'vm_project_uuid': 'c818fea2-bb8c-4dc4-8c5c-a693e3a97287', 'admin_state': 'Enabled', 'flow_key_idx': '29', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-03-23 04:29:14,358 - DEBUG - Agent 10.0.0.249 vrf name: default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926:ctest-vn1-82380926 2026-03-23 04:29:14,364 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926:ctest-vn1-82380926', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926', 'table_label': '-1', 'vxlan_id': '15', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.192:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2026-03-23 04:29:14,444 - DEBUG - VM ctest-vn1_vm2-75830156 : Tap interface tapd546366b-40 is set to Active 2026-03-23 04:29:14,445 - DEBUG - Tap interface tapd546366b-40 detail : {'index': '3', 'name': 'tapd546366b-40', 'uuid': 'd546366b-40c3-431a-b492-e54db2b3eadc', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926:ctest-vn1-82380926', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926', 'vm_uuid': '66b61710-fb4a-475e-891a-d151c66dbd84', 'vm_name': 'ctest-vn1_vm2-75830156', 'ip_addr': '192.168.1.4', 'mac_addr': '02:d5:46:36:6b:40', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '127', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-88137413:d546366b-40c3-431a-b492-e54db2b3eadc', 'sg_uuid_list': ['42966352-7ed2-4211-baec-acac6f7d89b2'], 'static_route_list': None, 'vm_project_uuid': 'c818fea2-bb8c-4dc4-8c5c-a693e3a97287', 'admin_state': 'Enabled', 'flow_key_idx': '29', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-03-23 04:29:14,560 - DEBUG - Starting Layer 2 verification in Agent 2026-03-23 04:29:14,574 - DEBUG - Layer 2 path is seen for VM MAC 02:d5:46:36:6b:40 in agent 10.0.0.249 2026-03-23 04:29:14,574 - DEBUG - Active layer 2 route in agent is present for VMI tapd546366b-40 2026-03-23 04:29:14,574 - DEBUG - L2 label(29) matches bw route table and interface table 2026-03-23 04:29:14,581 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 is found in Agent of node 10.0.0.249 2026-03-23 04:29:14,719 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 is consistent in agent 10.0.0.249 2026-03-23 04:29:14,720 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.249 2026-03-23 04:29:14,720 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 verification for VM ctest-vn1_vm2-75830156 in Agent 10.0.0.249 passed 2026-03-23 04:29:14,836 - DEBUG - Starting all layer 2 verification in agent 10.0.0.249 2026-03-23 04:29:14,851 - DEBUG - Route for VM MAC 02:d5:46:36:6b:40 is consistent in agent 10.0.0.249 2026-03-23 04:29:14,859 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 is found in Agent of node 10.0.0.65 2026-03-23 04:29:14,995 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 is consistent in agent 10.0.0.65 2026-03-23 04:29:14,995 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.65 2026-03-23 04:29:14,995 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 verification for VM ctest-vn1_vm2-75830156 in Agent 10.0.0.65 passed 2026-03-23 04:29:15,110 - DEBUG - Starting all layer 2 verification in agent 10.0.0.65 2026-03-23 04:29:15,129 - DEBUG - Route for VM MAC 02:d5:46:36:6b:40 is consistent in agent 10.0.0.65 2026-03-23 04:29:15,129 - INFO - VM ctest-vn1_vm2-75830156 verifications in Compute nodes passed 2026-03-23 04:29:15,403 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.249 2026-03-23 04:29:15,410 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2026-03-23 04:29:15,539 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.65 2026-03-23 04:29:15,546 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.192, 25; Got 10.20.0.192,25 2026-03-23 04:29:15,546 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.65 2026-03-23 04:29:15,546 - INFO - Validated routes of VM ctest-vn1_vm2-75830156 in all vrouters 2026-03-23 04:29:16,069 - DEBUG - Starting all layer2 verification in 10.0.0.50 Control Node 2026-03-23 04:29:16,211 - DEBUG - Layer2 route found for VM MAC 02:d5:46:36:6b:40 in Control-node 10.0.0.50 2026-03-23 04:29:16,211 - DEBUG - L2 Label for VM ctest-vn1_vm2-75830156 same between Control-node 10.0.0.50 and Agent, Expected: 29, Seen: 29 2026-03-23 04:29:16,211 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2026-03-23 04:29:16,335 - DEBUG - Layer2 route found for VM MAC 02:d5:46:36:6b:40 in Control-node 10.0.0.27 2026-03-23 04:29:16,335 - DEBUG - L2 Label for VM ctest-vn1_vm2-75830156 same between Control-node 10.0.0.27 and Agent, Expected: 29, Seen: 29 2026-03-23 04:29:16,335 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2026-03-23 04:29:16,469 - DEBUG - Layer2 route found for VM MAC 02:d5:46:36:6b:40 in Control-node 10.0.0.23 2026-03-23 04:29:16,469 - DEBUG - L2 Label for VM ctest-vn1_vm2-75830156 same between Control-node 10.0.0.23 and Agent, Expected: 29, Seen: 29 2026-03-23 04:29:16,469 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-75830156 passed 2026-03-23 04:29:16,470 - DEBUG - Verifying the vm in opserver 2026-03-23 04:29:16,470 - DEBUG - Verifying in collector 10.0.0.23 ... 2026-03-23 04:29:16,470 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/66b61710-fb4a-475e-891a-d151c66dbd84?flat 2026-03-23 04:29:16,487 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-88137413:d546366b-40c3-431a-b492-e54db2b3eadc?flat 2026-03-23 04:29:16,513 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-88137413:d546366b-40c3-431a-b492-e54db2b3eadc?flat 2026-03-23 04:29:16,534 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/66b61710-fb4a-475e-891a-d151c66dbd84?flat 2026-03-23 04:29:16,551 - DEBUG - Verifying vm in vn uve 2026-03-23 04:29:16,551 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-88137413:d546366b-40c3-431a-b492-e54db2b3eadc?flat 2026-03-23 04:29:16,573 - DEBUG - VM uve shows interface as {'fixed_ip4_list': ['192.168.1.4'], 'gateway': '192.168.1.1', 'ip6_address': '::', 'vn_uuid': '85170818-fdb6-4ffe-97b2-b69ec42fae4e', 'vhostuser_mode': 0, 'vm_name': 'ctest-vn1_vm2-75830156', 'rx_vlan': 65535, 'ip6_active': False, 'is_health_check_active': True, 'mac_address': '02:d5:46:36:6b:40', 'port_mirror_enabled': False, 'label': 25, 'admin_state': True, 'active': True, 'vm_uuid': '66b61710-fb4a-475e-891a-d151c66dbd84', 'ip4_active': True, 'ip_address': '192.168.1.4', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926', 'tx_vlan': 65535, '__T': 1774239942512638, 'uuid': 'd546366b-40c3-431a-b492-e54db2b3eadc', 'l2_active': True} 2026-03-23 04:29:16,573 - DEBUG - VM uve shows ip address as ['192.168.1.4', '::'] 2026-03-23 04:29:16,573 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 2026-03-23 04:29:16,573 - INFO - Verifying through opserver in 10.0.0.27 2026-03-23 04:29:16,574 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926?flat 2026-03-23 04:29:16,604 - DEBUG - expected vm list 66b61710-fb4a-475e-891a-d151c66dbd84 2026-03-23 04:29:16,604 - DEBUG - Extracted vm list ['66b61710-fb4a-475e-891a-d151c66dbd84'] 2026-03-23 04:29:16,604 - INFO - VM 66b61710-fb4a-475e-891a-d151c66dbd84 is present in default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 2026-03-23 04:29:16,604 - DEBUG - Verifying vm in vrouter uve 2026-03-23 04:29:16,604 - DEBUG - Getting info from collector 10.0.0.23.. 2026-03-23 04:29:16,604 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/66b61710-fb4a-475e-891a-d151c66dbd84?flat 2026-03-23 04:29:16,621 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-5514-1'] 2026-03-23 04:29:16,621 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-1:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:29:16,670 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-1?flat 2026-03-23 04:29:16,701 - DEBUG - VM 66b61710-fb4a-475e-891a-d151c66dbd84 is present in vrouter an-jenkins-deploy-platform-ansible-os-5514-1 uve 2026-03-23 04:29:16,701 - DEBUG - Validated that VM ctest-vn1_vm2-75830156 is in Vrouter an-jenkins-deploy-platform-ansible-os-5514-1 UVE 2026-03-23 04:29:16,701 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2026-03-23 04:29:16,710 - DEBUG - Expected tap interface of VM uuid 66b61710-fb4a-475e-891a-d151c66dbd84 is default-domain:ctest-TestBasicVMVN0-88137413:d546366b-40c3-431a-b492-e54db2b3eadc 2026-03-23 04:29:16,710 - DEBUG - Expected VN of VM uuid 66b61710-fb4a-475e-891a-d151c66dbd84 is default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 2026-03-23 04:29:16,710 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-1:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:29:16,762 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-1?flat 2026-03-23 04:29:16,798 - DEBUG - VM 66b61710-fb4a-475e-891a-d151c66dbd84 is present in vrouter an-jenkins-deploy-platform-ansible-os-5514-1 uve 2026-03-23 04:29:16,798 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-88137413:d546366b-40c3-431a-b492-e54db2b3eadc of vm 66b61710-fb4a-475e-891a-d151c66dbd84 is present in vrouter an-jenkins-deploy-platform-ansible-os-5514-1 uve 2026-03-23 04:29:16,798 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 of vm 66b61710-fb4a-475e-891a-d151c66dbd84 is present in vrouter an-jenkins-deploy-platform-ansible-os-5514-1 uve 2026-03-23 04:29:16,798 - DEBUG - Verifying the 66b61710-fb4a-475e-891a-d151c66dbd84 virtual network link through opserver 10.0.0.27 2026-03-23 04:29:16,798 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-03-23 04:29:16,813 - DEBUG - VM link and name as {'name': '66b61710-fb4a-475e-891a-d151c66dbd84', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/66b61710-fb4a-475e-891a-d151c66dbd84?flat'} 2026-03-23 04:29:16,813 - DEBUG - Verifying the 66b61710-fb4a-475e-891a-d151c66dbd84 virtual network link through opserver 10.0.0.50 2026-03-23 04:29:16,813 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2026-03-23 04:29:16,826 - DEBUG - VM link and name as {'name': '66b61710-fb4a-475e-891a-d151c66dbd84', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-machine/66b61710-fb4a-475e-891a-d151c66dbd84?flat'} 2026-03-23 04:29:16,826 - DEBUG - Verifying the 66b61710-fb4a-475e-891a-d151c66dbd84 virtual network link through opserver 10.0.0.23 2026-03-23 04:29:16,826 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-03-23 04:29:16,840 - DEBUG - VM link and name as {'name': '66b61710-fb4a-475e-891a-d151c66dbd84', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/66b61710-fb4a-475e-891a-d151c66dbd84?flat'} 2026-03-23 04:29:16,840 - INFO - VM ctest-vn1_vm2-75830156 validations in Opserver passed 2026-03-23 04:29:16,840 - INFO - Checking the ping between the VM again 2026-03-23 04:29:16,840 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:29:16,840 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.65, gateway password: c0ntrail123 2026-03-23 04:29:16,840 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4 2026-03-23 04:29:19,382 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data. 64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=7.10 ms 64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.36 ms 64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.35 ms --- 192.168.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.356/3.275/7.108/2.710 ms 2026-03-23 04:29:19,382 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-42110935 passed 2026-03-23 04:29:19,382 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:29:19,382 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.249, gateway password: c0ntrail123 2026-03-23 04:29:19,382 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3 2026-03-23 04:29:21,978 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. 64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=4.30 ms 64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.56 ms 64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.67 ms --- 192.168.1.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.568/2.516/4.304/1.265 ms 2026-03-23 04:29:21,979 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-75830156 passed 2026-03-23 04:29:21,979 - INFO - Deleting VM ctest-vn1_vm2-75830156 2026-03-23 04:29:22,074 - DEBUG - Verifying in api server 10.0.0.23 2026-03-23 04:29:22,074 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/66b61710-fb4a-475e-891a-d151c66dbd84 2026-03-23 04:29:22,085 - DEBUG - VM ID 66b61710-fb4a-475e-891a-d151c66dbd84 of VM ctest-vn1_vm2-75830156 is still found in API Server 2026-03-23 04:29:24,085 - DEBUG - Verifying in api server 10.0.0.23 2026-03-23 04:29:24,086 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/66b61710-fb4a-475e-891a-d151c66dbd84 2026-03-23 04:29:24,093 - DEBUG - Response Code: 404 2026-03-23 04:29:24,093 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/66b61710-fb4a-475e-891a-d151c66dbd84 2026-03-23 04:29:24,100 - DEBUG - Response Code: 404 2026-03-23 04:29:24,100 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/66b61710-fb4a-475e-891a-d151c66dbd84 2026-03-23 04:29:24,106 - DEBUG - Response Code: 404 2026-03-23 04:29:24,106 - INFO - VM ctest-vn1_vm2-75830156 is fully removed in API-Server 2026-03-23 04:29:24,263 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2026-03-23 04:29:24,308 - DEBUG - Validated that vrouter 10.0.0.249 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 2026-03-23 04:29:24,433 - DEBUG - VM route 192.168.1.4 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 of compute 10.0.0.65 2026-03-23 04:29:24,438 - DEBUG - Validated that vrouter 10.0.0.65 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 2026-03-23 04:29:24,438 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 2026-03-23 04:29:24,439 - INFO - VM ctest-vn1_vm2-75830156 is removed in Compute, and routes are removed in all compute nodes 2026-03-23 04:29:24,690 - INFO - Routes for VM ctest-vn1_vm2-75830156 is removed in all control-nodes 2026-03-23 04:29:24,691 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-1:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:29:24,762 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-1?flat 2026-03-23 04:29:24,794 - DEBUG - VM 66b61710-fb4a-475e-891a-d151c66dbd84 is still present in vrouter an-jenkins-deploy-platform-ansible-os-5514-1 uve 2026-03-23 04:29:24,795 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-03-23 04:29:24,795 - DEBUG - interface for vm 66b61710-fb4a-475e-891a-d151c66dbd84 deleted from vrouter uve ... 2026-03-23 04:29:24,795 - DEBUG - Deleted interface not in error interface list ... 2026-03-23 04:29:27,796 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-1:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:29:27,851 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-1?flat 2026-03-23 04:29:27,884 - DEBUG - VM 66b61710-fb4a-475e-891a-d151c66dbd84 is still present in vrouter an-jenkins-deploy-platform-ansible-os-5514-1 uve 2026-03-23 04:29:27,884 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-03-23 04:29:27,885 - DEBUG - interface for vm 66b61710-fb4a-475e-891a-d151c66dbd84 deleted from vrouter uve ... 2026-03-23 04:29:27,885 - DEBUG - Deleted interface not in error interface list ... 2026-03-23 04:29:30,885 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-1:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:29:30,945 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-1?flat 2026-03-23 04:29:30,978 - DEBUG - VM 66b61710-fb4a-475e-891a-d151c66dbd84 is still present in vrouter an-jenkins-deploy-platform-ansible-os-5514-1 uve 2026-03-23 04:29:30,978 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-03-23 04:29:30,978 - DEBUG - interface for vm 66b61710-fb4a-475e-891a-d151c66dbd84 deleted from vrouter uve ... 2026-03-23 04:29:30,978 - DEBUG - Deleted interface not in error interface list ... 2026-03-23 04:29:33,978 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-1:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:29:34,042 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-1?flat 2026-03-23 04:29:34,078 - DEBUG - VM 66b61710-fb4a-475e-891a-d151c66dbd84 is still present in vrouter an-jenkins-deploy-platform-ansible-os-5514-1 uve 2026-03-23 04:29:34,078 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-03-23 04:29:34,078 - DEBUG - interface for vm 66b61710-fb4a-475e-891a-d151c66dbd84 deleted from vrouter uve ... 2026-03-23 04:29:34,078 - DEBUG - Deleted interface not in error interface list ... 2026-03-23 04:29:37,079 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-1:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:29:37,140 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-1?flat 2026-03-23 04:29:37,176 - DEBUG - VM 66b61710-fb4a-475e-891a-d151c66dbd84 is still present in vrouter an-jenkins-deploy-platform-ansible-os-5514-1 uve 2026-03-23 04:29:37,176 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-03-23 04:29:37,176 - DEBUG - interface for vm 66b61710-fb4a-475e-891a-d151c66dbd84 deleted from vrouter uve ... 2026-03-23 04:29:37,176 - DEBUG - Deleted interface not in error interface list ... 2026-03-23 04:29:40,177 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-1:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:29:40,240 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-1?flat 2026-03-23 04:29:40,271 - DEBUG - VM 66b61710-fb4a-475e-891a-d151c66dbd84 is still present in vrouter an-jenkins-deploy-platform-ansible-os-5514-1 uve 2026-03-23 04:29:40,271 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-03-23 04:29:40,271 - DEBUG - interface for vm 66b61710-fb4a-475e-891a-d151c66dbd84 deleted from vrouter uve ... 2026-03-23 04:29:40,271 - DEBUG - Deleted interface not in error interface list ... 2026-03-23 04:29:43,272 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-1:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:29:43,337 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-1?flat 2026-03-23 04:29:43,372 - DEBUG - VM 66b61710-fb4a-475e-891a-d151c66dbd84 is still present in vrouter an-jenkins-deploy-platform-ansible-os-5514-1 uve 2026-03-23 04:29:43,372 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-03-23 04:29:43,372 - DEBUG - interface for vm 66b61710-fb4a-475e-891a-d151c66dbd84 deleted from vrouter uve ... 2026-03-23 04:29:43,372 - DEBUG - Deleted interface not in error interface list ... 2026-03-23 04:29:46,373 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-1:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:29:46,427 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-1?flat 2026-03-23 04:29:46,461 - DEBUG - VM 66b61710-fb4a-475e-891a-d151c66dbd84 is still present in vrouter an-jenkins-deploy-platform-ansible-os-5514-1 uve 2026-03-23 04:29:46,461 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-03-23 04:29:46,461 - DEBUG - interface for vm 66b61710-fb4a-475e-891a-d151c66dbd84 deleted from vrouter uve ... 2026-03-23 04:29:46,461 - DEBUG - Deleted interface not in error interface list ... 2026-03-23 04:29:49,462 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-1:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:29:49,517 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-1?flat 2026-03-23 04:29:49,557 - DEBUG - VM 66b61710-fb4a-475e-891a-d151c66dbd84 is not present in vrouter an-jenkins-deploy-platform-ansible-os-5514-1 uve 2026-03-23 04:29:49,557 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-03-23 04:29:49,557 - DEBUG - interface for vm 66b61710-fb4a-475e-891a-d151c66dbd84 deleted from vrouter uve ... 2026-03-23 04:29:49,557 - DEBUG - Deleted interface not in error interface list ... 2026-03-23 04:29:49,557 - DEBUG - Validated that VM 66b61710-fb4a-475e-891a-d151c66dbd84 is removed in Vrouter UVE 2026-03-23 04:29:49,557 - DEBUG - Verifying through opserver in 10.0.0.27 2026-03-23 04:29:49,557 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926?flat 2026-03-23 04:29:49,584 - DEBUG - 66b61710-fb4a-475e-891a-d151c66dbd84 BM not in default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 uve 2026-03-23 04:29:49,584 - DEBUG - Validated that VM 66b61710-fb4a-475e-891a-d151c66dbd84 is not present in the VN UVE 2026-03-23 04:29:49,584 - DEBUG - Verifying the 66b61710-fb4a-475e-891a-d151c66dbd84 virtual network link through opserver 10.0.0.27 2026-03-23 04:29:49,584 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-03-23 04:29:49,598 - INFO - 66b61710-fb4a-475e-891a-d151c66dbd84 vm uve deleted from opserver 2026-03-23 04:29:49,598 - DEBUG - Verifying the 66b61710-fb4a-475e-891a-d151c66dbd84 virtual network link through opserver 10.0.0.50 2026-03-23 04:29:49,599 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2026-03-23 04:29:49,613 - INFO - 66b61710-fb4a-475e-891a-d151c66dbd84 vm uve deleted from opserver 2026-03-23 04:29:49,613 - DEBUG - Verifying the 66b61710-fb4a-475e-891a-d151c66dbd84 virtual network link through opserver 10.0.0.23 2026-03-23 04:29:49,613 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-03-23 04:29:49,627 - INFO - 66b61710-fb4a-475e-891a-d151c66dbd84 vm uve deleted from opserver 2026-03-23 04:29:49,627 - INFO - Validated that VM 66b61710-fb4a-475e-891a-d151c66dbd84 is removed from Opserver 2026-03-23 04:29:49,627 - INFO - Deleting VM ctest-vn1_vm1-42110935 2026-03-23 04:29:49,706 - DEBUG - Verifying in api server 10.0.0.23 2026-03-23 04:29:49,706 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/8810c406-dbb4-49e2-9adb-065042ff5d8b 2026-03-23 04:29:49,716 - DEBUG - VM ID 8810c406-dbb4-49e2-9adb-065042ff5d8b of VM ctest-vn1_vm1-42110935 is still found in API Server 2026-03-23 04:29:51,717 - DEBUG - Verifying in api server 10.0.0.23 2026-03-23 04:29:51,717 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/8810c406-dbb4-49e2-9adb-065042ff5d8b 2026-03-23 04:29:51,724 - DEBUG - Response Code: 404 2026-03-23 04:29:51,725 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/8810c406-dbb4-49e2-9adb-065042ff5d8b 2026-03-23 04:29:51,731 - DEBUG - Response Code: 404 2026-03-23 04:29:51,731 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/8810c406-dbb4-49e2-9adb-065042ff5d8b 2026-03-23 04:29:51,738 - DEBUG - Response Code: 404 2026-03-23 04:29:51,738 - INFO - VM ctest-vn1_vm1-42110935 is fully removed in API-Server 2026-03-23 04:29:51,909 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2026-03-23 04:29:51,959 - DEBUG - Validated that vrouter 10.0.0.249 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 2026-03-23 04:29:51,974 - DEBUG - Validated that vrouter 10.0.0.65 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 2026-03-23 04:29:51,974 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 2026-03-23 04:29:51,974 - INFO - VM ctest-vn1_vm1-42110935 is removed in Compute, and routes are removed in all compute nodes 2026-03-23 04:29:52,206 - INFO - Routes for VM ctest-vn1_vm1-42110935 is removed in all control-nodes 2026-03-23 04:29:52,206 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-2:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:29:52,264 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-2?flat 2026-03-23 04:29:52,296 - DEBUG - VM 8810c406-dbb4-49e2-9adb-065042ff5d8b is not present in vrouter an-jenkins-deploy-platform-ansible-os-5514-2 uve 2026-03-23 04:29:52,296 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-03-23 04:29:52,296 - DEBUG - interface for vm 8810c406-dbb4-49e2-9adb-065042ff5d8b deleted from vrouter uve ... 2026-03-23 04:29:52,296 - DEBUG - Deleted interface not in error interface list ... 2026-03-23 04:29:52,296 - DEBUG - Validated that VM 8810c406-dbb4-49e2-9adb-065042ff5d8b is removed in Vrouter UVE 2026-03-23 04:29:52,296 - DEBUG - Verifying through opserver in 10.0.0.27 2026-03-23 04:29:52,296 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926?flat 2026-03-23 04:29:52,319 - DEBUG - 8810c406-dbb4-49e2-9adb-065042ff5d8b BM not in default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-82380926 uve 2026-03-23 04:29:52,319 - DEBUG - Validated that VM 8810c406-dbb4-49e2-9adb-065042ff5d8b is not present in the VN UVE 2026-03-23 04:29:52,319 - DEBUG - Verifying the 8810c406-dbb4-49e2-9adb-065042ff5d8b virtual network link through opserver 10.0.0.27 2026-03-23 04:29:52,319 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-03-23 04:29:52,333 - INFO - 8810c406-dbb4-49e2-9adb-065042ff5d8b vm uve deleted from opserver 2026-03-23 04:29:52,333 - DEBUG - Verifying the 8810c406-dbb4-49e2-9adb-065042ff5d8b virtual network link through opserver 10.0.0.50 2026-03-23 04:29:52,333 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2026-03-23 04:29:52,346 - INFO - 8810c406-dbb4-49e2-9adb-065042ff5d8b vm uve deleted from opserver 2026-03-23 04:29:52,346 - DEBUG - Verifying the 8810c406-dbb4-49e2-9adb-065042ff5d8b virtual network link through opserver 10.0.0.23 2026-03-23 04:29:52,346 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-03-23 04:29:52,360 - INFO - 8810c406-dbb4-49e2-9adb-065042ff5d8b vm uve deleted from opserver 2026-03-23 04:29:52,360 - INFO - Validated that VM 8810c406-dbb4-49e2-9adb-065042ff5d8b is removed from Opserver 2026-03-23 04:29:52,360 - INFO - Deleting VN ctest-vn1-82380926 2026-03-23 04:29:52,511 - DEBUG - Response for deleting network () 2026-03-23 04:29:52,511 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/17973f9b-c6aa-4a8f-98c9-9e4edefec96f 2026-03-23 04:29:52,519 - DEBUG - Response Code: 404 2026-03-23 04:29:52,519 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-03-23 04:29:52,529 - DEBUG - Requesting: http://10.0.0.27:8082/domain/54adab80-1c58-45eb-8fc6-565ee3803789 2026-03-23 04:29:52,559 - DEBUG - Requesting: http://10.0.0.27:8082/project/c818fea2-bb8c-4dc4-8c5c-a693e3a97287 2026-03-23 04:29:52,674 - INFO - Validated that VN ctest-vn1-82380926 is not found in API Server 2026-03-23 04:29:52,690 - DEBUG - VN ctest-vn1-82380926 is not present in Agent 10.0.0.249 2026-03-23 04:29:52,704 - DEBUG - VN ctest-vn1-82380926 is not present in Agent 10.0.0.65 2026-03-23 04:29:52,704 - INFO - Validated that VN ctest-vn1-82380926 is not in any agent 2026-03-23 04:29:52,744 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-82380926 info 2026-03-23 04:29:54,034 - DEBUG - Skipping xmpp flap check
2026-03-23 04:29:54,034 - INFO - END TEST : test_control_node_switchover : PASSED[0:04:28]
2026-03-23 04:29:54,034 - INFO - -------------------------------------------------------------------------------- 2026-03-23 04:29:54,037 - INFO - ================================================================================ 2026-03-23 04:29:54,037 - INFO - STARTING TEST : test_process_restart_in_policy_between_vns 2026-03-23 04:29:54,037 - INFO - TEST DESCRIPTION : Test to validate that with policy having rule to check icmp fwding between VMs on different VNs , ping between VMs should pass with process restarts 1. Pick 2 VN's from resource pool which has one VM each 2. Create policy with icmp allow rule between those VN's and bind it networks 3. Ping from one VM to another VM 4. Restart process 'vrouter' and 'control' on setup 5. Ping again between VM's after process restart Pass criteria: Step 2,3,4 and 5 should pass 2026-03-23 04:29:55,329 - DEBUG - Skipping xmpp flap check 2026-03-23 04:29:55,329 - INFO - Initial checks done. Running the testcase now 2026-03-23 04:29:55,329 - INFO - 2026-03-23 04:29:55,337 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2026-03-23 04:29:55,651 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2026-03-23 04:29:56,227 - INFO - Created VN ctest-vn1-67631103, UUID :3bd95fd3-bf08-4424-9610-ec574d9aa238 2026-03-23 04:29:56,557 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-03-23 04:29:56,566 - DEBUG - Requesting: http://10.0.0.27:8082/domain/54adab80-1c58-45eb-8fc6-565ee3803789 2026-03-23 04:29:56,579 - DEBUG - Requesting: http://10.0.0.27:8082/project/c818fea2-bb8c-4dc4-8c5c-a693e3a97287 2026-03-23 04:29:56,697 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/3bd95fd3-bf08-4424-9610-ec574d9aa238 2026-03-23 04:29:56,714 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/3bd95fd3-bf08-4424-9610-ec574d9aa238 2026-03-23 04:29:56,725 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/38abe83a-48fc-48ae-ad52-170aca7450e8 2026-03-23 04:29:56,738 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/38abe83a-48fc-48ae-ad52-170aca7450e8 2026-03-23 04:29:56,747 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/a1b9913a-8fde-46bf-a64d-c334cb8c7e90 2026-03-23 04:29:56,756 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-03-23 04:29:56,756 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/3bd95fd3-bf08-4424-9610-ec574d9aa238 2026-03-23 04:29:56,767 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/38abe83a-48fc-48ae-ad52-170aca7450e8 2026-03-23 04:29:56,784 - INFO - Verified VN network id 15 for VN 3bd95fd3-bf08-4424-9610-ec574d9aa238 2026-03-23 04:29:56,784 - INFO - Verifications in API Server for VN ctest-vn1-67631103 passed 2026-03-23 04:29:56,784 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/3bd95fd3-bf08-4424-9610-ec574d9aa238 2026-03-23 04:29:56,796 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/38abe83a-48fc-48ae-ad52-170aca7450e8 2026-03-23 04:29:56,804 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/38abe83a-48fc-48ae-ad52-170aca7450e8 2026-03-23 04:29:56,813 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/a1b9913a-8fde-46bf-a64d-c334cb8c7e90 2026-03-23 04:29:56,826 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-88137413', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '4312583481231688740', 'uuid-lslong': '10813402565024850488'}, 'enable': 'true', 'created': '2026-03-23T04:29:56', 'last-modified': '2026-03-23T04:29:56', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.455717'} 2026-03-23 04:29:56,833 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-03-23 04:29:56,840 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-88137413', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '4312583481231688740', 'uuid-lslong': '10813402565024850488'}, 'enable': 'true', 'created': '2026-03-23T04:29:56', 'last-modified': '2026-03-23T04:29:56', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.467731'} 2026-03-23 04:29:56,846 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-03-23 04:29:56,852 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-88137413', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '4312583481231688740', 'uuid-lslong': '10813402565024850488'}, 'enable': 'true', 'created': '2026-03-23T04:29:56', 'last-modified': '2026-03-23T04:29:56', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.478611'} 2026-03-23 04:29:56,858 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-03-23 04:29:56,858 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-67631103 passed 2026-03-23 04:29:56,859 - DEBUG - ====Verifying policy data for ctest-vn1-67631103 in API_Server ====== 2026-03-23 04:29:56,859 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-03-23 04:29:56,866 - DEBUG - Requesting: http://10.0.0.27:8082/domain/54adab80-1c58-45eb-8fc6-565ee3803789 2026-03-23 04:29:56,880 - DEBUG - Requesting: http://10.0.0.27:8082/project/c818fea2-bb8c-4dc4-8c5c-a693e3a97287 2026-03-23 04:29:57,031 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/3bd95fd3-bf08-4424-9610-ec574d9aa238 2026-03-23 04:29:57,047 - DEBUG - =>VN ctest-vn1-67631103 has no policy to be verified 2026-03-23 04:29:57,047 - DEBUG - Verifying the vn in opserver 2026-03-23 04:29:57,047 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 virtual network link through opserver 10.0.0.27 2026-03-23 04:29:57,047 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2026-03-23 04:29:57,061 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103?flat'} 2026-03-23 04:29:57,061 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 virtual network link through opserver 10.0.0.50 2026-03-23 04:29:57,062 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2026-03-23 04:29:57,077 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103?flat'} 2026-03-23 04:29:57,077 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 virtual network link through opserver 10.0.0.23 2026-03-23 04:29:57,077 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2026-03-23 04:29:57,091 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103?flat'} 2026-03-23 04:29:57,091 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 is found in opserver 2026-03-23 04:29:57,117 - DEBUG - Do not have enough data to verify VN in agent 2026-03-23 04:29:57,132 - DEBUG - VRF ids for VN ctest-vn1-67631103: {} 2026-03-23 04:29:57,787 - INFO - Created VN ctest-vn2-98616019, UUID :4d9c728e-7915-4670-9c49-79024f2c614c 2026-03-23 04:29:58,134 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-03-23 04:29:58,145 - DEBUG - Requesting: http://10.0.0.27:8082/domain/54adab80-1c58-45eb-8fc6-565ee3803789 2026-03-23 04:29:58,159 - DEBUG - Requesting: http://10.0.0.27:8082/project/c818fea2-bb8c-4dc4-8c5c-a693e3a97287 2026-03-23 04:29:58,273 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/4d9c728e-7915-4670-9c49-79024f2c614c 2026-03-23 04:29:58,294 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/4d9c728e-7915-4670-9c49-79024f2c614c 2026-03-23 04:29:58,305 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/11fada6e-e190-4f4a-9dec-871575530518 2026-03-23 04:29:58,315 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/11fada6e-e190-4f4a-9dec-871575530518 2026-03-23 04:29:58,327 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/f7aafe46-6120-4afa-a84d-1b8952337f2d 2026-03-23 04:29:58,338 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-03-23 04:29:58,338 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/4d9c728e-7915-4670-9c49-79024f2c614c 2026-03-23 04:29:58,352 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/11fada6e-e190-4f4a-9dec-871575530518 2026-03-23 04:29:58,372 - INFO - Verified VN network id 16 for VN 4d9c728e-7915-4670-9c49-79024f2c614c 2026-03-23 04:29:58,373 - INFO - Verifications in API Server for VN ctest-vn2-98616019 passed 2026-03-23 04:29:58,373 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/4d9c728e-7915-4670-9c49-79024f2c614c 2026-03-23 04:29:58,387 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/11fada6e-e190-4f4a-9dec-871575530518 2026-03-23 04:29:58,398 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/11fada6e-e190-4f4a-9dec-871575530518 2026-03-23 04:29:58,407 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/f7aafe46-6120-4afa-a84d-1b8952337f2d 2026-03-23 04:29:58,424 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-88137413', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '5592470793529673328', 'uuid-lslong': '11261665394041839948'}, 'enable': 'true', 'created': '2026-03-23T04:29:57', 'last-modified': '2026-03-23T04:29:57', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.494884'} 2026-03-23 04:29:58,432 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-03-23 04:29:58,438 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-88137413', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '5592470793529673328', 'uuid-lslong': '11261665394041839948'}, 'enable': 'true', 'created': '2026-03-23T04:29:57', 'last-modified': '2026-03-23T04:29:57', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.504541'} 2026-03-23 04:29:58,446 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-03-23 04:29:58,456 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-88137413', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '5592470793529673328', 'uuid-lslong': '11261665394041839948'}, 'enable': 'true', 'created': '2026-03-23T04:29:57', 'last-modified': '2026-03-23T04:29:57', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.523339'} 2026-03-23 04:29:58,464 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-03-23 04:29:58,464 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn2-98616019 passed 2026-03-23 04:29:58,464 - DEBUG - ====Verifying policy data for ctest-vn2-98616019 in API_Server ====== 2026-03-23 04:29:58,464 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-03-23 04:29:58,474 - DEBUG - Requesting: http://10.0.0.27:8082/domain/54adab80-1c58-45eb-8fc6-565ee3803789 2026-03-23 04:29:58,488 - DEBUG - Requesting: http://10.0.0.27:8082/project/c818fea2-bb8c-4dc4-8c5c-a693e3a97287 2026-03-23 04:29:58,595 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/4d9c728e-7915-4670-9c49-79024f2c614c 2026-03-23 04:29:58,608 - DEBUG - =>VN ctest-vn2-98616019 has no policy to be verified 2026-03-23 04:29:58,608 - DEBUG - Verifying the vn in opserver 2026-03-23 04:29:58,608 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 virtual network link through opserver 10.0.0.27 2026-03-23 04:29:58,608 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2026-03-23 04:29:58,626 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019?flat'} 2026-03-23 04:29:58,626 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 virtual network link through opserver 10.0.0.50 2026-03-23 04:29:58,626 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2026-03-23 04:29:58,643 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019?flat'} 2026-03-23 04:29:58,643 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 virtual network link through opserver 10.0.0.23 2026-03-23 04:29:58,643 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2026-03-23 04:29:58,663 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019?flat'} 2026-03-23 04:29:58,664 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 is found in opserver 2026-03-23 04:29:58,690 - DEBUG - Do not have enough data to verify VN in agent 2026-03-23 04:29:58,704 - DEBUG - VRF ids for VN ctest-vn2-98616019: {} 2026-03-23 04:30:00,371 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-5514-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-5514-2) 2026-03-23 04:30:02,091 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-5514-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-5514-1) 2026-03-23 04:30:02,091 - INFO - Waiting for VM ctest-vn1_vm1-40167126 to be up.. 2026-03-23 04:30:02,203 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2026-03-23 04:30:07,316 - DEBUG - VM is in ACTIVE state now 2026-03-23 04:30:07,316 - INFO - VM name : ctest-vn1_vm1-40167126 2026-03-23 04:30:07,417 - DEBUG - VM ctest-vn1_vm1-40167126 ID is 91e652d8-37f4-498c-95d3-b9901e41ee32 2026-03-23 04:30:07,418 - DEBUG - VM ctest-vn1_vm1-40167126 launched on Node an-jenkins-deploy-platform-ansible-os-5514-2 2026-03-23 04:30:07,526 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/91e652d8-37f4-498c-95d3-b9901e41ee32 2026-03-23 04:30:07,538 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/b4e4e56d-5a12-4e50-8592-a87ecd35a758 2026-03-23 04:30:10,723 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1004ms') 2026-03-23 04:30:10,723 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-40167126 failed! 2026-03-23 04:30:10,737 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 is 192.168.1.254 and allocation pool is NOT set 2026-03-23 04:30:14,826 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2026-03-23 04:30:14,826 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-40167126 failed! 2026-03-23 04:30:14,842 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 is 192.168.1.254 and allocation pool is NOT set 2026-03-23 04:30:18,922 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2026-03-23 04:30:18,923 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-40167126 failed! 2026-03-23 04:30:18,938 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 is 192.168.1.254 and allocation pool is NOT set 2026-03-23 04:30:23,017 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms') 2026-03-23 04:30:23,017 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-40167126 failed! 2026-03-23 04:30:23,035 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 is 192.168.1.254 and allocation pool is NOT set 2026-03-23 04:30:27,115 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2026-03-23 04:30:27,115 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-40167126 failed! 2026-03-23 04:30:27,129 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 is 192.168.1.254 and allocation pool is NOT set 2026-03-23 04:30:31,212 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1016ms') 2026-03-23 04:30:31,212 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-40167126 failed! 2026-03-23 04:30:31,265 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 is 192.168.1.254 and allocation pool is NOT set 2026-03-23 04:30:35,339 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1007ms') 2026-03-23 04:30:35,339 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-40167126 failed! 2026-03-23 04:30:35,356 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 is 192.168.1.254 and allocation pool is NOT set 2026-03-23 04:30:39,428 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1006ms') 2026-03-23 04:30:39,428 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-40167126 failed! 2026-03-23 04:30:39,444 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 is 192.168.1.254 and allocation pool is NOT set 2026-03-23 04:30:43,532 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1021ms') 2026-03-23 04:30:43,533 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-40167126 failed! 2026-03-23 04:30:43,549 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 is 192.168.1.254 and allocation pool is NOT set 2026-03-23 04:30:45,620 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=10.6 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=3.21 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 3.214/6.884/10.555/3.670 ms') 2026-03-23 04:30:45,620 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-40167126 passed 2026-03-23 04:30:45,697 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:30:45,697 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-40167126, IP 192.168.1.252, Port 22 2026-03-23 04:30:45,764 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-03-23 04:30:45,991 - DEBUG - VM ctest-vn1_vm1-40167126 is NOT ready for SSH connections, VM status: ACTIVE 2026-03-23 04:30:50,992 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:30:50,992 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-40167126, IP 192.168.1.252, Port 22 2026-03-23 04:30:51,060 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-03-23 04:30:51,203 - DEBUG - VM ctest-vn1_vm1-40167126 is NOT ready for SSH connections, VM status: ACTIVE 2026-03-23 04:30:56,203 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:30:56,204 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-40167126, IP 192.168.1.252, Port 22 2026-03-23 04:30:56,271 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-03-23 04:30:56,372 - DEBUG - VM ctest-vn1_vm1-40167126 is NOT ready for SSH connections, VM status: ACTIVE 2026-03-23 04:31:01,372 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:31:01,373 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-40167126, IP 192.168.1.252, Port 22 2026-03-23 04:31:01,440 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-03-23 04:31:01,547 - DEBUG - VM ctest-vn1_vm1-40167126 is NOT ready for SSH connections, VM status: ACTIVE 2026-03-23 04:31:06,547 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:31:06,547 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-40167126, IP 192.168.1.252, Port 22 2026-03-23 04:31:06,725 - DEBUG - VM ctest-vn1_vm1-40167126 is ready for SSH connections 2026-03-23 04:31:06,725 - INFO - Waiting for VM ctest-vn2_vm1-75122358 to be up.. 2026-03-23 04:31:06,833 - DEBUG - VM is in ACTIVE state now 2026-03-23 04:31:06,833 - INFO - VM name : ctest-vn2_vm1-75122358 2026-03-23 04:31:06,943 - DEBUG - VM ctest-vn2_vm1-75122358 ID is 972096c0-ac28-48c1-978c-55fda1fd771c 2026-03-23 04:31:06,943 - DEBUG - VM ctest-vn2_vm1-75122358 launched on Node an-jenkins-deploy-platform-ansible-os-5514-1 2026-03-23 04:31:07,046 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/972096c0-ac28-48c1-978c-55fda1fd771c 2026-03-23 04:31:07,101 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/a55f5f87-b72c-4ad4-839d-777c18d186d2 2026-03-23 04:31:09,176 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=32.8 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=1.57 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 1.573/17.180/32.787/15.607 ms') 2026-03-23 04:31:09,176 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn2_vm1-75122358 passed 2026-03-23 04:31:09,249 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:31:09,249 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-75122358, IP 192.168.2.252, Port 22 2026-03-23 04:31:09,318 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-03-23 04:31:09,416 - DEBUG - VM ctest-vn2_vm1-75122358 is NOT ready for SSH connections, VM status: ACTIVE 2026-03-23 04:31:14,417 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:31:14,417 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-75122358, IP 192.168.2.252, Port 22 2026-03-23 04:31:14,488 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-03-23 04:31:14,609 - DEBUG - VM ctest-vn2_vm1-75122358 is NOT ready for SSH connections, VM status: ACTIVE 2026-03-23 04:31:19,610 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:31:19,610 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-75122358, IP 192.168.2.252, Port 22 2026-03-23 04:31:19,788 - DEBUG - VM ctest-vn2_vm1-75122358 is ready for SSH connections 2026-03-23 04:31:19,788 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:31:19,788 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.252, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.65, gateway password: c0ntrail123 2026-03-23 04:31:19,788 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.252 2026-03-23 04:32:07,497 - DEBUG - PING 192.168.2.252 (192.168.2.252) 56(84) bytes of data. 64 bytes from 192.168.2.252: icmp_req=1 ttl=63 time=9.33 ms 64 bytes from 192.168.2.252: icmp_req=2 ttl=63 time=1.73 ms 64 bytes from 192.168.2.252: icmp_req=3 ttl=63 time=1.59 ms --- 192.168.2.252 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 1.599/4.220/9.332/3.615 ms 2026-03-23 04:32:07,497 - INFO - Ping to IP 192.168.2.252 from VM ctest-vn1_vm1-40167126 passed 2026-03-23 04:32:10,945 - INFO - 10.0.0.249 2026-03-23 04:32:10,950 - INFO - agent:initializing 2026-03-23 04:32:10,950 - DEBUG - defaultdict(, {'10.0.0.249': {'agent': {'status': 'initializing', 'description': None}}}) 2026-03-23 04:32:10,950 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-03-23 04:32:16,050 - INFO - 10.0.0.249 2026-03-23 04:32:22,340 - INFO - agent:initializing (XMPP:control-node:10.20.0.14, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.14, XMPP:dns-server:10.20.0.25 connection down, No Configuration for self) 2026-03-23 04:32:22,341 - DEBUG - defaultdict(, {'10.0.0.249': {'agent': {'status': 'initializing', 'description': 'XMPP:control-node:10.20.0.14, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.14, XMPP:dns-server:10.20.0.25 connection down, No Configuration for self'}}}) 2026-03-23 04:32:22,341 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2026-03-23 04:32:27,443 - INFO - 10.0.0.249 2026-03-23 04:32:27,451 - INFO - agent:active 2026-03-23 04:32:27,455 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.249 2026-03-23 04:32:30,889 - INFO - 10.0.0.65 2026-03-23 04:32:30,894 - INFO - agent:initializing 2026-03-23 04:32:30,895 - DEBUG - defaultdict(, {'10.0.0.65': {'agent': {'status': 'initializing', 'description': None}}}) 2026-03-23 04:32:30,895 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-03-23 04:32:35,995 - INFO - 10.0.0.65 2026-03-23 04:32:36,001 - INFO - agent:active 2026-03-23 04:32:36,001 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.65 2026-03-23 04:32:36,562 - INFO - 10.0.0.27 2026-03-23 04:32:36,568 - INFO - control:initializing 2026-03-23 04:32:36,568 - DEBUG - defaultdict(, {'10.0.0.27': {'control': {'status': 'initializing', 'description': None}}}) 2026-03-23 04:32:36,568 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-03-23 04:32:41,691 - INFO - 10.0.0.27 2026-03-23 04:32:41,697 - INFO - control:active 2026-03-23 04:32:41,698 - INFO - Contrail services ['control'] are up on nodes 10.0.0.27 2026-03-23 04:32:42,186 - INFO - 10.0.0.50 2026-03-23 04:32:42,191 - INFO - control:initializing 2026-03-23 04:32:42,191 - DEBUG - defaultdict(, {'10.0.0.50': {'control': {'status': 'initializing', 'description': None}}}) 2026-03-23 04:32:42,191 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-03-23 04:32:47,290 - INFO - 10.0.0.50 2026-03-23 04:32:47,298 - INFO - control:active 2026-03-23 04:32:47,298 - INFO - Contrail services ['control'] are up on nodes 10.0.0.50 2026-03-23 04:32:47,719 - INFO - 10.0.0.23 2026-03-23 04:32:47,723 - INFO - control:initializing 2026-03-23 04:32:47,723 - DEBUG - defaultdict(, {'10.0.0.23': {'control': {'status': 'initializing', 'description': None}}}) 2026-03-23 04:32:47,723 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-03-23 04:32:52,815 - INFO - 10.0.0.23 2026-03-23 04:32:52,821 - INFO - control:active 2026-03-23 04:32:52,821 - INFO - Contrail services ['control'] are up on nodes 10.0.0.23 2026-03-23 04:32:53,757 - INFO - 10.0.0.27 2026-03-23 04:32:53,760 - INFO - api-server:initializing 2026-03-23 04:32:53,760 - DEBUG - defaultdict(, {'10.0.0.27': {'api-server': {'status': 'initializing', 'description': None}}}) 2026-03-23 04:32:53,760 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-03-23 04:32:58,914 - INFO - 10.0.0.27 2026-03-23 04:32:58,923 - INFO - api-server:initializing 2026-03-23 04:32:58,923 - DEBUG - defaultdict(, {'10.0.0.27': {'api-server': {'status': 'initializing', 'description': None}}}) 2026-03-23 04:32:58,923 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2026-03-23 04:33:04,061 - INFO - 10.0.0.27 2026-03-23 04:33:04,070 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2026-03-23 04:33:04,071 - DEBUG - defaultdict(, {'10.0.0.27': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2026-03-23 04:33:04,071 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2026-03-23 04:33:09,179 - INFO - 10.0.0.27 2026-03-23 04:33:09,184 - INFO - api-server:active 2026-03-23 04:33:09,184 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.27 2026-03-23 04:33:09,919 - INFO - 10.0.0.50 2026-03-23 04:33:09,926 - INFO - api-server:initializing 2026-03-23 04:33:09,926 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': None}}}) 2026-03-23 04:33:09,926 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-03-23 04:33:15,026 - INFO - 10.0.0.50 2026-03-23 04:33:15,032 - INFO - api-server:initializing 2026-03-23 04:33:15,032 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': None}}}) 2026-03-23 04:33:15,032 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2026-03-23 04:33:20,135 - INFO - 10.0.0.50 2026-03-23 04:33:20,141 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2026-03-23 04:33:20,141 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2026-03-23 04:33:20,141 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2026-03-23 04:33:25,243 - INFO - 10.0.0.50 2026-03-23 04:33:25,249 - INFO - api-server:active 2026-03-23 04:33:25,249 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.50 2026-03-23 04:33:25,886 - INFO - 10.0.0.23 2026-03-23 04:33:25,892 - INFO - api-server:initializing 2026-03-23 04:33:25,892 - DEBUG - defaultdict(, {'10.0.0.23': {'api-server': {'status': 'initializing', 'description': None}}}) 2026-03-23 04:33:25,892 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-03-23 04:33:30,994 - INFO - 10.0.0.23 2026-03-23 04:33:30,998 - INFO - api-server:initializing 2026-03-23 04:33:30,998 - DEBUG - defaultdict(, {'10.0.0.23': {'api-server': {'status': 'initializing', 'description': None}}}) 2026-03-23 04:33:30,999 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2026-03-23 04:33:36,088 - INFO - 10.0.0.23 2026-03-23 04:33:36,095 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2026-03-23 04:33:36,095 - DEBUG - defaultdict(, {'10.0.0.23': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2026-03-23 04:33:36,095 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2026-03-23 04:33:41,200 - INFO - 10.0.0.23 2026-03-23 04:33:41,205 - INFO - api-server:active 2026-03-23 04:33:41,205 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.23 2026-03-23 04:33:41,329 - INFO - 10.0.0.27 2026-03-23 04:33:41,329 - INFO - config 2026-03-23 04:33:41,336 - INFO - config-nodemgr:active 2026-03-23 04:33:41,342 - INFO - api-server:active 2026-03-23 04:33:41,347 - INFO - schema:backup 2026-03-23 04:33:41,351 - INFO - svc-monitor:backup 2026-03-23 04:33:41,356 - INFO - device-manager:backup 2026-03-23 04:33:41,357 - INFO - config-database 2026-03-23 04:33:41,357 - INFO - config-cassandra:active 2026-03-23 04:33:41,357 - INFO - config-zookeeper:active 2026-03-23 04:33:41,357 - INFO - config-rabbitmq:active 2026-03-23 04:33:41,357 - INFO - control 2026-03-23 04:33:41,363 - INFO - control-nodemgr:active 2026-03-23 04:33:41,369 - INFO - control:active 2026-03-23 04:33:41,369 - INFO - named:active 2026-03-23 04:33:41,375 - INFO - dns:active 2026-03-23 04:33:41,375 - INFO - analytics 2026-03-23 04:33:41,380 - INFO - analytics-nodemgr:active 2026-03-23 04:33:41,385 - INFO - analytics-api:active 2026-03-23 04:33:41,391 - INFO - collector:active 2026-03-23 04:33:41,391 - INFO - analytics-database 2026-03-23 04:33:41,391 - INFO - analytics-cassandra:active 2026-03-23 04:33:41,396 - INFO - analyticsdb-nodemgr:active 2026-03-23 04:33:41,401 - INFO - query-engine:active 2026-03-23 04:33:41,401 - INFO - webui 2026-03-23 04:33:41,401 - INFO - webui:active 2026-03-23 04:33:41,401 - INFO - webui-middleware:active 2026-03-23 04:33:41,401 - INFO - redis:active 2026-03-23 04:33:41,401 - INFO - analytics_snmp 2026-03-23 04:33:41,406 - INFO - snmp-collector:active 2026-03-23 04:33:41,406 - INFO - snmp-topology:active 2026-03-23 04:33:41,406 - INFO - analytics_alarm 2026-03-23 04:33:41,406 - INFO - alarmgen:active 2026-03-23 04:33:41,508 - INFO - 10.0.0.50 2026-03-23 04:33:41,508 - INFO - config 2026-03-23 04:33:41,515 - INFO - config-nodemgr:active 2026-03-23 04:33:41,520 - INFO - api-server:active 2026-03-23 04:33:41,524 - INFO - schema:backup 2026-03-23 04:33:41,529 - INFO - svc-monitor:active 2026-03-23 04:33:41,533 - INFO - device-manager:backup 2026-03-23 04:33:41,533 - INFO - config-database 2026-03-23 04:33:41,533 - INFO - config-cassandra:active 2026-03-23 04:33:41,533 - INFO - config-zookeeper:active 2026-03-23 04:33:41,533 - INFO - config-rabbitmq:active 2026-03-23 04:33:41,533 - INFO - control 2026-03-23 04:33:41,539 - INFO - control-nodemgr:active 2026-03-23 04:33:41,545 - INFO - control:active 2026-03-23 04:33:41,545 - INFO - named:active 2026-03-23 04:33:41,550 - INFO - dns:active 2026-03-23 04:33:41,550 - INFO - analytics 2026-03-23 04:33:41,557 - INFO - analytics-nodemgr:active 2026-03-23 04:33:41,563 - INFO - analytics-api:active 2026-03-23 04:33:41,568 - INFO - collector:active 2026-03-23 04:33:41,568 - INFO - analytics-database 2026-03-23 04:33:41,568 - INFO - analytics-cassandra:active 2026-03-23 04:33:41,574 - INFO - analyticsdb-nodemgr:active 2026-03-23 04:33:41,579 - INFO - query-engine:active 2026-03-23 04:33:41,580 - INFO - webui 2026-03-23 04:33:41,580 - INFO - webui:active 2026-03-23 04:33:41,580 - INFO - webui-middleware:active 2026-03-23 04:33:41,580 - INFO - redis:active 2026-03-23 04:33:41,580 - INFO - analytics_snmp 2026-03-23 04:33:41,584 - INFO - snmp-collector:active 2026-03-23 04:33:41,584 - INFO - snmp-topology:active 2026-03-23 04:33:41,584 - INFO - analytics_alarm 2026-03-23 04:33:41,585 - INFO - alarmgen:active 2026-03-23 04:33:41,687 - INFO - 10.0.0.23 2026-03-23 04:33:41,688 - INFO - config 2026-03-23 04:33:41,696 - INFO - config-nodemgr:active 2026-03-23 04:33:41,701 - INFO - api-server:active 2026-03-23 04:33:41,707 - INFO - schema:active 2026-03-23 04:33:41,711 - INFO - svc-monitor:backup 2026-03-23 04:33:41,717 - INFO - device-manager:active 2026-03-23 04:33:41,717 - INFO - config-database 2026-03-23 04:33:41,717 - INFO - config-cassandra:active 2026-03-23 04:33:41,717 - INFO - config-zookeeper:active 2026-03-23 04:33:41,717 - INFO - config-rabbitmq:active 2026-03-23 04:33:41,718 - INFO - control 2026-03-23 04:33:41,723 - INFO - control-nodemgr:active 2026-03-23 04:33:41,731 - INFO - control:active 2026-03-23 04:33:41,731 - INFO - named:active 2026-03-23 04:33:41,737 - INFO - dns:active 2026-03-23 04:33:41,738 - INFO - analytics 2026-03-23 04:33:41,743 - INFO - analytics-nodemgr:active 2026-03-23 04:33:41,749 - INFO - analytics-api:active 2026-03-23 04:33:41,754 - INFO - collector:active 2026-03-23 04:33:41,755 - INFO - analytics-database 2026-03-23 04:33:41,755 - INFO - analytics-cassandra:active 2026-03-23 04:33:41,761 - INFO - analyticsdb-nodemgr:active 2026-03-23 04:33:41,767 - INFO - query-engine:active 2026-03-23 04:33:41,767 - INFO - webui 2026-03-23 04:33:41,767 - INFO - webui:active 2026-03-23 04:33:41,767 - INFO - webui-middleware:active 2026-03-23 04:33:41,768 - INFO - redis:active 2026-03-23 04:33:41,768 - INFO - analytics_snmp 2026-03-23 04:33:41,773 - INFO - snmp-collector:active 2026-03-23 04:33:41,773 - INFO - snmp-topology:active 2026-03-23 04:33:41,773 - INFO - analytics_alarm 2026-03-23 04:33:41,773 - INFO - alarmgen:active 2026-03-23 04:33:41,873 - INFO - 10.0.0.249 2026-03-23 04:33:41,873 - INFO - vrouter 2026-03-23 04:33:41,879 - INFO - vrouter-nodemgr:active 2026-03-23 04:33:41,886 - INFO - agent:active 2026-03-23 04:33:41,999 - INFO - 10.0.0.65 2026-03-23 04:33:41,999 - INFO - vrouter 2026-03-23 04:33:42,008 - INFO - vrouter-nodemgr:active 2026-03-23 04:33:42,015 - INFO - agent:active 2026-03-23 04:33:42,016 - INFO - All the contrail services are up on all nodes 2026-03-23 04:33:42,016 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-1?flat 2026-03-23 04:33:42,136 - INFO - an-jenkins-deploy-platform-ansible-os-5514-1:contrail-vrouter-agent module connection to ['10.20.0.14:5269'] servers UP 2026-03-23 04:33:42,136 - INFO - an-jenkins-deploy-platform-ansible-os-5514-1 connected to xmpp 10.20.0.14 2026-03-23 04:33:42,136 - INFO - an-jenkins-deploy-platform-ansible-os-5514-1:contrail-vrouter-agent module connection to ['10.20.0.25:5269'] servers UP 2026-03-23 04:33:42,136 - INFO - an-jenkins-deploy-platform-ansible-os-5514-1 connected to xmpp 10.20.0.25 2026-03-23 04:33:42,136 - WARNING - an-jenkins-deploy-platform-ansible-os-5514-1:contrail-vrouter-agent module connection to ['10.20.0.18:5269'] servers NOT UP 2026-03-23 04:33:42,136 - INFO - an-jenkins-deploy-platform-ansible-os-5514-1:contrail-vrouter-agent module connection to ['10.20.0.14:53'] servers UP 2026-03-23 04:33:42,136 - INFO - an-jenkins-deploy-platform-ansible-os-5514-1 connected to dns 10.20.0.14 2026-03-23 04:33:42,136 - INFO - an-jenkins-deploy-platform-ansible-os-5514-1:contrail-vrouter-agent module connection to ['10.20.0.25:53'] servers UP 2026-03-23 04:33:42,136 - INFO - an-jenkins-deploy-platform-ansible-os-5514-1 connected to dns 10.20.0.25 2026-03-23 04:33:42,137 - WARNING - an-jenkins-deploy-platform-ansible-os-5514-1:contrail-vrouter-agent module connection to ['10.20.0.18:53'] servers NOT UP 2026-03-23 04:33:42,137 - WARNING - an-jenkins-deploy-platform-ansible-os-5514-1:contrail-vrouter-agent module connection to ['10.0.0.27:8086'] servers NOT UP 2026-03-23 04:33:42,137 - INFO - an-jenkins-deploy-platform-ansible-os-5514-1:contrail-vrouter-agent module connection to ['10.0.0.50:8086'] servers UP 2026-03-23 04:33:42,137 - INFO - an-jenkins-deploy-platform-ansible-os-5514-1 connected to collector 10.0.0.50 2026-03-23 04:33:42,138 - WARNING - an-jenkins-deploy-platform-ansible-os-5514-1:contrail-vrouter-agent module connection to ['10.0.0.23:8086'] servers NOT UP 2026-03-23 04:33:42,138 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-5514-1?flat 2026-03-23 04:33:42,220 - INFO - cn-jenkins-deploy-platform-ansible-os-5514-1:contrail-control module connection to 10.0.0.27:9041 servers UP 2026-03-23 04:33:42,220 - INFO - cn-jenkins-deploy-platform-ansible-os-5514-1:contrail-control module connection to 10.0.0.27:5673 servers UP 2026-03-23 04:33:42,220 - INFO - cn-jenkins-deploy-platform-ansible-os-5514-1:contrail-control module connection to ['10.0.0.27:8086'] servers UP 2026-03-23 04:33:42,225 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-5514-2?flat 2026-03-23 04:33:42,704 - INFO - cn-jenkins-deploy-platform-ansible-os-5514-2:contrail-control module connection to 10.0.0.27:9041 servers UP 2026-03-23 04:33:42,704 - INFO - cn-jenkins-deploy-platform-ansible-os-5514-2:contrail-control module connection to 10.0.0.27:5673 servers UP 2026-03-23 04:33:42,704 - INFO - cn-jenkins-deploy-platform-ansible-os-5514-2:contrail-control module connection to ['10.0.0.27:8086'] servers UP 2026-03-23 04:33:42,705 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-5514-3?flat 2026-03-23 04:33:42,731 - INFO - cn-jenkins-deploy-platform-ansible-os-5514-3:contrail-control module connection to 10.0.0.27:9041 servers UP 2026-03-23 04:33:42,731 - INFO - cn-jenkins-deploy-platform-ansible-os-5514-3:contrail-control module connection to 10.0.0.27:5673 servers UP 2026-03-23 04:33:42,731 - INFO - cn-jenkins-deploy-platform-ansible-os-5514-3:contrail-control module connection to ['10.0.0.27:8086'] servers UP 2026-03-23 04:33:42,732 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-5514-1.?flat 2026-03-23 04:33:42,761 - DEBUG - contrail-api is running 2026-03-23 04:33:42,762 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-5514-2.?flat 2026-03-23 04:33:42,809 - DEBUG - contrail-api is running 2026-03-23 04:33:42,809 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-5514-3.?flat 2026-03-23 04:33:42,843 - DEBUG - contrail-api is running 2026-03-23 04:33:43,187 - INFO - Created VN ctest-vn3-95184347, UUID :2beceb16-5205-4fc4-af27-95539c2b7c3a 2026-03-23 04:33:43,660 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-03-23 04:33:43,785 - DEBUG - Requesting: http://10.0.0.27:8082/domain/54adab80-1c58-45eb-8fc6-565ee3803789 2026-03-23 04:33:43,803 - DEBUG - Requesting: http://10.0.0.27:8082/project/c818fea2-bb8c-4dc4-8c5c-a693e3a97287 2026-03-23 04:33:43,939 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/3bd95fd3-bf08-4424-9610-ec574d9aa238 2026-03-23 04:33:43,956 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/3bd95fd3-bf08-4424-9610-ec574d9aa238 2026-03-23 04:33:43,971 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/38abe83a-48fc-48ae-ad52-170aca7450e8 2026-03-23 04:33:43,984 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/38abe83a-48fc-48ae-ad52-170aca7450e8 2026-03-23 04:33:43,996 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/a1b9913a-8fde-46bf-a64d-c334cb8c7e90 2026-03-23 04:33:44,007 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-03-23 04:33:44,007 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/3bd95fd3-bf08-4424-9610-ec574d9aa238 2026-03-23 04:33:44,025 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/38abe83a-48fc-48ae-ad52-170aca7450e8 2026-03-23 04:33:44,051 - INFO - Verified VN network id 15 for VN 3bd95fd3-bf08-4424-9610-ec574d9aa238 2026-03-23 04:33:44,051 - INFO - Verifications in API Server for VN ctest-vn1-67631103 passed 2026-03-23 04:33:44,051 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/3bd95fd3-bf08-4424-9610-ec574d9aa238 2026-03-23 04:33:44,064 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/38abe83a-48fc-48ae-ad52-170aca7450e8 2026-03-23 04:33:44,078 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/38abe83a-48fc-48ae-ad52-170aca7450e8 2026-03-23 04:33:44,088 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/a1b9913a-8fde-46bf-a64d-c334cb8c7e90 2026-03-23 04:33:44,108 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103', 'interests': '1', 'advertised': '1', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-88137413', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '4312583481231688740', 'uuid-lslong': '10813402565024850488'}, 'enable': 'true', 'created': '2026-03-23T04:29:56', 'last-modified': '2026-03-23T04:29:57', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:07.091347'} 2026-03-23 04:33:44,117 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-03-23 04:33:44,125 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-88137413', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '4312583481231688740', 'uuid-lslong': '10813402565024850488'}, 'enable': 'true', 'created': '2026-03-23T04:29:56', 'last-modified': '2026-03-23T04:29:57', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:01.483859'} 2026-03-23 04:33:44,134 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-03-23 04:33:44,140 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-88137413', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '4312583481231688740', 'uuid-lslong': '10813402565024850488'}, 'enable': 'true', 'created': '2026-03-23T04:29:56', 'last-modified': '2026-03-23T04:29:57', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:56.084857'} 2026-03-23 04:33:44,147 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-03-23 04:33:44,147 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-67631103 passed 2026-03-23 04:33:44,147 - DEBUG - ====Verifying policy data for ctest-vn1-67631103 in API_Server ====== 2026-03-23 04:33:44,147 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-03-23 04:33:44,155 - DEBUG - Requesting: http://10.0.0.27:8082/domain/54adab80-1c58-45eb-8fc6-565ee3803789 2026-03-23 04:33:44,168 - DEBUG - Requesting: http://10.0.0.27:8082/project/c818fea2-bb8c-4dc4-8c5c-a693e3a97287 2026-03-23 04:33:44,283 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/3bd95fd3-bf08-4424-9610-ec574d9aa238 2026-03-23 04:33:44,295 - DEBUG - ==>Verifying data for policy with id: 28976e12-e9bf-4db3-814f-994758b21f56, fqn: ['default-domain', 'ctest-TestBasicVMVN0-88137413', 'policy1'] 2026-03-23 04:33:44,295 - INFO - VN ctest-vn1-67631103 Policy verification: verify_vn_policy_in_api_server, status: True 2026-03-23 04:33:44,295 - DEBUG - Verifying the vn in opserver 2026-03-23 04:33:44,295 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 virtual network link through opserver 10.0.0.27 2026-03-23 04:33:44,295 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2026-03-23 04:33:44,315 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103?flat'} 2026-03-23 04:33:44,315 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 virtual network link through opserver 10.0.0.50 2026-03-23 04:33:44,315 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2026-03-23 04:33:44,402 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103?flat'} 2026-03-23 04:33:44,402 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 virtual network link through opserver 10.0.0.23 2026-03-23 04:33:44,402 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2026-03-23 04:33:44,493 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103?flat'} 2026-03-23 04:33:44,493 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 is found in opserver 2026-03-23 04:33:44,507 - DEBUG - VRF ids for VN ctest-vn1-67631103: {'10.0.0.65': '2'} 2026-03-23 04:33:46,071 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-5514-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-5514-2) 2026-03-23 04:33:46,136 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2026-03-23 04:33:51,237 - DEBUG - VM is in ACTIVE state now 2026-03-23 04:33:51,237 - INFO - VM name : ctest-vn1_vm2-66507978 2026-03-23 04:33:51,338 - DEBUG - VM ctest-vn1_vm2-66507978 ID is 6ad7964d-b8c4-449b-ac63-05c4a9545bff 2026-03-23 04:33:51,433 - DEBUG - VM is in ACTIVE state now 2026-03-23 04:33:51,433 - DEBUG - VM ctest-vn1_vm2-66507978 launched on Node an-jenkins-deploy-platform-ansible-os-5514-2 2026-03-23 04:33:51,534 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/6ad7964d-b8c4-449b-ac63-05c4a9545bff 2026-03-23 04:33:51,577 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/6ad7964d-b8c4-449b-ac63-05c4a9545bff 2026-03-23 04:33:51,586 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/e2168f2b-ca31-4c57-abcd-f5fd407f422f 2026-03-23 04:33:51,601 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/6ad7964d-b8c4-449b-ac63-05c4a9545bff 2026-03-23 04:33:51,609 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/e2168f2b-ca31-4c57-abcd-f5fd407f422f 2026-03-23 04:33:51,619 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/cdf59f4b-e86b-45f8-af5f-a6702f36ecbf 2026-03-23 04:33:51,632 - DEBUG - Verifying in api server 10.0.0.23 2026-03-23 04:33:51,632 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/6ad7964d-b8c4-449b-ac63-05c4a9545bff 2026-03-23 04:33:51,641 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/e2168f2b-ca31-4c57-abcd-f5fd407f422f 2026-03-23 04:33:51,654 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/cdf59f4b-e86b-45f8-af5f-a6702f36ecbf 2026-03-23 04:33:51,669 - INFO - VM ctest-vn1_vm2-66507978 verfication in all API Servers passed 2026-03-23 04:33:51,703 - DEBUG - VM ctest-vn1_vm2-66507978 Tap interface: {'index': '4', 'name': 'tape2168f2b-ca', 'uuid': 'e2168f2b-ca31-4c57-abcd-f5fd407f422f', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103:ctest-vn1-67631103', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103', 'vm_uuid': '6ad7964d-b8c4-449b-ac63-05c4a9545bff', 'vm_name': 'ctest-vn1_vm2-66507978', 'ip_addr': '192.168.1.251', 'mac_addr': '02:e2:16:8f:2b:ca', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '165', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-88137413:e2168f2b-ca31-4c57-abcd-f5fd407f422f', 'sg_uuid_list': ['42966352-7ed2-4211-baec-acac6f7d89b2'], 'static_route_list': None, 'vm_project_uuid': 'c818fea2-bb8c-4dc4-8c5c-a693e3a97287', 'admin_state': 'Enabled', 'flow_key_idx': '38', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-03-23 04:33:51,703 - DEBUG - Agent 10.0.0.65 vrf name: default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103:ctest-vn1-67631103 2026-03-23 04:33:51,716 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103:ctest-vn1-67631103', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103', 'table_label': '-1', 'vxlan_id': '15', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.201:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2026-03-23 04:33:51,805 - DEBUG - VM ctest-vn1_vm2-66507978 : Tap interface tape2168f2b-ca is set to Active 2026-03-23 04:33:51,805 - DEBUG - Tap interface tape2168f2b-ca detail : {'index': '4', 'name': 'tape2168f2b-ca', 'uuid': 'e2168f2b-ca31-4c57-abcd-f5fd407f422f', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103:ctest-vn1-67631103', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103', 'vm_uuid': '6ad7964d-b8c4-449b-ac63-05c4a9545bff', 'vm_name': 'ctest-vn1_vm2-66507978', 'ip_addr': '192.168.1.251', 'mac_addr': '02:e2:16:8f:2b:ca', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '165', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-88137413:e2168f2b-ca31-4c57-abcd-f5fd407f422f', 'sg_uuid_list': ['42966352-7ed2-4211-baec-acac6f7d89b2'], 'static_route_list': None, 'vm_project_uuid': 'c818fea2-bb8c-4dc4-8c5c-a693e3a97287', 'admin_state': 'Enabled', 'flow_key_idx': '38', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-03-23 04:33:51,920 - DEBUG - Starting Layer 2 verification in Agent 2026-03-23 04:33:51,937 - DEBUG - Layer 2 path is seen for VM MAC 02:e2:16:8f:2b:ca in agent 10.0.0.65 2026-03-23 04:33:51,938 - DEBUG - Active layer 2 route in agent is present for VMI tape2168f2b-ca 2026-03-23 04:33:51,938 - DEBUG - L2 label(34) matches bw route table and interface table 2026-03-23 04:33:51,954 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 is found in Agent of node 10.0.0.65 2026-03-23 04:33:52,046 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 is consistent in agent 10.0.0.65 2026-03-23 04:33:52,046 - DEBUG - Route for VM IP ['192.168.1.251'] is consistent in agent 10.0.0.65 2026-03-23 04:33:52,046 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 verification for VM ctest-vn1_vm2-66507978 in Agent 10.0.0.65 passed 2026-03-23 04:33:52,160 - DEBUG - Starting all layer 2 verification in agent 10.0.0.65 2026-03-23 04:33:52,176 - DEBUG - Route for VM MAC 02:e2:16:8f:2b:ca is consistent in agent 10.0.0.65 2026-03-23 04:33:55,370 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2026-03-23 04:33:55,370 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-66507978 failed! 2026-03-23 04:33:55,387 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 is 192.168.1.254 and allocation pool is NOT set 2026-03-23 04:33:59,461 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2026-03-23 04:33:59,461 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-66507978 failed! 2026-03-23 04:33:59,475 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 is 192.168.1.254 and allocation pool is NOT set 2026-03-23 04:34:03,557 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1020ms') 2026-03-23 04:34:03,557 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-66507978 failed! 2026-03-23 04:34:03,572 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 is 192.168.1.254 and allocation pool is NOT set 2026-03-23 04:34:07,658 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1025ms') 2026-03-23 04:34:07,658 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-66507978 failed! 2026-03-23 04:34:07,673 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 is 192.168.1.254 and allocation pool is NOT set 2026-03-23 04:34:11,754 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2026-03-23 04:34:11,754 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-66507978 failed! 2026-03-23 04:34:11,770 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 is 192.168.1.254 and allocation pool is NOT set 2026-03-23 04:34:15,849 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2026-03-23 04:34:15,849 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-66507978 failed! 2026-03-23 04:34:15,864 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 is 192.168.1.254 and allocation pool is NOT set 2026-03-23 04:34:19,946 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2026-03-23 04:34:19,946 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-66507978 failed! 2026-03-23 04:34:19,961 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 is 192.168.1.254 and allocation pool is NOT set 2026-03-23 04:34:24,038 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms') 2026-03-23 04:34:24,039 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-66507978 failed! 2026-03-23 04:34:24,056 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 is 192.168.1.254 and allocation pool is NOT set 2026-03-23 04:34:28,139 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2026-03-23 04:34:28,140 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-66507978 failed! 2026-03-23 04:34:28,157 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 is 192.168.1.254 and allocation pool is NOT set 2026-03-23 04:34:30,231 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=8.88 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=4.02 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 4.021/6.452/8.883/2.431 ms') 2026-03-23 04:34:30,231 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-66507978 passed 2026-03-23 04:34:30,231 - INFO - VM ctest-vn1_vm2-66507978 verifications in Compute nodes passed 2026-03-23 04:34:30,456 - DEBUG - Validated VM route 192.168.1.251 in vrouter of 10.0.0.65 2026-03-23 04:34:30,463 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2026-03-23 04:34:30,463 - INFO - Validated routes of VM ctest-vn1_vm2-66507978 in all vrouters 2026-03-23 04:34:30,917 - DEBUG - Starting all layer2 verification in 10.0.0.50 Control Node 2026-03-23 04:34:31,042 - DEBUG - Layer2 route found for VM MAC 02:e2:16:8f:2b:ca in Control-node 10.0.0.50 2026-03-23 04:34:31,043 - DEBUG - L2 Label for VM ctest-vn1_vm2-66507978 same between Control-node 10.0.0.50 and Agent, Expected: 34, Seen: 34 2026-03-23 04:34:31,043 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2026-03-23 04:34:31,180 - DEBUG - Layer2 route found for VM MAC 02:e2:16:8f:2b:ca in Control-node 10.0.0.27 2026-03-23 04:34:31,180 - DEBUG - L2 Label for VM ctest-vn1_vm2-66507978 same between Control-node 10.0.0.27 and Agent, Expected: 34, Seen: 34 2026-03-23 04:34:31,180 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-66507978 passed 2026-03-23 04:34:31,180 - DEBUG - Verifying the vm in opserver 2026-03-23 04:34:31,180 - DEBUG - Verifying in collector 10.0.0.23 ... 2026-03-23 04:34:31,180 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/6ad7964d-b8c4-449b-ac63-05c4a9545bff?flat 2026-03-23 04:34:31,268 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-88137413:e2168f2b-ca31-4c57-abcd-f5fd407f422f?flat 2026-03-23 04:34:31,295 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-88137413:e2168f2b-ca31-4c57-abcd-f5fd407f422f?flat 2026-03-23 04:34:31,319 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/6ad7964d-b8c4-449b-ac63-05c4a9545bff?flat 2026-03-23 04:34:31,444 - DEBUG - Verifying vm in vn uve 2026-03-23 04:34:31,444 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-88137413:e2168f2b-ca31-4c57-abcd-f5fd407f422f?flat 2026-03-23 04:34:31,467 - DEBUG - VM uve shows interface as {'tx_vlan': 65535, 'vhostuser_mode': 0, 'rx_vlan': 65535, 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103', 'ip6_active': False, 'ip4_active': True, 'fixed_ip4_list': ['192.168.1.251'], 'is_health_check_active': True, 'vn_uuid': '3bd95fd3-bf08-4424-9610-ec574d9aa238', 'active': True, 'l2_active': True, 'admin_state': True, 'vm_uuid': '6ad7964d-b8c4-449b-ac63-05c4a9545bff', 'uuid': 'e2168f2b-ca31-4c57-abcd-f5fd407f422f', 'ip6_address': '::', 'vm_name': 'ctest-vn1_vm2-66507978', 'ip_address': '192.168.1.251', 'label': 30, '__T': 1774240441605904, 'mac_address': '02:e2:16:8f:2b:ca', 'port_mirror_enabled': False, 'gateway': '192.168.1.254'} 2026-03-23 04:34:31,467 - DEBUG - VM uve shows ip address as ['192.168.1.251', '::'] 2026-03-23 04:34:31,467 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 2026-03-23 04:34:31,467 - INFO - Verifying through opserver in 10.0.0.27 2026-03-23 04:34:31,467 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103?flat 2026-03-23 04:34:31,500 - DEBUG - expected vm list 6ad7964d-b8c4-449b-ac63-05c4a9545bff 2026-03-23 04:34:31,500 - DEBUG - Extracted vm list ['6ad7964d-b8c4-449b-ac63-05c4a9545bff'] 2026-03-23 04:34:31,500 - INFO - VM 6ad7964d-b8c4-449b-ac63-05c4a9545bff is present in default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 2026-03-23 04:34:31,500 - DEBUG - Verifying vm in vrouter uve 2026-03-23 04:34:31,500 - DEBUG - Getting info from collector 10.0.0.23.. 2026-03-23 04:34:31,500 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/6ad7964d-b8c4-449b-ac63-05c4a9545bff?flat 2026-03-23 04:34:31,516 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-5514-2'] 2026-03-23 04:34:31,516 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-2:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:34:31,583 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-2?flat 2026-03-23 04:34:31,613 - DEBUG - VM 6ad7964d-b8c4-449b-ac63-05c4a9545bff is present in vrouter an-jenkins-deploy-platform-ansible-os-5514-2 uve 2026-03-23 04:34:31,613 - DEBUG - Validated that VM ctest-vn1_vm2-66507978 is in Vrouter an-jenkins-deploy-platform-ansible-os-5514-2 UVE 2026-03-23 04:34:31,613 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2026-03-23 04:34:31,624 - DEBUG - Expected tap interface of VM uuid 6ad7964d-b8c4-449b-ac63-05c4a9545bff is default-domain:ctest-TestBasicVMVN0-88137413:e2168f2b-ca31-4c57-abcd-f5fd407f422f 2026-03-23 04:34:31,624 - DEBUG - Expected VN of VM uuid 6ad7964d-b8c4-449b-ac63-05c4a9545bff is default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 2026-03-23 04:34:31,624 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-2:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:34:31,686 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-2?flat 2026-03-23 04:34:31,713 - DEBUG - VM 6ad7964d-b8c4-449b-ac63-05c4a9545bff is present in vrouter an-jenkins-deploy-platform-ansible-os-5514-2 uve 2026-03-23 04:34:31,713 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-88137413:e2168f2b-ca31-4c57-abcd-f5fd407f422f of vm 6ad7964d-b8c4-449b-ac63-05c4a9545bff is present in vrouter an-jenkins-deploy-platform-ansible-os-5514-2 uve 2026-03-23 04:34:31,714 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 of vm 6ad7964d-b8c4-449b-ac63-05c4a9545bff is present in vrouter an-jenkins-deploy-platform-ansible-os-5514-2 uve 2026-03-23 04:34:31,714 - DEBUG - Verifying the 6ad7964d-b8c4-449b-ac63-05c4a9545bff virtual network link through opserver 10.0.0.27 2026-03-23 04:34:31,714 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-03-23 04:34:31,730 - DEBUG - VM link and name as {'name': '6ad7964d-b8c4-449b-ac63-05c4a9545bff', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/6ad7964d-b8c4-449b-ac63-05c4a9545bff?flat'} 2026-03-23 04:34:31,730 - DEBUG - Verifying the 6ad7964d-b8c4-449b-ac63-05c4a9545bff virtual network link through opserver 10.0.0.50 2026-03-23 04:34:31,730 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2026-03-23 04:34:31,807 - DEBUG - VM link and name as {'name': '6ad7964d-b8c4-449b-ac63-05c4a9545bff', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-machine/6ad7964d-b8c4-449b-ac63-05c4a9545bff?flat'} 2026-03-23 04:34:31,807 - DEBUG - Verifying the 6ad7964d-b8c4-449b-ac63-05c4a9545bff virtual network link through opserver 10.0.0.23 2026-03-23 04:34:31,807 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-03-23 04:34:31,820 - DEBUG - VM link and name as {'name': '6ad7964d-b8c4-449b-ac63-05c4a9545bff', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/6ad7964d-b8c4-449b-ac63-05c4a9545bff?flat'} 2026-03-23 04:34:31,820 - INFO - VM ctest-vn1_vm2-66507978 validations in Opserver passed 2026-03-23 04:34:33,363 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-5514-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-5514-1) 2026-03-23 04:34:33,423 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2026-03-23 04:34:38,541 - DEBUG - VM is in ACTIVE state now 2026-03-23 04:34:38,541 - INFO - VM name : ctest-vn2_vm2-91518615 2026-03-23 04:34:38,662 - DEBUG - VM ctest-vn2_vm2-91518615 ID is d6aa0672-5952-4c6a-86ef-49496e016e7b 2026-03-23 04:34:38,788 - DEBUG - VM is in ACTIVE state now 2026-03-23 04:34:38,788 - DEBUG - VM ctest-vn2_vm2-91518615 launched on Node an-jenkins-deploy-platform-ansible-os-5514-1 2026-03-23 04:34:38,906 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/d6aa0672-5952-4c6a-86ef-49496e016e7b 2026-03-23 04:34:38,917 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/d6aa0672-5952-4c6a-86ef-49496e016e7b 2026-03-23 04:34:38,929 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/f28ff904-e721-42a6-a74f-5e7a574bee5d 2026-03-23 04:34:38,942 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/d6aa0672-5952-4c6a-86ef-49496e016e7b 2026-03-23 04:34:38,953 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/f28ff904-e721-42a6-a74f-5e7a574bee5d 2026-03-23 04:34:38,967 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/26fbbfbe-2dd9-492c-9d3f-9caf548b2a71 2026-03-23 04:34:38,979 - DEBUG - Verifying in api server 10.0.0.23 2026-03-23 04:34:38,979 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/d6aa0672-5952-4c6a-86ef-49496e016e7b 2026-03-23 04:34:38,988 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/f28ff904-e721-42a6-a74f-5e7a574bee5d 2026-03-23 04:34:38,999 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/26fbbfbe-2dd9-492c-9d3f-9caf548b2a71 2026-03-23 04:34:39,008 - INFO - VM ctest-vn2_vm2-91518615 verfication in all API Servers passed 2026-03-23 04:34:39,033 - DEBUG - VM ctest-vn2_vm2-91518615 Tap interface: {'index': '4', 'name': 'tapf28ff904-e7', 'uuid': 'f28ff904-e721-42a6-a74f-5e7a574bee5d', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019:ctest-vn2-98616019', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '16', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019', 'vm_uuid': 'd6aa0672-5952-4c6a-86ef-49496e016e7b', 'vm_name': 'ctest-vn2_vm2-91518615', 'ip_addr': '192.168.2.251', 'mac_addr': '02:f2:8f:f9:04:e7', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '131', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-88137413:f28ff904-e721-42a6-a74f-5e7a574bee5d', 'sg_uuid_list': ['42966352-7ed2-4211-baec-acac6f7d89b2'], 'static_route_list': None, 'vm_project_uuid': 'c818fea2-bb8c-4dc4-8c5c-a693e3a97287', 'admin_state': 'Enabled', 'flow_key_idx': '32', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-03-23 04:34:39,033 - DEBUG - Agent 10.0.0.249 vrf name: default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019:ctest-vn2-98616019 2026-03-23 04:34:39,041 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019:ctest-vn2-98616019', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019', 'table_label': '-1', 'vxlan_id': '16', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.192:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2026-03-23 04:34:39,132 - DEBUG - VM ctest-vn2_vm2-91518615 : Tap interface tapf28ff904-e7 is set to Active 2026-03-23 04:34:39,133 - DEBUG - Tap interface tapf28ff904-e7 detail : {'index': '4', 'name': 'tapf28ff904-e7', 'uuid': 'f28ff904-e721-42a6-a74f-5e7a574bee5d', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019:ctest-vn2-98616019', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '16', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019', 'vm_uuid': 'd6aa0672-5952-4c6a-86ef-49496e016e7b', 'vm_name': 'ctest-vn2_vm2-91518615', 'ip_addr': '192.168.2.251', 'mac_addr': '02:f2:8f:f9:04:e7', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '131', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-88137413:f28ff904-e721-42a6-a74f-5e7a574bee5d', 'sg_uuid_list': ['42966352-7ed2-4211-baec-acac6f7d89b2'], 'static_route_list': None, 'vm_project_uuid': 'c818fea2-bb8c-4dc4-8c5c-a693e3a97287', 'admin_state': 'Enabled', 'flow_key_idx': '32', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-03-23 04:34:39,256 - DEBUG - Starting Layer 2 verification in Agent 2026-03-23 04:34:39,271 - DEBUG - Layer 2 path is seen for VM MAC 02:f2:8f:f9:04:e7 in agent 10.0.0.249 2026-03-23 04:34:39,272 - DEBUG - Active layer 2 route in agent is present for VMI tapf28ff904-e7 2026-03-23 04:34:39,272 - DEBUG - L2 label(34) matches bw route table and interface table 2026-03-23 04:34:39,280 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 is found in Agent of node 10.0.0.249 2026-03-23 04:34:39,418 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 is consistent in agent 10.0.0.249 2026-03-23 04:34:39,418 - DEBUG - Route for VM IP ['192.168.2.251'] is consistent in agent 10.0.0.249 2026-03-23 04:34:39,419 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 verification for VM ctest-vn2_vm2-91518615 in Agent 10.0.0.249 passed 2026-03-23 04:34:39,534 - DEBUG - Starting all layer 2 verification in agent 10.0.0.249 2026-03-23 04:34:39,552 - DEBUG - Route for VM MAC 02:f2:8f:f9:04:e7 is consistent in agent 10.0.0.249 2026-03-23 04:34:42,752 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1004ms') 2026-03-23 04:34:42,752 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-91518615 failed! 2026-03-23 04:34:42,767 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 is 192.168.2.254 and allocation pool is NOT set 2026-03-23 04:34:46,846 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1016ms') 2026-03-23 04:34:46,846 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-91518615 failed! 2026-03-23 04:34:46,860 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 is 192.168.2.254 and allocation pool is NOT set 2026-03-23 04:34:50,937 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2026-03-23 04:34:50,937 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-91518615 failed! 2026-03-23 04:34:50,954 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 is 192.168.2.254 and allocation pool is NOT set 2026-03-23 04:34:55,040 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2026-03-23 04:34:55,041 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-91518615 failed! 2026-03-23 04:34:55,058 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 is 192.168.2.254 and allocation pool is NOT set 2026-03-23 04:34:59,138 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1012ms') 2026-03-23 04:34:59,138 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-91518615 failed! 2026-03-23 04:34:59,155 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 is 192.168.2.254 and allocation pool is NOT set 2026-03-23 04:35:03,227 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1012ms') 2026-03-23 04:35:03,227 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-91518615 failed! 2026-03-23 04:35:03,248 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 is 192.168.2.254 and allocation pool is NOT set 2026-03-23 04:35:07,331 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms') 2026-03-23 04:35:07,331 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-91518615 failed! 2026-03-23 04:35:07,349 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 is 192.168.2.254 and allocation pool is NOT set 2026-03-23 04:35:11,424 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1006ms') 2026-03-23 04:35:11,425 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-91518615 failed! 2026-03-23 04:35:11,440 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 is 192.168.2.254 and allocation pool is NOT set 2026-03-23 04:35:15,517 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms') 2026-03-23 04:35:15,517 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-91518615 failed! 2026-03-23 04:35:15,540 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 is 192.168.2.254 and allocation pool is NOT set 2026-03-23 04:35:19,617 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms') 2026-03-23 04:35:19,618 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-91518615 failed! 2026-03-23 04:35:19,632 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 is 192.168.2.254 and allocation pool is NOT set 2026-03-23 04:35:21,700 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=14.2 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=4.41 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 4.411/9.284/14.157/4.873 ms') 2026-03-23 04:35:21,700 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-91518615 passed 2026-03-23 04:35:21,700 - INFO - VM ctest-vn2_vm2-91518615 verifications in Compute nodes passed 2026-03-23 04:35:21,956 - DEBUG - Validated VM route 192.168.2.251 in vrouter of 10.0.0.249 2026-03-23 04:35:21,963 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2026-03-23 04:35:21,963 - INFO - Validated routes of VM ctest-vn2_vm2-91518615 in all vrouters 2026-03-23 04:35:22,380 - DEBUG - Starting all layer2 verification in 10.0.0.50 Control Node 2026-03-23 04:35:22,507 - DEBUG - Layer2 route found for VM MAC 02:f2:8f:f9:04:e7 in Control-node 10.0.0.50 2026-03-23 04:35:22,507 - DEBUG - L2 Label for VM ctest-vn2_vm2-91518615 same between Control-node 10.0.0.50 and Agent, Expected: 34, Seen: 34 2026-03-23 04:35:22,507 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2026-03-23 04:35:22,689 - DEBUG - Layer2 route found for VM MAC 02:f2:8f:f9:04:e7 in Control-node 10.0.0.27 2026-03-23 04:35:22,689 - DEBUG - L2 Label for VM ctest-vn2_vm2-91518615 same between Control-node 10.0.0.27 and Agent, Expected: 34, Seen: 34 2026-03-23 04:35:22,689 - INFO - Verification in Control-nodes for VM ctest-vn2_vm2-91518615 passed 2026-03-23 04:35:22,690 - DEBUG - Verifying the vm in opserver 2026-03-23 04:35:22,690 - DEBUG - Verifying in collector 10.0.0.23 ... 2026-03-23 04:35:22,690 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/d6aa0672-5952-4c6a-86ef-49496e016e7b?flat 2026-03-23 04:35:22,707 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-88137413:f28ff904-e721-42a6-a74f-5e7a574bee5d?flat 2026-03-23 04:35:22,730 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-88137413:f28ff904-e721-42a6-a74f-5e7a574bee5d?flat 2026-03-23 04:35:22,758 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/d6aa0672-5952-4c6a-86ef-49496e016e7b?flat 2026-03-23 04:35:22,776 - DEBUG - Verifying vm in vn uve 2026-03-23 04:35:22,776 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-88137413:f28ff904-e721-42a6-a74f-5e7a574bee5d?flat 2026-03-23 04:35:22,801 - DEBUG - VM uve shows interface as {'fixed_ip4_list': ['192.168.2.251'], 'gateway': '192.168.2.254', 'ip6_address': '::', 'vn_uuid': '4d9c728e-7915-4670-9c49-79024f2c614c', 'vhostuser_mode': 0, 'vm_name': 'ctest-vn2_vm2-91518615', 'rx_vlan': 65535, 'ip6_active': False, 'is_health_check_active': True, 'mac_address': '02:f2:8f:f9:04:e7', 'port_mirror_enabled': False, 'label': 30, 'admin_state': True, 'active': True, 'vm_uuid': 'd6aa0672-5952-4c6a-86ef-49496e016e7b', 'ip4_active': True, 'ip_address': '192.168.2.251', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019', 'tx_vlan': 65535, '__T': 1774240481881069, 'uuid': 'f28ff904-e721-42a6-a74f-5e7a574bee5d', 'l2_active': True} 2026-03-23 04:35:22,801 - DEBUG - VM uve shows ip address as ['192.168.2.251', '::'] 2026-03-23 04:35:22,801 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 2026-03-23 04:35:22,801 - INFO - Verifying through opserver in 10.0.0.27 2026-03-23 04:35:22,801 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019?flat 2026-03-23 04:35:22,829 - DEBUG - expected vm list d6aa0672-5952-4c6a-86ef-49496e016e7b 2026-03-23 04:35:22,829 - DEBUG - Extracted vm list ['d6aa0672-5952-4c6a-86ef-49496e016e7b'] 2026-03-23 04:35:22,829 - INFO - VM d6aa0672-5952-4c6a-86ef-49496e016e7b is present in default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 2026-03-23 04:35:22,829 - DEBUG - Verifying vm in vrouter uve 2026-03-23 04:35:22,829 - DEBUG - Getting info from collector 10.0.0.23.. 2026-03-23 04:35:22,829 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/d6aa0672-5952-4c6a-86ef-49496e016e7b?flat 2026-03-23 04:35:22,846 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-5514-1'] 2026-03-23 04:35:22,846 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-1:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:35:22,909 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-1?flat 2026-03-23 04:35:22,943 - DEBUG - VM d6aa0672-5952-4c6a-86ef-49496e016e7b is present in vrouter an-jenkins-deploy-platform-ansible-os-5514-1 uve 2026-03-23 04:35:22,943 - DEBUG - Validated that VM ctest-vn2_vm2-91518615 is in Vrouter an-jenkins-deploy-platform-ansible-os-5514-1 UVE 2026-03-23 04:35:22,943 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2026-03-23 04:35:22,952 - DEBUG - Expected tap interface of VM uuid d6aa0672-5952-4c6a-86ef-49496e016e7b is default-domain:ctest-TestBasicVMVN0-88137413:f28ff904-e721-42a6-a74f-5e7a574bee5d 2026-03-23 04:35:22,952 - DEBUG - Expected VN of VM uuid d6aa0672-5952-4c6a-86ef-49496e016e7b is default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 2026-03-23 04:35:22,952 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-1:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:35:23,007 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-1?flat 2026-03-23 04:35:23,126 - DEBUG - VM d6aa0672-5952-4c6a-86ef-49496e016e7b is present in vrouter an-jenkins-deploy-platform-ansible-os-5514-1 uve 2026-03-23 04:35:23,126 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-88137413:f28ff904-e721-42a6-a74f-5e7a574bee5d of vm d6aa0672-5952-4c6a-86ef-49496e016e7b is present in vrouter an-jenkins-deploy-platform-ansible-os-5514-1 uve 2026-03-23 04:35:23,126 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 of vm d6aa0672-5952-4c6a-86ef-49496e016e7b is present in vrouter an-jenkins-deploy-platform-ansible-os-5514-1 uve 2026-03-23 04:35:23,127 - DEBUG - Verifying the d6aa0672-5952-4c6a-86ef-49496e016e7b virtual network link through opserver 10.0.0.27 2026-03-23 04:35:23,127 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-03-23 04:35:23,144 - DEBUG - VM link and name as {'name': 'd6aa0672-5952-4c6a-86ef-49496e016e7b', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/d6aa0672-5952-4c6a-86ef-49496e016e7b?flat'} 2026-03-23 04:35:23,144 - DEBUG - Verifying the d6aa0672-5952-4c6a-86ef-49496e016e7b virtual network link through opserver 10.0.0.50 2026-03-23 04:35:23,144 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2026-03-23 04:35:23,157 - DEBUG - VM link and name as {'name': 'd6aa0672-5952-4c6a-86ef-49496e016e7b', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-machine/d6aa0672-5952-4c6a-86ef-49496e016e7b?flat'} 2026-03-23 04:35:23,158 - DEBUG - Verifying the d6aa0672-5952-4c6a-86ef-49496e016e7b virtual network link through opserver 10.0.0.23 2026-03-23 04:35:23,158 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-03-23 04:35:23,171 - DEBUG - VM link and name as {'name': 'd6aa0672-5952-4c6a-86ef-49496e016e7b', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/d6aa0672-5952-4c6a-86ef-49496e016e7b?flat'} 2026-03-23 04:35:23,171 - INFO - VM ctest-vn2_vm2-91518615 validations in Opserver passed 2026-03-23 04:35:23,171 - INFO - Waiting for VM ctest-vn1_vm2-66507978 to be up.. 2026-03-23 04:35:23,171 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:35:23,171 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/6ad7964d-b8c4-449b-ac63-05c4a9545bff 2026-03-23 04:35:23,184 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/e2168f2b-ca31-4c57-abcd-f5fd407f422f 2026-03-23 04:35:24,349 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=4.85 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=1.06 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 1.060/2.954/4.849/1.894 ms') 2026-03-23 04:35:24,350 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-66507978 passed 2026-03-23 04:35:24,423 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:35:24,424 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-66507978, IP 192.168.1.251, Port 22 2026-03-23 04:35:24,600 - DEBUG - VM ctest-vn1_vm2-66507978 is ready for SSH connections 2026-03-23 04:35:24,600 - INFO - Waiting for VM ctest-vn2_vm2-91518615 to be up.. 2026-03-23 04:35:24,600 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:35:24,600 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/d6aa0672-5952-4c6a-86ef-49496e016e7b 2026-03-23 04:35:24,610 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/f28ff904-e721-42a6-a74f-5e7a574bee5d 2026-03-23 04:35:25,794 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=2.21 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=2.49 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 2.205/2.347/2.490/0.142 ms') 2026-03-23 04:35:25,794 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-91518615 passed 2026-03-23 04:35:25,866 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:35:25,866 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-91518615, IP 192.168.2.251, Port 22 2026-03-23 04:35:25,938 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-03-23 04:35:26,056 - DEBUG - VM ctest-vn2_vm2-91518615 is NOT ready for SSH connections, VM status: ACTIVE 2026-03-23 04:35:31,057 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:35:31,058 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-91518615, IP 192.168.2.251, Port 22 2026-03-23 04:35:31,126 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-03-23 04:35:31,231 - DEBUG - VM ctest-vn2_vm2-91518615 is NOT ready for SSH connections, VM status: ACTIVE 2026-03-23 04:35:36,232 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:35:36,232 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-91518615, IP 192.168.2.251, Port 22 2026-03-23 04:35:37,826 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-03-23 04:35:37,955 - DEBUG - VM ctest-vn2_vm2-91518615 is NOT ready for SSH connections, VM status: ACTIVE 2026-03-23 04:35:42,956 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:35:42,957 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-91518615, IP 192.168.2.251, Port 22 2026-03-23 04:35:43,028 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-03-23 04:35:43,142 - DEBUG - VM ctest-vn2_vm2-91518615 is NOT ready for SSH connections, VM status: ACTIVE 2026-03-23 04:35:48,142 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:35:48,143 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-91518615, IP 192.168.2.251, Port 22 2026-03-23 04:35:48,315 - DEBUG - VM ctest-vn2_vm2-91518615 is ready for SSH connections 2026-03-23 04:35:48,315 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-03-23 04:35:48,315 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.251, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.65, gateway password: c0ntrail123 2026-03-23 04:35:48,315 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.251 2026-03-23 04:36:30,972 - DEBUG - PING 192.168.2.251 (192.168.2.251) 56(84) bytes of data. 64 bytes from 192.168.2.251: icmp_req=1 ttl=63 time=10.6 ms 64 bytes from 192.168.2.251: icmp_req=2 ttl=63 time=1.42 ms 64 bytes from 192.168.2.251: icmp_req=3 ttl=63 time=1.46 ms --- 192.168.2.251 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.425/4.522/10.674/4.350 ms 2026-03-23 04:36:30,972 - INFO - Ping to IP 192.168.2.251 from VM ctest-vn1_vm2-66507978 passed 2026-03-23 04:36:30,972 - INFO - Deleting VM ctest-vn2_vm2-91518615 2026-03-23 04:36:31,168 - DEBUG - Verifying in api server 10.0.0.23 2026-03-23 04:36:31,168 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/d6aa0672-5952-4c6a-86ef-49496e016e7b 2026-03-23 04:36:31,181 - DEBUG - VM ID d6aa0672-5952-4c6a-86ef-49496e016e7b of VM ctest-vn2_vm2-91518615 is still found in API Server 2026-03-23 04:36:33,182 - DEBUG - Verifying in api server 10.0.0.23 2026-03-23 04:36:33,182 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/d6aa0672-5952-4c6a-86ef-49496e016e7b 2026-03-23 04:36:33,191 - DEBUG - Response Code: 404 2026-03-23 04:36:33,192 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/d6aa0672-5952-4c6a-86ef-49496e016e7b 2026-03-23 04:36:33,198 - DEBUG - Response Code: 404 2026-03-23 04:36:33,199 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/d6aa0672-5952-4c6a-86ef-49496e016e7b 2026-03-23 04:36:33,207 - DEBUG - Response Code: 404 2026-03-23 04:36:33,207 - INFO - VM ctest-vn2_vm2-91518615 is fully removed in API-Server 2026-03-23 04:36:33,374 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2026-03-23 04:36:33,573 - DEBUG - VM route 192.168.2.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 of compute 10.0.0.249 2026-03-23 04:36:33,579 - DEBUG - Validated that vrouter 10.0.0.249 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 2026-03-23 04:36:33,580 - DEBUG - Validated that vrouter 10.0.0.65 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 2026-03-23 04:36:33,580 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 2026-03-23 04:36:33,580 - INFO - VM ctest-vn2_vm2-91518615 is removed in Compute, and routes are removed in all compute nodes 2026-03-23 04:36:33,810 - INFO - Routes for VM ctest-vn2_vm2-91518615 is removed in all control-nodes 2026-03-23 04:36:33,810 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-1:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:36:33,881 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-1?flat 2026-03-23 04:36:33,926 - DEBUG - VM d6aa0672-5952-4c6a-86ef-49496e016e7b is still present in vrouter an-jenkins-deploy-platform-ansible-os-5514-1 uve 2026-03-23 04:36:33,928 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-03-23 04:36:33,928 - DEBUG - interface for vm d6aa0672-5952-4c6a-86ef-49496e016e7b deleted from vrouter uve ... 2026-03-23 04:36:33,928 - DEBUG - Deleted interface not in error interface list ... 2026-03-23 04:36:36,929 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-1:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:36:36,990 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-1?flat 2026-03-23 04:36:37,024 - DEBUG - VM d6aa0672-5952-4c6a-86ef-49496e016e7b is still present in vrouter an-jenkins-deploy-platform-ansible-os-5514-1 uve 2026-03-23 04:36:37,024 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-03-23 04:36:37,024 - DEBUG - interface for vm d6aa0672-5952-4c6a-86ef-49496e016e7b deleted from vrouter uve ... 2026-03-23 04:36:37,024 - DEBUG - Deleted interface not in error interface list ... 2026-03-23 04:36:40,025 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-1:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:36:40,077 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-1?flat 2026-03-23 04:36:40,111 - DEBUG - VM d6aa0672-5952-4c6a-86ef-49496e016e7b is still present in vrouter an-jenkins-deploy-platform-ansible-os-5514-1 uve 2026-03-23 04:36:40,111 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-03-23 04:36:40,111 - DEBUG - interface for vm d6aa0672-5952-4c6a-86ef-49496e016e7b deleted from vrouter uve ... 2026-03-23 04:36:40,111 - DEBUG - Deleted interface not in error interface list ... 2026-03-23 04:36:43,111 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-1:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:36:43,170 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-1?flat 2026-03-23 04:36:43,205 - DEBUG - VM d6aa0672-5952-4c6a-86ef-49496e016e7b is not present in vrouter an-jenkins-deploy-platform-ansible-os-5514-1 uve 2026-03-23 04:36:43,205 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-03-23 04:36:43,205 - DEBUG - interface for vm d6aa0672-5952-4c6a-86ef-49496e016e7b deleted from vrouter uve ... 2026-03-23 04:36:43,205 - DEBUG - Deleted interface not in error interface list ... 2026-03-23 04:36:43,205 - DEBUG - Validated that VM d6aa0672-5952-4c6a-86ef-49496e016e7b is removed in Vrouter UVE 2026-03-23 04:36:43,205 - DEBUG - Verifying through opserver in 10.0.0.27 2026-03-23 04:36:43,205 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019?flat 2026-03-23 04:36:43,232 - DEBUG - d6aa0672-5952-4c6a-86ef-49496e016e7b BM not in default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn2-98616019 uve 2026-03-23 04:36:43,233 - DEBUG - Validated that VM d6aa0672-5952-4c6a-86ef-49496e016e7b is not present in the VN UVE 2026-03-23 04:36:43,233 - DEBUG - Verifying the d6aa0672-5952-4c6a-86ef-49496e016e7b virtual network link through opserver 10.0.0.27 2026-03-23 04:36:43,233 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-03-23 04:36:43,260 - INFO - d6aa0672-5952-4c6a-86ef-49496e016e7b vm uve deleted from opserver 2026-03-23 04:36:43,260 - DEBUG - Verifying the d6aa0672-5952-4c6a-86ef-49496e016e7b virtual network link through opserver 10.0.0.50 2026-03-23 04:36:43,260 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2026-03-23 04:36:43,276 - INFO - d6aa0672-5952-4c6a-86ef-49496e016e7b vm uve deleted from opserver 2026-03-23 04:36:43,276 - DEBUG - Verifying the d6aa0672-5952-4c6a-86ef-49496e016e7b virtual network link through opserver 10.0.0.23 2026-03-23 04:36:43,276 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-03-23 04:36:43,290 - INFO - d6aa0672-5952-4c6a-86ef-49496e016e7b vm uve deleted from opserver 2026-03-23 04:36:43,290 - INFO - Validated that VM d6aa0672-5952-4c6a-86ef-49496e016e7b is removed from Opserver 2026-03-23 04:36:43,290 - INFO - Deleting VM ctest-vn1_vm2-66507978 2026-03-23 04:36:43,385 - DEBUG - Verifying in api server 10.0.0.23 2026-03-23 04:36:43,385 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/6ad7964d-b8c4-449b-ac63-05c4a9545bff 2026-03-23 04:36:43,395 - DEBUG - VM ID 6ad7964d-b8c4-449b-ac63-05c4a9545bff of VM ctest-vn1_vm2-66507978 is still found in API Server 2026-03-23 04:36:45,396 - DEBUG - Verifying in api server 10.0.0.23 2026-03-23 04:36:45,396 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/6ad7964d-b8c4-449b-ac63-05c4a9545bff 2026-03-23 04:36:45,404 - DEBUG - Response Code: 404 2026-03-23 04:36:45,405 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/6ad7964d-b8c4-449b-ac63-05c4a9545bff 2026-03-23 04:36:45,412 - DEBUG - Response Code: 404 2026-03-23 04:36:45,412 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/6ad7964d-b8c4-449b-ac63-05c4a9545bff 2026-03-23 04:36:45,418 - DEBUG - Response Code: 404 2026-03-23 04:36:45,418 - INFO - VM ctest-vn1_vm2-66507978 is fully removed in API-Server 2026-03-23 04:36:45,589 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2026-03-23 04:36:45,630 - DEBUG - Validated that vrouter 10.0.0.249 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 2026-03-23 04:36:45,784 - DEBUG - VM route 192.168.1.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 of compute 10.0.0.65 2026-03-23 04:36:45,791 - DEBUG - Validated that vrouter 10.0.0.65 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 2026-03-23 04:36:45,791 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 2026-03-23 04:36:45,792 - INFO - VM ctest-vn1_vm2-66507978 is removed in Compute, and routes are removed in all compute nodes 2026-03-23 04:36:46,025 - INFO - Routes for VM ctest-vn1_vm2-66507978 is removed in all control-nodes 2026-03-23 04:36:46,025 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-2:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:36:46,102 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-2?flat 2026-03-23 04:36:46,135 - DEBUG - VM 6ad7964d-b8c4-449b-ac63-05c4a9545bff is still present in vrouter an-jenkins-deploy-platform-ansible-os-5514-2 uve 2026-03-23 04:36:46,135 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-03-23 04:36:46,135 - DEBUG - interface for vm 6ad7964d-b8c4-449b-ac63-05c4a9545bff deleted from vrouter uve ... 2026-03-23 04:36:46,135 - DEBUG - Deleted interface not in error interface list ... 2026-03-23 04:36:49,137 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-2:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:36:49,198 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-2?flat 2026-03-23 04:36:49,227 - DEBUG - VM 6ad7964d-b8c4-449b-ac63-05c4a9545bff is still present in vrouter an-jenkins-deploy-platform-ansible-os-5514-2 uve 2026-03-23 04:36:49,227 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-03-23 04:36:49,227 - DEBUG - interface for vm 6ad7964d-b8c4-449b-ac63-05c4a9545bff deleted from vrouter uve ... 2026-03-23 04:36:49,227 - DEBUG - Deleted interface not in error interface list ... 2026-03-23 04:36:52,228 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-2:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:36:52,295 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-2?flat 2026-03-23 04:36:52,326 - DEBUG - VM 6ad7964d-b8c4-449b-ac63-05c4a9545bff is still present in vrouter an-jenkins-deploy-platform-ansible-os-5514-2 uve 2026-03-23 04:36:52,326 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-03-23 04:36:52,326 - DEBUG - interface for vm 6ad7964d-b8c4-449b-ac63-05c4a9545bff deleted from vrouter uve ... 2026-03-23 04:36:52,326 - DEBUG - Deleted interface not in error interface list ... 2026-03-23 04:36:55,327 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-2:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:36:55,389 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-2?flat 2026-03-23 04:36:55,426 - DEBUG - VM 6ad7964d-b8c4-449b-ac63-05c4a9545bff is still present in vrouter an-jenkins-deploy-platform-ansible-os-5514-2 uve 2026-03-23 04:36:55,426 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-03-23 04:36:55,426 - DEBUG - interface for vm 6ad7964d-b8c4-449b-ac63-05c4a9545bff deleted from vrouter uve ... 2026-03-23 04:36:55,426 - DEBUG - Deleted interface not in error interface list ... 2026-03-23 04:36:58,426 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-2:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:36:58,496 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-2?flat 2026-03-23 04:36:58,534 - DEBUG - VM 6ad7964d-b8c4-449b-ac63-05c4a9545bff is still present in vrouter an-jenkins-deploy-platform-ansible-os-5514-2 uve 2026-03-23 04:36:58,534 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-03-23 04:36:58,534 - DEBUG - interface for vm 6ad7964d-b8c4-449b-ac63-05c4a9545bff deleted from vrouter uve ... 2026-03-23 04:36:58,534 - DEBUG - Deleted interface not in error interface list ... 2026-03-23 04:37:01,535 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5514-2:Compute:contrail-vrouter-agent:0?flat 2026-03-23 04:37:01,618 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5514-2?flat 2026-03-23 04:37:01,654 - DEBUG - VM 6ad7964d-b8c4-449b-ac63-05c4a9545bff is not present in vrouter an-jenkins-deploy-platform-ansible-os-5514-2 uve 2026-03-23 04:37:01,654 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-03-23 04:37:01,654 - DEBUG - interface for vm 6ad7964d-b8c4-449b-ac63-05c4a9545bff deleted from vrouter uve ... 2026-03-23 04:37:01,654 - DEBUG - Deleted interface not in error interface list ... 2026-03-23 04:37:01,654 - DEBUG - Validated that VM 6ad7964d-b8c4-449b-ac63-05c4a9545bff is removed in Vrouter UVE 2026-03-23 04:37:01,654 - DEBUG - Verifying through opserver in 10.0.0.27 2026-03-23 04:37:01,654 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103?flat 2026-03-23 04:37:01,695 - DEBUG - 6ad7964d-b8c4-449b-ac63-05c4a9545bff BM not in default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 uve 2026-03-23 04:37:01,695 - DEBUG - Validated that VM 6ad7964d-b8c4-449b-ac63-05c4a9545bff is not present in the VN UVE 2026-03-23 04:37:01,695 - DEBUG - Verifying the 6ad7964d-b8c4-449b-ac63-05c4a9545bff virtual network link through opserver 10.0.0.27 2026-03-23 04:37:01,695 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-03-23 04:37:01,711 - INFO - 6ad7964d-b8c4-449b-ac63-05c4a9545bff vm uve deleted from opserver 2026-03-23 04:37:01,711 - DEBUG - Verifying the 6ad7964d-b8c4-449b-ac63-05c4a9545bff virtual network link through opserver 10.0.0.50 2026-03-23 04:37:01,711 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2026-03-23 04:37:01,726 - INFO - 6ad7964d-b8c4-449b-ac63-05c4a9545bff vm uve deleted from opserver 2026-03-23 04:37:01,727 - DEBUG - Verifying the 6ad7964d-b8c4-449b-ac63-05c4a9545bff virtual network link through opserver 10.0.0.23 2026-03-23 04:37:01,727 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-03-23 04:37:01,743 - INFO - 6ad7964d-b8c4-449b-ac63-05c4a9545bff vm uve deleted from opserver 2026-03-23 04:37:01,743 - INFO - Validated that VM 6ad7964d-b8c4-449b-ac63-05c4a9545bff is removed from Opserver 2026-03-23 04:37:01,743 - INFO - Deleting VN ctest-vn3-95184347 2026-03-23 04:37:01,884 - INFO - Deleting VM ctest-vn2_vm1-75122358 2026-03-23 04:37:02,034 - INFO - Deleting VM ctest-vn1_vm1-40167126 2026-03-23 04:37:02,321 - INFO - Deleting VN ctest-vn2-98616019 2026-03-23 04:37:02,399 - DEBUG - RefsExistError Delete when resource still referred: ['http://10.0.0.23:8082/virtual-machine-interface/a55f5f87-b72c-4ad4-839d-777c18d186d2'] while deleting VN ctest-vn2-98616019..Will retry 2026-03-23 04:37:04,505 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/11fada6e-e190-4f4a-9dec-871575530518 2026-03-23 04:37:04,514 - DEBUG - Response Code: 404 2026-03-23 04:37:04,515 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-03-23 04:37:04,521 - DEBUG - Requesting: http://10.0.0.27:8082/domain/54adab80-1c58-45eb-8fc6-565ee3803789 2026-03-23 04:37:04,540 - DEBUG - Requesting: http://10.0.0.27:8082/project/c818fea2-bb8c-4dc4-8c5c-a693e3a97287 2026-03-23 04:37:04,661 - INFO - Validated that VN ctest-vn2-98616019 is not found in API Server 2026-03-23 04:37:04,673 - DEBUG - VN ctest-vn2-98616019 is not present in Agent 10.0.0.249 2026-03-23 04:37:04,687 - DEBUG - VN ctest-vn2-98616019 is not present in Agent 10.0.0.65 2026-03-23 04:37:04,688 - INFO - Validated that VN ctest-vn2-98616019 is not in any agent 2026-03-23 04:37:04,724 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn2-98616019 info 2026-03-23 04:37:04,893 - INFO - Deleting VN ctest-vn1-67631103 2026-03-23 04:37:05,036 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/38abe83a-48fc-48ae-ad52-170aca7450e8 2026-03-23 04:37:05,044 - DEBUG - Response Code: 404 2026-03-23 04:37:05,045 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-03-23 04:37:05,057 - DEBUG - Requesting: http://10.0.0.27:8082/domain/54adab80-1c58-45eb-8fc6-565ee3803789 2026-03-23 04:37:05,079 - DEBUG - Requesting: http://10.0.0.27:8082/project/c818fea2-bb8c-4dc4-8c5c-a693e3a97287 2026-03-23 04:37:05,196 - INFO - Validated that VN ctest-vn1-67631103 is not found in API Server 2026-03-23 04:37:05,213 - DEBUG - VN ctest-vn1-67631103 is not present in Agent 10.0.0.249 2026-03-23 04:37:05,228 - DEBUG - VN ctest-vn1-67631103 is not present in Agent 10.0.0.65 2026-03-23 04:37:05,228 - INFO - Validated that VN ctest-vn1-67631103 is not in any agent 2026-03-23 04:37:05,236 - DEBUG - VRF 2 is not seen in agent 10.0.0.65 2026-03-23 04:37:05,244 - DEBUG - Vrouter 10.0.0.65 does not have vrf 2 for VN ctest-vn1-67631103 2026-03-23 04:37:05,244 - INFO - Validated that all vrouters do not have the route table for VN default-domain:ctest-TestBasicVMVN0-88137413:ctest-vn1-67631103 2026-03-23 04:37:05,273 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-67631103 info 2026-03-23 04:37:05,311 - INFO - Deleted policy policy2 2026-03-23 04:37:05,359 - INFO - Deleted policy policy1 2026-03-23 04:37:06,656 - DEBUG - Skipping xmpp flap check
2026-03-23 04:37:06,657 - INFO - END TEST : test_process_restart_in_policy_between_vns : PASSED[0:07:12]
2026-03-23 04:37:06,657 - INFO - -------------------------------------------------------------------------------- 2026-03-23 04:37:06,660 - INFO - ================================================================================ 2026-03-23 04:37:06,660 - INFO - STARTING TEST : test_underlay_broadcast_traffic_handling 2026-03-23 04:37:06,660 - INFO - TEST DESCRIPTION : Test the underlay brocast traffic handling by vrouter. (Bug-1545229). 1. Send broadcast traffic from one compute node. 2. Other compute in same subnet should receive that traffic. 3. Receiving compute should treat this traffic as underlay. 4. Compute should not replicate the packet and send the copy back. Pass criteria: Step 3-4 should pass Maintainer : chhandak@juniper.net 2026-03-23 04:37:07,953 - DEBUG - Skipping xmpp flap check 2026-03-23 04:37:07,954 - INFO - Initial checks done. Running the testcase now 2026-03-23 04:37:07,954 - INFO - 2026-03-23 04:37:08,359 - INFO - Agent interface name: ens6 2026-03-23 04:37:08,442 - DEBUG - Executing command: sudo tcpdump -nni ens6 -U host 10.20.0.255 -w /tmp/ens6_ctest-random-45374477.pcap 2026-03-23 04:37:13,846 - INFO - Pinging broacast address 10.20.0.255 from compute 10.20.0.192 2026-03-23 04:37:28,160 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2026-03-23 04:37:30,312 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-45374477.pcap | grep -c length 2026-03-23 04:37:30,331 - DEBUG - STDOUT: 10 2026-03-23 04:37:30,331 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-45374477.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2026-03-23 04:37:30,332 - INFO - 10 packets are found in tcpdump output as expected 2026-03-23 04:37:30,332 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2026-03-23 04:37:32,376 - INFO - Packet count matched: Compute 10.0.0.65 has receive only 10 packet from source IP 10.20.0.255. No duplicate packet seen 2026-03-23 04:37:33,659 - DEBUG - Skipping xmpp flap check
2026-03-23 04:37:33,659 - INFO - END TEST : test_underlay_broadcast_traffic_handling : PASSED[0:00:27]
2026-03-23 04:37:33,659 - INFO - -------------------------------------------------------------------------------- 2026-03-23 04:37:34,555 - INFO - Deleted project: ctest-TestBasicVMVN0-88137413, ID : c818fea2-bb8c-4dc4-8c5c-a693e3a97287