2026-01-05 04:02:57,823 - INFO - Domain Default found not creating 2026-01-05 04:02:57,994 - INFO - Project ctest-TestBasicVMVN0-57160223 not found, creating it 2026-01-05 04:02:58,589 - INFO - Created Project:ctest-TestBasicVMVN0-57160223, ID : f5158d13-df10-419a-8547-25010c329027 2026-01-05 04:03:00,473 - INFO - ================================================================================ 2026-01-05 04:03:00,473 - INFO - STARTING TEST : test_control_node_switchover 2026-01-05 04:03:00,473 - INFO - TEST DESCRIPTION : Stop the control node and check peering with agent fallback to other control node. 1. Pick one VN from respource pool which has 2 VM's in it 2. Verify ping between VM's 3. Find active control node in cluster by agent inspect 4. Stop control service on active control node 5. Verify agents are connected to new active control-node using xmpp connections 6. Bring back control service on previous active node 7. Verify ping between VM's again after bringing up control serveice Pass criteria: Step 2,5 and 7 should pass 2026-01-05 04:03:01,773 - DEBUG - Skipping xmpp flap check 2026-01-05 04:03:01,773 - INFO - Initial checks done. Running the testcase now 2026-01-05 04:03:01,773 - INFO - 2026-01-05 04:03:02,468 - DEBUG - Response for create_network : {'network': {'id': '69f1f3ea-db67-4d75-9446-77519c7f1e19', 'name': 'ctest-vn1-87060689', 'tenant_id': 'f5158d13df10419a854725010c329027', 'project_id': 'f5158d13df10419a854725010c329027', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestBasicVMVN0-57160223', 'ctest-vn1-87060689'], 'port_security_enabled': True, 'description': ''}} 2026-01-05 04:03:02,734 - DEBUG - Response for create_subnet : {'subnet': {'id': 'a6ca08f9-777d-4c5b-b2ec-efd85426371f', 'name': '', 'tenant_id': 'f5158d13df10419a854725010c329027', 'network_id': '69f1f3ea-db67-4d75-9446-77519c7f1e19', 'ip_version': 4, 'cidr': '192.168.1.0/24', 'allocation_pools': [{'start': '192.168.1.2', 'end': '192.168.1.254'}], 'gateway_ip': '192.168.1.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '192.168.1.2', 'tags': [], 'project_id': 'f5158d13df10419a854725010c329027'}} 2026-01-05 04:03:02,758 - INFO - Created VN ctest-vn1-87060689 2026-01-05 04:03:02,769 - DEBUG - VN ctest-vn1-87060689 UUID is 69f1f3ea-db67-4d75-9446-77519c7f1e19 2026-01-05 04:03:02,845 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-05 04:03:03,115 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-05 04:03:03,144 - DEBUG - Requesting: http://10.0.0.27:8082/domain/182056ec-3382-44e1-b0fd-9a9d794f24a6 2026-01-05 04:03:03,191 - DEBUG - Requesting: http://10.0.0.27:8082/project/f5158d13-df10-419a-8547-25010c329027 2026-01-05 04:03:03,295 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/69f1f3ea-db67-4d75-9446-77519c7f1e19 2026-01-05 04:03:03,314 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/69f1f3ea-db67-4d75-9446-77519c7f1e19 2026-01-05 04:03:03,327 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/d960a6b1-1e3b-462f-b658-300292fe4ef8 2026-01-05 04:03:03,346 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/d960a6b1-1e3b-462f-b658-300292fe4ef8 2026-01-05 04:03:03,358 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/33d7434a-9e6d-4270-a9a8-b88eb5ad6664 2026-01-05 04:03:03,368 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-05 04:03:03,368 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/69f1f3ea-db67-4d75-9446-77519c7f1e19 2026-01-05 04:03:03,382 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/d960a6b1-1e3b-462f-b658-300292fe4ef8 2026-01-05 04:03:03,414 - INFO - Verified VN network id 14 for VN 69f1f3ea-db67-4d75-9446-77519c7f1e19 2026-01-05 04:03:03,415 - INFO - Verifications in API Server for VN ctest-vn1-87060689 passed 2026-01-05 04:03:03,415 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/69f1f3ea-db67-4d75-9446-77519c7f1e19 2026-01-05 04:03:03,434 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/d960a6b1-1e3b-462f-b658-300292fe4ef8 2026-01-05 04:03:03,446 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/d960a6b1-1e3b-462f-b658-300292fe4ef8 2026-01-05 04:03:03,453 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/33d7434a-9e6d-4270-a9a8-b88eb5ad6664 2026-01-05 04:03:03,469 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-57160223', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '7634151033398578549', 'uuid-lslong': '10684358358384713241'}, 'enable': 'true', 'created': '2026-01-05T04:03:02', 'last-modified': '2026-01-05T04:03:02', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.774127'} 2026-01-05 04:03:03,479 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-05 04:03:03,488 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-57160223', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '7634151033398578549', 'uuid-lslong': '10684358358384713241'}, 'enable': 'true', 'created': '2026-01-05T04:03:02', 'last-modified': '2026-01-05T04:03:02', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.798674'} 2026-01-05 04:03:03,507 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-05 04:03:03,519 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-57160223', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '7634151033398578549', 'uuid-lslong': '10684358358384713241'}, 'enable': 'true', 'created': '2026-01-05T04:03:02', 'last-modified': '2026-01-05T04:03:02', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.823916'} 2026-01-05 04:03:03,525 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-05 04:03:03,525 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-87060689 passed 2026-01-05 04:03:03,526 - DEBUG - ====Verifying policy data for ctest-vn1-87060689 in API_Server ====== 2026-01-05 04:03:03,526 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-05 04:03:03,532 - DEBUG - Requesting: http://10.0.0.27:8082/domain/182056ec-3382-44e1-b0fd-9a9d794f24a6 2026-01-05 04:03:03,544 - DEBUG - Requesting: http://10.0.0.27:8082/project/f5158d13-df10-419a-8547-25010c329027 2026-01-05 04:03:03,655 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/69f1f3ea-db67-4d75-9446-77519c7f1e19 2026-01-05 04:03:03,667 - DEBUG - =>VN ctest-vn1-87060689 has no policy to be verified 2026-01-05 04:03:03,667 - DEBUG - Verifying the vn in opserver 2026-01-05 04:03:03,667 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 virtual network link through opserver 10.0.0.27 2026-01-05 04:03:03,668 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2026-01-05 04:03:04,058 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2026-01-05 04:03:04,148 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689?flat'} 2026-01-05 04:03:04,148 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 virtual network link through opserver 10.0.0.33 2026-01-05 04:03:04,148 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2026-01-05 04:03:04,428 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2026-01-05 04:03:04,499 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689?flat'} 2026-01-05 04:03:04,499 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 virtual network link through opserver 10.0.0.23 2026-01-05 04:03:04,499 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2026-01-05 04:03:04,766 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2026-01-05 04:03:04,841 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689?flat'} 2026-01-05 04:03:04,841 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 is found in opserver 2026-01-05 04:03:04,862 - DEBUG - Do not have enough data to verify VN in agent 2026-01-05 04:03:04,872 - DEBUG - VRF ids for VN ctest-vn1-87060689: {} 2026-01-05 04:03:04,933 - DEBUG - Services list from nova: [, , , ] 2026-01-05 04:03:05,934 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4961-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4961-2) 2026-01-05 04:03:07,245 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4961-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4961-1) 2026-01-05 04:03:07,245 - INFO - Waiting for VM ctest-vn1_vm1-32174352 to be up.. 2026-01-05 04:03:07,357 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2026-01-05 04:03:12,443 - DEBUG - VM is in ACTIVE state now 2026-01-05 04:03:12,443 - INFO - VM name : ctest-vn1_vm1-32174352 2026-01-05 04:03:12,520 - DEBUG - VM ctest-vn1_vm1-32174352 ID is df729c3c-12f1-4be5-8cb5-ca401fe2ce75 2026-01-05 04:03:12,541 - DEBUG - VM ctest-vn1_vm1-32174352 launched on Node an-jenkins-deploy-platform-ansible-os-4961-2 2026-01-05 04:03:12,623 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/df729c3c-12f1-4be5-8cb5-ca401fe2ce75 2026-01-05 04:03:12,895 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/df729c3c-12f1-4be5-8cb5-ca401fe2ce75 2026-01-05 04:03:12,930 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/e00c1436-f88f-44ff-a8ab-7ff845d1afe7 2026-01-05 04:03:16,099 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1009ms') 2026-01-05 04:03:16,099 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-32174352 failed! 2026-01-05 04:03:16,114 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 is 192.168.1.1 and allocation pool is NOT set 2026-01-05 04:03:20,198 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2026-01-05 04:03:20,198 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-32174352 failed! 2026-01-05 04:03:20,213 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 is 192.168.1.1 and allocation pool is NOT set 2026-01-05 04:03:24,298 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2026-01-05 04:03:24,299 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-32174352 failed! 2026-01-05 04:03:24,312 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 is 192.168.1.1 and allocation pool is NOT set 2026-01-05 04:03:28,394 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2026-01-05 04:03:28,394 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-32174352 failed! 2026-01-05 04:03:28,408 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 is 192.168.1.1 and allocation pool is NOT set 2026-01-05 04:03:32,490 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2026-01-05 04:03:32,490 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-32174352 failed! 2026-01-05 04:03:32,505 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 is 192.168.1.1 and allocation pool is NOT set 2026-01-05 04:03:36,592 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2026-01-05 04:03:36,592 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-32174352 failed! 2026-01-05 04:03:36,609 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 is 192.168.1.1 and allocation pool is NOT set 2026-01-05 04:03:40,685 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1009ms') 2026-01-05 04:03:40,685 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-32174352 failed! 2026-01-05 04:03:40,701 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 is 192.168.1.1 and allocation pool is NOT set 2026-01-05 04:03:44,780 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1007ms') 2026-01-05 04:03:44,780 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-32174352 failed! 2026-01-05 04:03:44,796 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 is 192.168.1.1 and allocation pool is NOT set 2026-01-05 04:03:46,863 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=4.88 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=3.39 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 3.393/4.138/4.883/0.745 ms') 2026-01-05 04:03:46,864 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-32174352 passed 2026-01-05 04:03:46,935 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:03:46,935 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-32174352, IP 192.168.1.3, Port 22 2026-01-05 04:03:47,003 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-01-05 04:03:47,143 - DEBUG - VM ctest-vn1_vm1-32174352 is NOT ready for SSH connections, VM status: ACTIVE 2026-01-05 04:03:52,144 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:03:52,144 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-32174352, IP 192.168.1.3, Port 22 2026-01-05 04:03:52,211 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-01-05 04:03:52,298 - DEBUG - VM ctest-vn1_vm1-32174352 is NOT ready for SSH connections, VM status: ACTIVE 2026-01-05 04:03:57,299 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:03:57,299 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-32174352, IP 192.168.1.3, Port 22 2026-01-05 04:03:57,368 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-01-05 04:03:57,470 - DEBUG - VM ctest-vn1_vm1-32174352 is NOT ready for SSH connections, VM status: ACTIVE 2026-01-05 04:04:02,471 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:04:02,471 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-32174352, IP 192.168.1.3, Port 22 2026-01-05 04:04:02,539 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-01-05 04:04:02,618 - DEBUG - VM ctest-vn1_vm1-32174352 is NOT ready for SSH connections, VM status: ACTIVE 2026-01-05 04:04:07,618 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:04:07,618 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-32174352, IP 192.168.1.3, Port 22 2026-01-05 04:04:07,800 - DEBUG - VM ctest-vn1_vm1-32174352 is ready for SSH connections 2026-01-05 04:04:07,800 - INFO - Waiting for VM ctest-vn1_vm2-12382718 to be up.. 2026-01-05 04:04:07,888 - DEBUG - VM is in ACTIVE state now 2026-01-05 04:04:07,888 - INFO - VM name : ctest-vn1_vm2-12382718 2026-01-05 04:04:07,972 - DEBUG - VM ctest-vn1_vm2-12382718 ID is d4b768b0-8eb4-4217-9bda-f6bccc70d87c 2026-01-05 04:04:07,972 - DEBUG - VM ctest-vn1_vm2-12382718 launched on Node an-jenkins-deploy-platform-ansible-os-4961-1 2026-01-05 04:04:08,054 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/d4b768b0-8eb4-4217-9bda-f6bccc70d87c 2026-01-05 04:04:08,065 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/663be07d-7403-4f73-a9e4-ea59a3aaab60 2026-01-05 04:04:09,248 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=5.68 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=1.41 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 1.407/3.542/5.677/2.135 ms') 2026-01-05 04:04:09,248 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm2-12382718 passed 2026-01-05 04:04:09,319 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:04:09,319 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-12382718, IP 192.168.1.4, Port 22 2026-01-05 04:04:09,479 - DEBUG - VM ctest-vn1_vm2-12382718 is ready for SSH connections 2026-01-05 04:04:09,480 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:04:09,480 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.56, gateway password: c0ntrail123 2026-01-05 04:04:09,480 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4 2026-01-05 04:04:30,278 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data. 64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=10.3 ms 64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=2.44 ms 64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.32 ms --- 192.168.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.322/4.707/10.351/4.017 ms 2026-01-05 04:04:30,278 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-32174352 passed 2026-01-05 04:04:30,278 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:04:30,278 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2026-01-05 04:04:30,278 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3 2026-01-05 04:05:12,139 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. 64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=8.51 ms 64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.21 ms 64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.14 ms --- 192.168.1.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2005ms rtt min/avg/max/mdev = 1.146/3.624/8.512/3.456 ms 2026-01-05 04:05:12,139 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-12382718 passed 2026-01-05 04:05:12,145 - INFO - Active control node from the Agent 10.0.0.56 is 10.0.0.33 2026-01-05 04:05:12,145 - INFO - Stoping the Control service in 10.0.0.33 2026-01-05 04:05:12,542 - INFO - 10.0.0.33 2026-01-05 04:05:12,542 - INFO - control:inactive 2026-01-05 04:05:12,542 - INFO - Contrail services control are down on nodes 10.0.0.33 2026-01-05 04:05:17,550 - INFO - Active control node from the Agent 10.0.0.56 is 10.0.0.23 2026-01-05 04:05:17,550 - INFO - Starting the Control service in 10.0.0.33 2026-01-05 04:05:17,883 - INFO - 10.0.0.33 2026-01-05 04:05:17,886 - INFO - control:initializing 2026-01-05 04:05:17,886 - DEBUG - defaultdict(, {'10.0.0.33': {'control': {'status': 'initializing', 'description': None}}}) 2026-01-05 04:05:17,887 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-01-05 04:05:22,988 - INFO - 10.0.0.33 2026-01-05 04:05:22,994 - INFO - control:active 2026-01-05 04:05:22,994 - INFO - Contrail services ['control'] are up on nodes 10.0.0.33 2026-01-05 04:05:28,131 - DEBUG - VM is in ACTIVE state now 2026-01-05 04:05:28,131 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:05:28,131 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/df729c3c-12f1-4be5-8cb5-ca401fe2ce75 2026-01-05 04:05:28,140 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/e00c1436-f88f-44ff-a8ab-7ff845d1afe7 2026-01-05 04:05:28,149 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/df729c3c-12f1-4be5-8cb5-ca401fe2ce75 2026-01-05 04:05:28,158 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/e00c1436-f88f-44ff-a8ab-7ff845d1afe7 2026-01-05 04:05:28,169 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/502d1611-8093-4745-9873-e6cdaab0cf17 2026-01-05 04:05:28,181 - DEBUG - Verifying in api server 10.0.0.23 2026-01-05 04:05:28,181 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/df729c3c-12f1-4be5-8cb5-ca401fe2ce75 2026-01-05 04:05:28,188 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/e00c1436-f88f-44ff-a8ab-7ff845d1afe7 2026-01-05 04:05:28,198 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/502d1611-8093-4745-9873-e6cdaab0cf17 2026-01-05 04:05:28,208 - INFO - VM ctest-vn1_vm1-32174352 verfication in all API Servers passed 2026-01-05 04:05:28,229 - DEBUG - VM ctest-vn1_vm1-32174352 Tap interface: {'index': '3', 'name': 'tape00c1436-f8', 'uuid': 'e00c1436-f88f-44ff-a8ab-7ff845d1afe7', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689:ctest-vn1-87060689', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689', 'vm_uuid': 'df729c3c-12f1-4be5-8cb5-ca401fe2ce75', 'vm_name': 'ctest-vn1_vm1-32174352', 'ip_addr': '192.168.1.3', 'mac_addr': '02:e0:0c:14:36:f8', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '161', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-57160223:e00c1436-f88f-44ff-a8ab-7ff845d1afe7', 'sg_uuid_list': ['5479f490-6e0d-425c-ab38-c03dcacd392b'], 'static_route_list': None, 'vm_project_uuid': 'f5158d13-df10-419a-8547-25010c329027', 'admin_state': 'Enabled', 'flow_key_idx': '32', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-01-05 04:05:28,229 - DEBUG - Agent 10.0.0.56 vrf name: default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689:ctest-vn1-87060689 2026-01-05 04:05:28,236 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689:ctest-vn1-87060689', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689', 'table_label': '-1', 'vxlan_id': '14', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.192:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2026-01-05 04:05:28,319 - DEBUG - VM ctest-vn1_vm1-32174352 : Tap interface tape00c1436-f8 is set to Active 2026-01-05 04:05:28,319 - DEBUG - Tap interface tape00c1436-f8 detail : {'index': '3', 'name': 'tape00c1436-f8', 'uuid': 'e00c1436-f88f-44ff-a8ab-7ff845d1afe7', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689:ctest-vn1-87060689', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689', 'vm_uuid': 'df729c3c-12f1-4be5-8cb5-ca401fe2ce75', 'vm_name': 'ctest-vn1_vm1-32174352', 'ip_addr': '192.168.1.3', 'mac_addr': '02:e0:0c:14:36:f8', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '161', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-57160223:e00c1436-f88f-44ff-a8ab-7ff845d1afe7', 'sg_uuid_list': ['5479f490-6e0d-425c-ab38-c03dcacd392b'], 'static_route_list': None, 'vm_project_uuid': 'f5158d13-df10-419a-8547-25010c329027', 'admin_state': 'Enabled', 'flow_key_idx': '32', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-01-05 04:05:28,427 - DEBUG - Starting Layer 2 verification in Agent 2026-01-05 04:05:28,438 - DEBUG - Layer 2 path is seen for VM MAC 02:e0:0c:14:36:f8 in agent 10.0.0.56 2026-01-05 04:05:28,439 - DEBUG - Active layer 2 route in agent is present for VMI tape00c1436-f8 2026-01-05 04:05:28,439 - DEBUG - L2 label(29) matches bw route table and interface table 2026-01-05 04:05:28,446 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 is found in Agent of node 10.0.0.57 2026-01-05 04:05:28,576 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 is consistent in agent 10.0.0.57 2026-01-05 04:05:28,576 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.57 2026-01-05 04:05:28,576 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 verification for VM ctest-vn1_vm1-32174352 in Agent 10.0.0.57 passed 2026-01-05 04:05:28,695 - DEBUG - Starting all layer 2 verification in agent 10.0.0.57 2026-01-05 04:05:28,706 - DEBUG - Route for VM MAC 02:e0:0c:14:36:f8 is consistent in agent 10.0.0.57 2026-01-05 04:05:28,712 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 is found in Agent of node 10.0.0.56 2026-01-05 04:05:28,843 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 is consistent in agent 10.0.0.56 2026-01-05 04:05:28,843 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.56 2026-01-05 04:05:28,843 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 verification for VM ctest-vn1_vm1-32174352 in Agent 10.0.0.56 passed 2026-01-05 04:05:28,960 - DEBUG - Starting all layer 2 verification in agent 10.0.0.56 2026-01-05 04:05:28,971 - DEBUG - Route for VM MAC 02:e0:0c:14:36:f8 is consistent in agent 10.0.0.56 2026-01-05 04:05:28,971 - INFO - VM ctest-vn1_vm1-32174352 verifications in Compute nodes passed 2026-01-05 04:05:29,205 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.57 2026-01-05 04:05:29,211 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.192, 25; Got 10.20.0.192,25 2026-01-05 04:05:29,211 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.57 2026-01-05 04:05:29,317 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.56 2026-01-05 04:05:29,323 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2026-01-05 04:05:29,323 - INFO - Validated routes of VM ctest-vn1_vm1-32174352 in all vrouters 2026-01-05 04:05:29,836 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2026-01-05 04:05:29,965 - DEBUG - Layer2 route found for VM MAC 02:e0:0c:14:36:f8 in Control-node 10.0.0.23 2026-01-05 04:05:29,965 - DEBUG - L2 Label for VM ctest-vn1_vm1-32174352 same between Control-node 10.0.0.23 and Agent, Expected: 29, Seen: 29 2026-01-05 04:05:29,965 - DEBUG - Starting all layer2 verification in 10.0.0.33 Control Node 2026-01-05 04:05:30,094 - DEBUG - Layer2 route found for VM MAC 02:e0:0c:14:36:f8 in Control-node 10.0.0.33 2026-01-05 04:05:30,094 - DEBUG - L2 Label for VM ctest-vn1_vm1-32174352 same between Control-node 10.0.0.33 and Agent, Expected: 29, Seen: 29 2026-01-05 04:05:30,094 - INFO - Verification in Control-nodes for VM ctest-vn1_vm1-32174352 passed 2026-01-05 04:05:30,094 - DEBUG - Verifying the vm in opserver 2026-01-05 04:05:30,094 - DEBUG - Verifying in collector 10.0.0.23 ... 2026-01-05 04:05:30,094 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/df729c3c-12f1-4be5-8cb5-ca401fe2ce75?flat 2026-01-05 04:05:30,112 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-57160223:e00c1436-f88f-44ff-a8ab-7ff845d1afe7?flat 2026-01-05 04:05:30,210 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-57160223:e00c1436-f88f-44ff-a8ab-7ff845d1afe7?flat 2026-01-05 04:05:30,237 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/df729c3c-12f1-4be5-8cb5-ca401fe2ce75?flat 2026-01-05 04:05:30,304 - DEBUG - Verifying vm in vn uve 2026-01-05 04:05:30,304 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-57160223:e00c1436-f88f-44ff-a8ab-7ff845d1afe7?flat 2026-01-05 04:05:30,388 - DEBUG - VM uve shows interface as {'mac_address': '02:e0:0c:14:36:f8', 'vn_uuid': '69f1f3ea-db67-4d75-9446-77519c7f1e19', 'vhostuser_mode': 0, 'admin_state': True, 'rx_vlan': 65535, 'port_mirror_enabled': False, 'active': True, 'tx_vlan': 65535, 'ip6_active': False, 'ip_address': '192.168.1.3', 'vm_uuid': 'df729c3c-12f1-4be5-8cb5-ca401fe2ce75', 'fixed_ip4_list': ['192.168.1.3'], 'l2_active': True, 'uuid': 'e00c1436-f88f-44ff-a8ab-7ff845d1afe7', 'label': 25, 'is_health_check_active': True, 'gateway': '192.168.1.1', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689', 'vm_name': 'ctest-vn1_vm1-32174352', '__T': 1767585928020019, 'ip6_address': '::', 'ip4_active': True} 2026-01-05 04:05:30,388 - DEBUG - VM uve shows ip address as ['192.168.1.3', '::'] 2026-01-05 04:05:30,388 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 2026-01-05 04:05:30,388 - INFO - Verifying through opserver in 10.0.0.27 2026-01-05 04:05:30,388 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689?flat 2026-01-05 04:05:30,418 - DEBUG - expected vm list df729c3c-12f1-4be5-8cb5-ca401fe2ce75 2026-01-05 04:05:30,418 - DEBUG - Extracted vm list ['df729c3c-12f1-4be5-8cb5-ca401fe2ce75'] 2026-01-05 04:05:30,418 - INFO - VM df729c3c-12f1-4be5-8cb5-ca401fe2ce75 is present in default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 2026-01-05 04:05:30,418 - DEBUG - Verifying vm in vrouter uve 2026-01-05 04:05:30,418 - DEBUG - Getting info from collector 10.0.0.23.. 2026-01-05 04:05:30,418 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/df729c3c-12f1-4be5-8cb5-ca401fe2ce75?flat 2026-01-05 04:05:30,496 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4961-2'] 2026-01-05 04:05:30,496 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4961-2:Compute:contrail-vrouter-agent:0?flat 2026-01-05 04:05:30,534 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4961-2?flat 2026-01-05 04:05:30,569 - DEBUG - VM df729c3c-12f1-4be5-8cb5-ca401fe2ce75 is present in vrouter an-jenkins-deploy-platform-ansible-os-4961-2 uve 2026-01-05 04:05:30,569 - DEBUG - Validated that VM ctest-vn1_vm1-32174352 is in Vrouter an-jenkins-deploy-platform-ansible-os-4961-2 UVE 2026-01-05 04:05:30,569 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2026-01-05 04:05:30,577 - DEBUG - Expected tap interface of VM uuid df729c3c-12f1-4be5-8cb5-ca401fe2ce75 is default-domain:ctest-TestBasicVMVN0-57160223:e00c1436-f88f-44ff-a8ab-7ff845d1afe7 2026-01-05 04:05:30,577 - DEBUG - Expected VN of VM uuid df729c3c-12f1-4be5-8cb5-ca401fe2ce75 is default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 2026-01-05 04:05:30,577 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4961-2:Compute:contrail-vrouter-agent:0?flat 2026-01-05 04:05:30,612 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4961-2?flat 2026-01-05 04:05:30,646 - DEBUG - VM df729c3c-12f1-4be5-8cb5-ca401fe2ce75 is present in vrouter an-jenkins-deploy-platform-ansible-os-4961-2 uve 2026-01-05 04:05:30,646 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-57160223:e00c1436-f88f-44ff-a8ab-7ff845d1afe7 of vm df729c3c-12f1-4be5-8cb5-ca401fe2ce75 is present in vrouter an-jenkins-deploy-platform-ansible-os-4961-2 uve 2026-01-05 04:05:30,646 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 of vm df729c3c-12f1-4be5-8cb5-ca401fe2ce75 is present in vrouter an-jenkins-deploy-platform-ansible-os-4961-2 uve 2026-01-05 04:05:30,646 - DEBUG - Verifying the df729c3c-12f1-4be5-8cb5-ca401fe2ce75 virtual network link through opserver 10.0.0.27 2026-01-05 04:05:30,646 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-01-05 04:05:30,659 - DEBUG - VM link and name as {'name': 'df729c3c-12f1-4be5-8cb5-ca401fe2ce75', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/df729c3c-12f1-4be5-8cb5-ca401fe2ce75?flat'} 2026-01-05 04:05:30,659 - DEBUG - Verifying the df729c3c-12f1-4be5-8cb5-ca401fe2ce75 virtual network link through opserver 10.0.0.33 2026-01-05 04:05:30,659 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2026-01-05 04:05:30,723 - DEBUG - VM link and name as {'name': 'df729c3c-12f1-4be5-8cb5-ca401fe2ce75', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-machine/df729c3c-12f1-4be5-8cb5-ca401fe2ce75?flat'} 2026-01-05 04:05:30,723 - DEBUG - Verifying the df729c3c-12f1-4be5-8cb5-ca401fe2ce75 virtual network link through opserver 10.0.0.23 2026-01-05 04:05:30,723 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-01-05 04:05:30,736 - DEBUG - VM link and name as {'name': 'df729c3c-12f1-4be5-8cb5-ca401fe2ce75', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/df729c3c-12f1-4be5-8cb5-ca401fe2ce75?flat'} 2026-01-05 04:05:30,736 - INFO - VM ctest-vn1_vm1-32174352 validations in Opserver passed 2026-01-05 04:05:30,825 - DEBUG - VM is in ACTIVE state now 2026-01-05 04:05:30,825 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:05:30,825 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/d4b768b0-8eb4-4217-9bda-f6bccc70d87c 2026-01-05 04:05:30,833 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/663be07d-7403-4f73-a9e4-ea59a3aaab60 2026-01-05 04:05:30,844 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/d4b768b0-8eb4-4217-9bda-f6bccc70d87c 2026-01-05 04:05:30,851 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/663be07d-7403-4f73-a9e4-ea59a3aaab60 2026-01-05 04:05:30,862 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/0826dc42-19ed-4bf9-859d-60885a96670e 2026-01-05 04:05:30,882 - DEBUG - Verifying in api server 10.0.0.23 2026-01-05 04:05:30,882 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/d4b768b0-8eb4-4217-9bda-f6bccc70d87c 2026-01-05 04:05:30,891 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/663be07d-7403-4f73-a9e4-ea59a3aaab60 2026-01-05 04:05:30,902 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/0826dc42-19ed-4bf9-859d-60885a96670e 2026-01-05 04:05:30,920 - INFO - VM ctest-vn1_vm2-12382718 verfication in all API Servers passed 2026-01-05 04:05:30,946 - DEBUG - VM ctest-vn1_vm2-12382718 Tap interface: {'index': '3', 'name': 'tap663be07d-74', 'uuid': '663be07d-7403-4f73-a9e4-ea59a3aaab60', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689:ctest-vn1-87060689', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689', 'vm_uuid': 'd4b768b0-8eb4-4217-9bda-f6bccc70d87c', 'vm_name': 'ctest-vn1_vm2-12382718', 'ip_addr': '192.168.1.4', 'mac_addr': '02:66:3b:e0:7d:74', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '127', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-57160223:663be07d-7403-4f73-a9e4-ea59a3aaab60', 'sg_uuid_list': ['5479f490-6e0d-425c-ab38-c03dcacd392b'], 'static_route_list': None, 'vm_project_uuid': 'f5158d13-df10-419a-8547-25010c329027', 'admin_state': 'Enabled', 'flow_key_idx': '26', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-01-05 04:05:30,946 - DEBUG - Agent 10.0.0.57 vrf name: default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689:ctest-vn1-87060689 2026-01-05 04:05:30,952 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689:ctest-vn1-87060689', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689', 'table_label': '-1', 'vxlan_id': '14', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.18:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2026-01-05 04:05:31,029 - DEBUG - VM ctest-vn1_vm2-12382718 : Tap interface tap663be07d-74 is set to Active 2026-01-05 04:05:31,029 - DEBUG - Tap interface tap663be07d-74 detail : {'index': '3', 'name': 'tap663be07d-74', 'uuid': '663be07d-7403-4f73-a9e4-ea59a3aaab60', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689:ctest-vn1-87060689', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689', 'vm_uuid': 'd4b768b0-8eb4-4217-9bda-f6bccc70d87c', 'vm_name': 'ctest-vn1_vm2-12382718', 'ip_addr': '192.168.1.4', 'mac_addr': '02:66:3b:e0:7d:74', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '127', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-57160223:663be07d-7403-4f73-a9e4-ea59a3aaab60', 'sg_uuid_list': ['5479f490-6e0d-425c-ab38-c03dcacd392b'], 'static_route_list': None, 'vm_project_uuid': 'f5158d13-df10-419a-8547-25010c329027', 'admin_state': 'Enabled', 'flow_key_idx': '26', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-01-05 04:05:31,145 - DEBUG - Starting Layer 2 verification in Agent 2026-01-05 04:05:31,157 - DEBUG - Layer 2 path is seen for VM MAC 02:66:3b:e0:7d:74 in agent 10.0.0.57 2026-01-05 04:05:31,158 - DEBUG - Active layer 2 route in agent is present for VMI tap663be07d-74 2026-01-05 04:05:31,158 - DEBUG - L2 label(29) matches bw route table and interface table 2026-01-05 04:05:31,163 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 is found in Agent of node 10.0.0.57 2026-01-05 04:05:31,290 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 is consistent in agent 10.0.0.57 2026-01-05 04:05:31,291 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.57 2026-01-05 04:05:31,291 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 verification for VM ctest-vn1_vm2-12382718 in Agent 10.0.0.57 passed 2026-01-05 04:05:31,406 - DEBUG - Starting all layer 2 verification in agent 10.0.0.57 2026-01-05 04:05:31,417 - DEBUG - Route for VM MAC 02:66:3b:e0:7d:74 is consistent in agent 10.0.0.57 2026-01-05 04:05:31,422 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 is found in Agent of node 10.0.0.56 2026-01-05 04:05:31,557 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 is consistent in agent 10.0.0.56 2026-01-05 04:05:31,557 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.56 2026-01-05 04:05:31,557 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 verification for VM ctest-vn1_vm2-12382718 in Agent 10.0.0.56 passed 2026-01-05 04:05:31,686 - DEBUG - Starting all layer 2 verification in agent 10.0.0.56 2026-01-05 04:05:31,697 - DEBUG - Route for VM MAC 02:66:3b:e0:7d:74 is consistent in agent 10.0.0.56 2026-01-05 04:05:31,697 - INFO - VM ctest-vn1_vm2-12382718 verifications in Compute nodes passed 2026-01-05 04:05:31,935 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.57 2026-01-05 04:05:31,940 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2026-01-05 04:05:32,045 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.56 2026-01-05 04:05:32,050 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.18, 24; Got 10.20.0.18,24 2026-01-05 04:05:32,051 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.56 2026-01-05 04:05:32,051 - INFO - Validated routes of VM ctest-vn1_vm2-12382718 in all vrouters 2026-01-05 04:05:32,582 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2026-01-05 04:05:32,708 - DEBUG - Layer2 route found for VM MAC 02:66:3b:e0:7d:74 in Control-node 10.0.0.23 2026-01-05 04:05:32,708 - DEBUG - L2 Label for VM ctest-vn1_vm2-12382718 same between Control-node 10.0.0.23 and Agent, Expected: 29, Seen: 29 2026-01-05 04:05:32,708 - DEBUG - Starting all layer2 verification in 10.0.0.33 Control Node 2026-01-05 04:05:32,847 - DEBUG - Layer2 route found for VM MAC 02:66:3b:e0:7d:74 in Control-node 10.0.0.33 2026-01-05 04:05:32,847 - DEBUG - L2 Label for VM ctest-vn1_vm2-12382718 same between Control-node 10.0.0.33 and Agent, Expected: 29, Seen: 29 2026-01-05 04:05:32,847 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-12382718 passed 2026-01-05 04:05:32,847 - DEBUG - Verifying the vm in opserver 2026-01-05 04:05:32,848 - DEBUG - Verifying in collector 10.0.0.23 ... 2026-01-05 04:05:32,848 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/d4b768b0-8eb4-4217-9bda-f6bccc70d87c?flat 2026-01-05 04:05:32,864 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-57160223:663be07d-7403-4f73-a9e4-ea59a3aaab60?flat 2026-01-05 04:05:32,887 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-57160223:663be07d-7403-4f73-a9e4-ea59a3aaab60?flat 2026-01-05 04:05:32,905 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/d4b768b0-8eb4-4217-9bda-f6bccc70d87c?flat 2026-01-05 04:05:32,920 - DEBUG - Verifying vm in vn uve 2026-01-05 04:05:32,920 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-57160223:663be07d-7403-4f73-a9e4-ea59a3aaab60?flat 2026-01-05 04:05:32,940 - DEBUG - VM uve shows interface as {'vhostuser_mode': 0, 'gateway': '192.168.1.1', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689', 'ip6_active': False, 'ip4_active': True, 'active': True, 'ip6_address': '::', 'mac_address': '02:66:3b:e0:7d:74', 'label': 24, 'vm_uuid': 'd4b768b0-8eb4-4217-9bda-f6bccc70d87c', 'port_mirror_enabled': False, 'vn_uuid': '69f1f3ea-db67-4d75-9446-77519c7f1e19', 'tx_vlan': 65535, 'l2_active': True, 'rx_vlan': 65535, 'admin_state': True, '__T': 1767585799598233, 'fixed_ip4_list': ['192.168.1.4'], 'is_health_check_active': True, 'ip_address': '192.168.1.4', 'vm_name': 'ctest-vn1_vm2-12382718', 'uuid': '663be07d-7403-4f73-a9e4-ea59a3aaab60'} 2026-01-05 04:05:32,940 - DEBUG - VM uve shows ip address as ['192.168.1.4', '::'] 2026-01-05 04:05:32,940 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 2026-01-05 04:05:32,940 - INFO - Verifying through opserver in 10.0.0.27 2026-01-05 04:05:32,941 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689?flat 2026-01-05 04:05:32,966 - DEBUG - expected vm list d4b768b0-8eb4-4217-9bda-f6bccc70d87c 2026-01-05 04:05:32,966 - DEBUG - Extracted vm list ['d4b768b0-8eb4-4217-9bda-f6bccc70d87c'] 2026-01-05 04:05:32,966 - INFO - VM d4b768b0-8eb4-4217-9bda-f6bccc70d87c is present in default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 2026-01-05 04:05:32,966 - DEBUG - Verifying vm in vrouter uve 2026-01-05 04:05:32,966 - DEBUG - Getting info from collector 10.0.0.23.. 2026-01-05 04:05:32,966 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/d4b768b0-8eb4-4217-9bda-f6bccc70d87c?flat 2026-01-05 04:05:32,983 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4961-1'] 2026-01-05 04:05:32,984 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4961-1:Compute:contrail-vrouter-agent:0?flat 2026-01-05 04:05:33,028 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4961-1?flat 2026-01-05 04:05:33,058 - DEBUG - VM d4b768b0-8eb4-4217-9bda-f6bccc70d87c is present in vrouter an-jenkins-deploy-platform-ansible-os-4961-1 uve 2026-01-05 04:05:33,058 - DEBUG - Validated that VM ctest-vn1_vm2-12382718 is in Vrouter an-jenkins-deploy-platform-ansible-os-4961-1 UVE 2026-01-05 04:05:33,058 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2026-01-05 04:05:33,065 - DEBUG - Expected tap interface of VM uuid d4b768b0-8eb4-4217-9bda-f6bccc70d87c is default-domain:ctest-TestBasicVMVN0-57160223:663be07d-7403-4f73-a9e4-ea59a3aaab60 2026-01-05 04:05:33,065 - DEBUG - Expected VN of VM uuid d4b768b0-8eb4-4217-9bda-f6bccc70d87c is default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 2026-01-05 04:05:33,065 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4961-1:Compute:contrail-vrouter-agent:0?flat 2026-01-05 04:05:33,116 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4961-1?flat 2026-01-05 04:05:33,194 - DEBUG - VM d4b768b0-8eb4-4217-9bda-f6bccc70d87c is present in vrouter an-jenkins-deploy-platform-ansible-os-4961-1 uve 2026-01-05 04:05:33,194 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-57160223:663be07d-7403-4f73-a9e4-ea59a3aaab60 of vm d4b768b0-8eb4-4217-9bda-f6bccc70d87c is present in vrouter an-jenkins-deploy-platform-ansible-os-4961-1 uve 2026-01-05 04:05:33,194 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 of vm d4b768b0-8eb4-4217-9bda-f6bccc70d87c is present in vrouter an-jenkins-deploy-platform-ansible-os-4961-1 uve 2026-01-05 04:05:33,194 - DEBUG - Verifying the d4b768b0-8eb4-4217-9bda-f6bccc70d87c virtual network link through opserver 10.0.0.27 2026-01-05 04:05:33,194 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-01-05 04:05:33,206 - DEBUG - VM link and name as {'name': 'd4b768b0-8eb4-4217-9bda-f6bccc70d87c', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/d4b768b0-8eb4-4217-9bda-f6bccc70d87c?flat'} 2026-01-05 04:05:33,206 - DEBUG - Verifying the d4b768b0-8eb4-4217-9bda-f6bccc70d87c virtual network link through opserver 10.0.0.33 2026-01-05 04:05:33,206 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2026-01-05 04:05:33,219 - DEBUG - VM link and name as {'name': 'd4b768b0-8eb4-4217-9bda-f6bccc70d87c', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-machine/d4b768b0-8eb4-4217-9bda-f6bccc70d87c?flat'} 2026-01-05 04:05:33,219 - DEBUG - Verifying the d4b768b0-8eb4-4217-9bda-f6bccc70d87c virtual network link through opserver 10.0.0.23 2026-01-05 04:05:33,219 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-01-05 04:05:33,231 - DEBUG - VM link and name as {'name': 'd4b768b0-8eb4-4217-9bda-f6bccc70d87c', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/d4b768b0-8eb4-4217-9bda-f6bccc70d87c?flat'} 2026-01-05 04:05:33,232 - INFO - VM ctest-vn1_vm2-12382718 validations in Opserver passed 2026-01-05 04:05:33,232 - INFO - Checking the ping between the VM again 2026-01-05 04:05:33,232 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:05:33,232 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.56, gateway password: c0ntrail123 2026-01-05 04:05:33,232 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4 2026-01-05 04:05:35,754 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data. 64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=6.38 ms 64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.23 ms 64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.05 ms --- 192.168.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.056/2.891/6.388/2.473 ms 2026-01-05 04:05:35,754 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-32174352 passed 2026-01-05 04:05:35,754 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:05:35,754 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2026-01-05 04:05:35,754 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3 2026-01-05 04:05:38,257 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. 64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=1.15 ms 64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.02 ms 64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.20 ms --- 192.168.1.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.025/1.127/1.200/0.079 ms 2026-01-05 04:05:38,258 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-12382718 passed 2026-01-05 04:05:38,258 - INFO - Deleting VM ctest-vn1_vm2-12382718 2026-01-05 04:05:38,336 - DEBUG - Verifying in api server 10.0.0.23 2026-01-05 04:05:38,336 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/d4b768b0-8eb4-4217-9bda-f6bccc70d87c 2026-01-05 04:05:38,344 - DEBUG - VM ID d4b768b0-8eb4-4217-9bda-f6bccc70d87c of VM ctest-vn1_vm2-12382718 is still found in API Server 2026-01-05 04:05:40,345 - DEBUG - Verifying in api server 10.0.0.23 2026-01-05 04:05:40,345 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/d4b768b0-8eb4-4217-9bda-f6bccc70d87c 2026-01-05 04:05:40,352 - DEBUG - Response Code: 404 2026-01-05 04:05:40,352 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/d4b768b0-8eb4-4217-9bda-f6bccc70d87c 2026-01-05 04:05:40,358 - DEBUG - Response Code: 404 2026-01-05 04:05:40,358 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/d4b768b0-8eb4-4217-9bda-f6bccc70d87c 2026-01-05 04:05:40,363 - DEBUG - Response Code: 404 2026-01-05 04:05:40,363 - INFO - VM ctest-vn1_vm2-12382718 is fully removed in API-Server 2026-01-05 04:05:40,513 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2026-01-05 04:05:40,550 - DEBUG - Validated that vrouter 10.0.0.57 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 2026-01-05 04:05:40,663 - DEBUG - VM route 192.168.1.4 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 of compute 10.0.0.56 2026-01-05 04:05:40,669 - DEBUG - Validated that vrouter 10.0.0.56 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 2026-01-05 04:05:40,669 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 2026-01-05 04:05:40,669 - INFO - VM ctest-vn1_vm2-12382718 is removed in Compute, and routes are removed in all compute nodes 2026-01-05 04:05:40,919 - INFO - Routes for VM ctest-vn1_vm2-12382718 is removed in all control-nodes 2026-01-05 04:05:40,919 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4961-1:Compute:contrail-vrouter-agent:0?flat 2026-01-05 04:05:40,965 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4961-1?flat 2026-01-05 04:05:40,996 - DEBUG - VM d4b768b0-8eb4-4217-9bda-f6bccc70d87c is still present in vrouter an-jenkins-deploy-platform-ansible-os-4961-1 uve 2026-01-05 04:05:40,997 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-05 04:05:40,997 - DEBUG - interface for vm d4b768b0-8eb4-4217-9bda-f6bccc70d87c deleted from vrouter uve ... 2026-01-05 04:05:40,997 - DEBUG - Deleted interface not in error interface list ... 2026-01-05 04:05:43,998 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4961-1:Compute:contrail-vrouter-agent:0?flat 2026-01-05 04:05:44,049 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4961-1?flat 2026-01-05 04:05:44,088 - DEBUG - VM d4b768b0-8eb4-4217-9bda-f6bccc70d87c is still present in vrouter an-jenkins-deploy-platform-ansible-os-4961-1 uve 2026-01-05 04:05:44,088 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-05 04:05:44,088 - DEBUG - interface for vm d4b768b0-8eb4-4217-9bda-f6bccc70d87c deleted from vrouter uve ... 2026-01-05 04:05:44,088 - DEBUG - Deleted interface not in error interface list ... 2026-01-05 04:05:47,089 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4961-1:Compute:contrail-vrouter-agent:0?flat 2026-01-05 04:05:47,137 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4961-1?flat 2026-01-05 04:05:47,170 - DEBUG - VM d4b768b0-8eb4-4217-9bda-f6bccc70d87c is still present in vrouter an-jenkins-deploy-platform-ansible-os-4961-1 uve 2026-01-05 04:05:47,170 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-05 04:05:47,170 - DEBUG - interface for vm d4b768b0-8eb4-4217-9bda-f6bccc70d87c deleted from vrouter uve ... 2026-01-05 04:05:47,170 - DEBUG - Deleted interface not in error interface list ... 2026-01-05 04:05:50,172 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4961-1:Compute:contrail-vrouter-agent:0?flat 2026-01-05 04:05:50,231 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4961-1?flat 2026-01-05 04:05:50,266 - DEBUG - VM d4b768b0-8eb4-4217-9bda-f6bccc70d87c is not present in vrouter an-jenkins-deploy-platform-ansible-os-4961-1 uve 2026-01-05 04:05:50,266 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-05 04:05:50,266 - DEBUG - interface for vm d4b768b0-8eb4-4217-9bda-f6bccc70d87c deleted from vrouter uve ... 2026-01-05 04:05:50,266 - DEBUG - Deleted interface not in error interface list ... 2026-01-05 04:05:50,266 - DEBUG - Validated that VM d4b768b0-8eb4-4217-9bda-f6bccc70d87c is removed in Vrouter UVE 2026-01-05 04:05:50,266 - DEBUG - Verifying through opserver in 10.0.0.27 2026-01-05 04:05:50,267 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689?flat 2026-01-05 04:05:50,297 - DEBUG - d4b768b0-8eb4-4217-9bda-f6bccc70d87c BM not in default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 uve 2026-01-05 04:05:50,297 - DEBUG - Validated that VM d4b768b0-8eb4-4217-9bda-f6bccc70d87c is not present in the VN UVE 2026-01-05 04:05:50,297 - DEBUG - Verifying the d4b768b0-8eb4-4217-9bda-f6bccc70d87c virtual network link through opserver 10.0.0.27 2026-01-05 04:05:50,297 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-01-05 04:05:50,310 - INFO - d4b768b0-8eb4-4217-9bda-f6bccc70d87c vm uve deleted from opserver 2026-01-05 04:05:50,310 - DEBUG - Verifying the d4b768b0-8eb4-4217-9bda-f6bccc70d87c virtual network link through opserver 10.0.0.33 2026-01-05 04:05:50,310 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2026-01-05 04:05:50,324 - INFO - d4b768b0-8eb4-4217-9bda-f6bccc70d87c vm uve deleted from opserver 2026-01-05 04:05:50,324 - DEBUG - Verifying the d4b768b0-8eb4-4217-9bda-f6bccc70d87c virtual network link through opserver 10.0.0.23 2026-01-05 04:05:50,324 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-01-05 04:05:50,336 - INFO - d4b768b0-8eb4-4217-9bda-f6bccc70d87c vm uve deleted from opserver 2026-01-05 04:05:50,336 - INFO - Validated that VM d4b768b0-8eb4-4217-9bda-f6bccc70d87c is removed from Opserver 2026-01-05 04:05:50,337 - INFO - Deleting VM ctest-vn1_vm1-32174352 2026-01-05 04:05:50,412 - DEBUG - Verifying in api server 10.0.0.23 2026-01-05 04:05:50,412 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/df729c3c-12f1-4be5-8cb5-ca401fe2ce75 2026-01-05 04:05:50,422 - DEBUG - VM ID df729c3c-12f1-4be5-8cb5-ca401fe2ce75 of VM ctest-vn1_vm1-32174352 is still found in API Server 2026-01-05 04:05:52,423 - DEBUG - Verifying in api server 10.0.0.23 2026-01-05 04:05:52,423 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/df729c3c-12f1-4be5-8cb5-ca401fe2ce75 2026-01-05 04:05:52,431 - DEBUG - Response Code: 404 2026-01-05 04:05:52,431 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/df729c3c-12f1-4be5-8cb5-ca401fe2ce75 2026-01-05 04:05:52,437 - DEBUG - Response Code: 404 2026-01-05 04:05:52,437 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/df729c3c-12f1-4be5-8cb5-ca401fe2ce75 2026-01-05 04:05:52,443 - DEBUG - Response Code: 404 2026-01-05 04:05:52,443 - INFO - VM ctest-vn1_vm1-32174352 is fully removed in API-Server 2026-01-05 04:05:52,593 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2026-01-05 04:05:52,639 - DEBUG - Validated that vrouter 10.0.0.57 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 2026-01-05 04:05:52,649 - DEBUG - Validated that vrouter 10.0.0.56 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 2026-01-05 04:05:52,649 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 2026-01-05 04:05:52,649 - INFO - VM ctest-vn1_vm1-32174352 is removed in Compute, and routes are removed in all compute nodes 2026-01-05 04:05:52,890 - INFO - Routes for VM ctest-vn1_vm1-32174352 is removed in all control-nodes 2026-01-05 04:05:52,890 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4961-2:Compute:contrail-vrouter-agent:0?flat 2026-01-05 04:05:52,927 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4961-2?flat 2026-01-05 04:05:52,957 - DEBUG - VM df729c3c-12f1-4be5-8cb5-ca401fe2ce75 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4961-2 uve 2026-01-05 04:05:52,957 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-05 04:05:52,957 - DEBUG - interface for vm df729c3c-12f1-4be5-8cb5-ca401fe2ce75 deleted from vrouter uve ... 2026-01-05 04:05:52,957 - DEBUG - Deleted interface not in error interface list ... 2026-01-05 04:05:55,957 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4961-2:Compute:contrail-vrouter-agent:0?flat 2026-01-05 04:05:55,991 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4961-2?flat 2026-01-05 04:05:56,021 - DEBUG - VM df729c3c-12f1-4be5-8cb5-ca401fe2ce75 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4961-2 uve 2026-01-05 04:05:56,021 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-05 04:05:56,021 - DEBUG - interface for vm df729c3c-12f1-4be5-8cb5-ca401fe2ce75 deleted from vrouter uve ... 2026-01-05 04:05:56,021 - DEBUG - Deleted interface not in error interface list ... 2026-01-05 04:05:59,022 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4961-2:Compute:contrail-vrouter-agent:0?flat 2026-01-05 04:05:59,062 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4961-2?flat 2026-01-05 04:05:59,096 - DEBUG - VM df729c3c-12f1-4be5-8cb5-ca401fe2ce75 is not present in vrouter an-jenkins-deploy-platform-ansible-os-4961-2 uve 2026-01-05 04:05:59,096 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-05 04:05:59,096 - DEBUG - interface for vm df729c3c-12f1-4be5-8cb5-ca401fe2ce75 deleted from vrouter uve ... 2026-01-05 04:05:59,096 - DEBUG - Deleted interface not in error interface list ... 2026-01-05 04:05:59,096 - DEBUG - Validated that VM df729c3c-12f1-4be5-8cb5-ca401fe2ce75 is removed in Vrouter UVE 2026-01-05 04:05:59,096 - DEBUG - Verifying through opserver in 10.0.0.27 2026-01-05 04:05:59,097 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689?flat 2026-01-05 04:05:59,113 - DEBUG - df729c3c-12f1-4be5-8cb5-ca401fe2ce75 BM not in default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-87060689 uve 2026-01-05 04:05:59,113 - DEBUG - Validated that VM df729c3c-12f1-4be5-8cb5-ca401fe2ce75 is not present in the VN UVE 2026-01-05 04:05:59,113 - DEBUG - Verifying the df729c3c-12f1-4be5-8cb5-ca401fe2ce75 virtual network link through opserver 10.0.0.27 2026-01-05 04:05:59,113 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-01-05 04:05:59,124 - INFO - df729c3c-12f1-4be5-8cb5-ca401fe2ce75 vm uve deleted from opserver 2026-01-05 04:05:59,124 - DEBUG - Verifying the df729c3c-12f1-4be5-8cb5-ca401fe2ce75 virtual network link through opserver 10.0.0.33 2026-01-05 04:05:59,124 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2026-01-05 04:05:59,137 - INFO - df729c3c-12f1-4be5-8cb5-ca401fe2ce75 vm uve deleted from opserver 2026-01-05 04:05:59,137 - DEBUG - Verifying the df729c3c-12f1-4be5-8cb5-ca401fe2ce75 virtual network link through opserver 10.0.0.23 2026-01-05 04:05:59,137 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-01-05 04:05:59,153 - INFO - df729c3c-12f1-4be5-8cb5-ca401fe2ce75 vm uve deleted from opserver 2026-01-05 04:05:59,153 - INFO - Validated that VM df729c3c-12f1-4be5-8cb5-ca401fe2ce75 is removed from Opserver 2026-01-05 04:05:59,153 - INFO - Deleting VN ctest-vn1-87060689 2026-01-05 04:05:59,319 - DEBUG - Response for deleting network () 2026-01-05 04:05:59,319 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/d960a6b1-1e3b-462f-b658-300292fe4ef8 2026-01-05 04:05:59,326 - DEBUG - Response Code: 404 2026-01-05 04:05:59,326 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-05 04:05:59,333 - DEBUG - Requesting: http://10.0.0.27:8082/domain/182056ec-3382-44e1-b0fd-9a9d794f24a6 2026-01-05 04:05:59,349 - DEBUG - Requesting: http://10.0.0.27:8082/project/f5158d13-df10-419a-8547-25010c329027 2026-01-05 04:05:59,459 - INFO - Validated that VN ctest-vn1-87060689 is not found in API Server 2026-01-05 04:05:59,471 - DEBUG - VN ctest-vn1-87060689 is not present in Agent 10.0.0.57 2026-01-05 04:05:59,484 - DEBUG - VN ctest-vn1-87060689 is not present in Agent 10.0.0.56 2026-01-05 04:05:59,484 - INFO - Validated that VN ctest-vn1-87060689 is not in any agent 2026-01-05 04:05:59,519 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-87060689 info 2026-01-05 04:06:00,797 - DEBUG - Skipping xmpp flap check
2026-01-05 04:06:00,798 - INFO - END TEST : test_control_node_switchover : PASSED[0:03:00]
2026-01-05 04:06:00,798 - INFO - -------------------------------------------------------------------------------- 2026-01-05 04:06:00,804 - INFO - ================================================================================ 2026-01-05 04:06:00,804 - INFO - STARTING TEST : test_process_restart_in_policy_between_vns 2026-01-05 04:06:00,804 - INFO - TEST DESCRIPTION : Test to validate that with policy having rule to check icmp fwding between VMs on different VNs , ping between VMs should pass with process restarts 1. Pick 2 VN's from resource pool which has one VM each 2. Create policy with icmp allow rule between those VN's and bind it networks 3. Ping from one VM to another VM 4. Restart process 'vrouter' and 'control' on setup 5. Ping again between VM's after process restart Pass criteria: Step 2,3,4 and 5 should pass 2026-01-05 04:06:02,085 - DEBUG - Skipping xmpp flap check 2026-01-05 04:06:02,085 - INFO - Initial checks done. Running the testcase now 2026-01-05 04:06:02,085 - INFO - 2026-01-05 04:06:02,092 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2026-01-05 04:06:02,378 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2026-01-05 04:06:02,910 - INFO - Created VN ctest-vn1-82287566, UUID :f704f219-363b-4ecc-bfba-8ab00b5ddf8c 2026-01-05 04:06:03,267 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-05 04:06:03,274 - DEBUG - Requesting: http://10.0.0.27:8082/domain/182056ec-3382-44e1-b0fd-9a9d794f24a6 2026-01-05 04:06:03,287 - DEBUG - Requesting: http://10.0.0.27:8082/project/f5158d13-df10-419a-8547-25010c329027 2026-01-05 04:06:03,428 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f704f219-363b-4ecc-bfba-8ab00b5ddf8c 2026-01-05 04:06:03,444 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f704f219-363b-4ecc-bfba-8ab00b5ddf8c 2026-01-05 04:06:03,456 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/c094f679-de01-43c8-829b-44b6e80b8f2c 2026-01-05 04:06:03,467 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/c094f679-de01-43c8-829b-44b6e80b8f2c 2026-01-05 04:06:03,476 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/da57e858-e917-4811-aeb0-24c6906c14a3 2026-01-05 04:06:03,484 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-05 04:06:03,484 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f704f219-363b-4ecc-bfba-8ab00b5ddf8c 2026-01-05 04:06:03,501 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/c094f679-de01-43c8-829b-44b6e80b8f2c 2026-01-05 04:06:03,520 - INFO - Verified VN network id 14 for VN f704f219-363b-4ecc-bfba-8ab00b5ddf8c 2026-01-05 04:06:03,520 - INFO - Verifications in API Server for VN ctest-vn1-82287566 passed 2026-01-05 04:06:03,520 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f704f219-363b-4ecc-bfba-8ab00b5ddf8c 2026-01-05 04:06:03,531 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/c094f679-de01-43c8-829b-44b6e80b8f2c 2026-01-05 04:06:03,541 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/c094f679-de01-43c8-829b-44b6e80b8f2c 2026-01-05 04:06:03,550 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/da57e858-e917-4811-aeb0-24c6906c14a3 2026-01-05 04:06:03,564 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-57160223', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17799617817373003468', 'uuid-lslong': '13815507295621996428'}, 'enable': 'true', 'created': '2026-01-05T04:06:02', 'last-modified': '2026-01-05T04:06:03', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.504811'} 2026-01-05 04:06:03,573 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-05 04:06:03,581 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-57160223', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17799617817373003468', 'uuid-lslong': '13815507295621996428'}, 'enable': 'true', 'created': '2026-01-05T04:06:02', 'last-modified': '2026-01-05T04:06:03', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.524691'} 2026-01-05 04:06:03,587 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-05 04:06:03,592 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-57160223', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17799617817373003468', 'uuid-lslong': '13815507295621996428'}, 'enable': 'true', 'created': '2026-01-05T04:06:02', 'last-modified': '2026-01-05T04:06:03', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.531820'} 2026-01-05 04:06:03,598 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-05 04:06:03,598 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-82287566 passed 2026-01-05 04:06:03,598 - DEBUG - ====Verifying policy data for ctest-vn1-82287566 in API_Server ====== 2026-01-05 04:06:03,598 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-05 04:06:03,605 - DEBUG - Requesting: http://10.0.0.27:8082/domain/182056ec-3382-44e1-b0fd-9a9d794f24a6 2026-01-05 04:06:03,617 - DEBUG - Requesting: http://10.0.0.27:8082/project/f5158d13-df10-419a-8547-25010c329027 2026-01-05 04:06:03,728 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f704f219-363b-4ecc-bfba-8ab00b5ddf8c 2026-01-05 04:06:03,746 - DEBUG - =>VN ctest-vn1-82287566 has no policy to be verified 2026-01-05 04:06:03,746 - DEBUG - Verifying the vn in opserver 2026-01-05 04:06:03,746 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 virtual network link through opserver 10.0.0.27 2026-01-05 04:06:03,747 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2026-01-05 04:06:03,759 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566?flat'} 2026-01-05 04:06:03,759 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 virtual network link through opserver 10.0.0.33 2026-01-05 04:06:03,759 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2026-01-05 04:06:03,772 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566?flat'} 2026-01-05 04:06:03,773 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 virtual network link through opserver 10.0.0.23 2026-01-05 04:06:03,773 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2026-01-05 04:06:03,786 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566?flat'} 2026-01-05 04:06:03,786 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 is found in opserver 2026-01-05 04:06:03,806 - DEBUG - Do not have enough data to verify VN in agent 2026-01-05 04:06:03,815 - DEBUG - VRF ids for VN ctest-vn1-82287566: {} 2026-01-05 04:06:04,490 - INFO - Created VN ctest-vn2-71838408, UUID :e088200e-19f3-44cc-9ccd-270934d1a1e6 2026-01-05 04:06:04,802 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-05 04:06:04,809 - DEBUG - Requesting: http://10.0.0.27:8082/domain/182056ec-3382-44e1-b0fd-9a9d794f24a6 2026-01-05 04:06:04,826 - DEBUG - Requesting: http://10.0.0.27:8082/project/f5158d13-df10-419a-8547-25010c329027 2026-01-05 04:06:04,923 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/e088200e-19f3-44cc-9ccd-270934d1a1e6 2026-01-05 04:06:04,938 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/e088200e-19f3-44cc-9ccd-270934d1a1e6 2026-01-05 04:06:04,948 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/1159fc4a-f75e-43df-960f-95a3dd4096d8 2026-01-05 04:06:04,959 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/1159fc4a-f75e-43df-960f-95a3dd4096d8 2026-01-05 04:06:04,967 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/c1a4cc14-58c4-49d8-9401-4f9bdf895ae2 2026-01-05 04:06:04,975 - DEBUG - Route Targets: ['target:64512:8000005'] 2026-01-05 04:06:04,975 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/e088200e-19f3-44cc-9ccd-270934d1a1e6 2026-01-05 04:06:04,985 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/1159fc4a-f75e-43df-960f-95a3dd4096d8 2026-01-05 04:06:05,002 - INFO - Verified VN network id 15 for VN e088200e-19f3-44cc-9ccd-270934d1a1e6 2026-01-05 04:06:05,002 - INFO - Verifications in API Server for VN ctest-vn2-71838408 passed 2026-01-05 04:06:05,002 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/e088200e-19f3-44cc-9ccd-270934d1a1e6 2026-01-05 04:06:05,016 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/1159fc4a-f75e-43df-960f-95a3dd4096d8 2026-01-05 04:06:05,023 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/1159fc4a-f75e-43df-960f-95a3dd4096d8 2026-01-05 04:06:05,031 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/c1a4cc14-58c4-49d8-9401-4f9bdf895ae2 2026-01-05 04:06:05,045 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-57160223', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '16179216906265511116', 'uuid-lslong': '11298729960636785126'}, 'enable': 'true', 'created': '2026-01-05T04:06:04', 'last-modified': '2026-01-05T04:06:04', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.407603'} 2026-01-05 04:06:05,051 - DEBUG - Route Targets: ['target:64512:8000005'] 2026-01-05 04:06:05,057 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-57160223', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '16179216906265511116', 'uuid-lslong': '11298729960636785126'}, 'enable': 'true', 'created': '2026-01-05T04:06:04', 'last-modified': '2026-01-05T04:06:04', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.422292'} 2026-01-05 04:06:05,064 - DEBUG - Route Targets: ['target:64512:8000005'] 2026-01-05 04:06:05,069 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-57160223', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '16179216906265511116', 'uuid-lslong': '11298729960636785126'}, 'enable': 'true', 'created': '2026-01-05T04:06:04', 'last-modified': '2026-01-05T04:06:04', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.434261'} 2026-01-05 04:06:05,075 - DEBUG - Route Targets: ['target:64512:8000005'] 2026-01-05 04:06:05,076 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn2-71838408 passed 2026-01-05 04:06:05,076 - DEBUG - ====Verifying policy data for ctest-vn2-71838408 in API_Server ====== 2026-01-05 04:06:05,076 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-05 04:06:05,082 - DEBUG - Requesting: http://10.0.0.27:8082/domain/182056ec-3382-44e1-b0fd-9a9d794f24a6 2026-01-05 04:06:05,093 - DEBUG - Requesting: http://10.0.0.27:8082/project/f5158d13-df10-419a-8547-25010c329027 2026-01-05 04:06:05,187 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/e088200e-19f3-44cc-9ccd-270934d1a1e6 2026-01-05 04:06:05,197 - DEBUG - =>VN ctest-vn2-71838408 has no policy to be verified 2026-01-05 04:06:05,197 - DEBUG - Verifying the vn in opserver 2026-01-05 04:06:05,197 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 virtual network link through opserver 10.0.0.27 2026-01-05 04:06:05,197 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2026-01-05 04:06:05,210 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408?flat'} 2026-01-05 04:06:05,210 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 virtual network link through opserver 10.0.0.33 2026-01-05 04:06:05,210 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2026-01-05 04:06:05,223 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408?flat'} 2026-01-05 04:06:05,223 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 virtual network link through opserver 10.0.0.23 2026-01-05 04:06:05,223 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2026-01-05 04:06:05,236 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408?flat'} 2026-01-05 04:06:05,236 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 is found in opserver 2026-01-05 04:06:05,260 - DEBUG - Do not have enough data to verify VN in agent 2026-01-05 04:06:05,270 - DEBUG - VRF ids for VN ctest-vn2-71838408: {} 2026-01-05 04:06:06,777 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4961-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4961-2) 2026-01-05 04:06:08,085 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4961-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4961-1) 2026-01-05 04:06:08,085 - INFO - Waiting for VM ctest-vn1_vm1-76914702 to be up.. 2026-01-05 04:06:08,188 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2026-01-05 04:06:13,271 - DEBUG - VM is in ACTIVE state now 2026-01-05 04:06:13,271 - INFO - VM name : ctest-vn1_vm1-76914702 2026-01-05 04:06:13,349 - DEBUG - VM ctest-vn1_vm1-76914702 ID is 2fc993fc-7ba9-48c7-b8f6-e17620bafb14 2026-01-05 04:06:13,349 - DEBUG - VM ctest-vn1_vm1-76914702 launched on Node an-jenkins-deploy-platform-ansible-os-4961-2 2026-01-05 04:06:13,426 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/2fc993fc-7ba9-48c7-b8f6-e17620bafb14 2026-01-05 04:06:13,435 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/8f6d8ed8-432d-4edb-b222-4933e504e3e1 2026-01-05 04:06:16,615 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1029ms') 2026-01-05 04:06:16,616 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-76914702 failed! 2026-01-05 04:06:16,629 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 is 192.168.1.254 and allocation pool is NOT set 2026-01-05 04:06:20,714 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1021ms') 2026-01-05 04:06:20,714 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-76914702 failed! 2026-01-05 04:06:20,728 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 is 192.168.1.254 and allocation pool is NOT set 2026-01-05 04:06:24,811 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1020ms') 2026-01-05 04:06:24,812 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-76914702 failed! 2026-01-05 04:06:24,825 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 is 192.168.1.254 and allocation pool is NOT set 2026-01-05 04:06:28,903 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2026-01-05 04:06:28,903 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-76914702 failed! 2026-01-05 04:06:28,918 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 is 192.168.1.254 and allocation pool is NOT set 2026-01-05 04:06:33,003 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2026-01-05 04:06:33,003 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-76914702 failed! 2026-01-05 04:06:33,020 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 is 192.168.1.254 and allocation pool is NOT set 2026-01-05 04:06:37,100 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2026-01-05 04:06:37,100 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-76914702 failed! 2026-01-05 04:06:37,115 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 is 192.168.1.254 and allocation pool is NOT set 2026-01-05 04:06:41,188 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2026-01-05 04:06:41,188 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-76914702 failed! 2026-01-05 04:06:41,201 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 is 192.168.1.254 and allocation pool is NOT set 2026-01-05 04:06:45,291 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2026-01-05 04:06:45,292 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-76914702 failed! 2026-01-05 04:06:45,306 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 is 192.168.1.254 and allocation pool is NOT set 2026-01-05 04:06:49,388 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2026-01-05 04:06:49,388 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-76914702 failed! 2026-01-05 04:06:49,402 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 is 192.168.1.254 and allocation pool is NOT set 2026-01-05 04:06:51,473 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=5.20 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=5.67 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 5.203/5.435/5.668/0.232 ms') 2026-01-05 04:06:51,473 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-76914702 passed 2026-01-05 04:06:51,543 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:06:51,543 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-76914702, IP 192.168.1.252, Port 22 2026-01-05 04:06:51,611 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-01-05 04:06:51,693 - DEBUG - VM ctest-vn1_vm1-76914702 is NOT ready for SSH connections, VM status: ACTIVE 2026-01-05 04:06:56,694 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:06:56,694 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-76914702, IP 192.168.1.252, Port 22 2026-01-05 04:06:56,759 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-01-05 04:06:56,838 - DEBUG - VM ctest-vn1_vm1-76914702 is NOT ready for SSH connections, VM status: ACTIVE 2026-01-05 04:07:01,838 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:07:01,838 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-76914702, IP 192.168.1.252, Port 22 2026-01-05 04:07:01,908 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-01-05 04:07:01,989 - DEBUG - VM ctest-vn1_vm1-76914702 is NOT ready for SSH connections, VM status: ACTIVE 2026-01-05 04:07:06,990 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:07:06,990 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-76914702, IP 192.168.1.252, Port 22 2026-01-05 04:07:07,055 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-01-05 04:07:07,142 - DEBUG - VM ctest-vn1_vm1-76914702 is NOT ready for SSH connections, VM status: ACTIVE 2026-01-05 04:07:12,142 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:07:12,142 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-76914702, IP 192.168.1.252, Port 22 2026-01-05 04:07:12,302 - DEBUG - VM ctest-vn1_vm1-76914702 is ready for SSH connections 2026-01-05 04:07:12,302 - INFO - Waiting for VM ctest-vn2_vm1-00171585 to be up.. 2026-01-05 04:07:12,386 - DEBUG - VM is in ACTIVE state now 2026-01-05 04:07:12,386 - INFO - VM name : ctest-vn2_vm1-00171585 2026-01-05 04:07:12,474 - DEBUG - VM ctest-vn2_vm1-00171585 ID is 6d3db3c8-c3bb-45eb-b68e-0df411a69970 2026-01-05 04:07:12,474 - DEBUG - VM ctest-vn2_vm1-00171585 launched on Node an-jenkins-deploy-platform-ansible-os-4961-1 2026-01-05 04:07:12,555 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/6d3db3c8-c3bb-45eb-b68e-0df411a69970 2026-01-05 04:07:12,566 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/27c64578-bddf-48b4-b2cc-0d511db60dd4 2026-01-05 04:07:13,725 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=4.04 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=1.87 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 1.872/2.954/4.036/1.082 ms') 2026-01-05 04:07:13,726 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn2_vm1-00171585 passed 2026-01-05 04:07:13,794 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:07:13,795 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-00171585, IP 192.168.2.252, Port 22 2026-01-05 04:07:13,966 - DEBUG - VM ctest-vn2_vm1-00171585 is ready for SSH connections 2026-01-05 04:07:13,966 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:07:13,966 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.252, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.56, gateway password: c0ntrail123 2026-01-05 04:07:13,967 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.252 2026-01-05 04:07:33,742 - DEBUG - PING 192.168.2.252 (192.168.2.252) 56(84) bytes of data. 64 bytes from 192.168.2.252: icmp_req=1 ttl=63 time=11.6 ms 64 bytes from 192.168.2.252: icmp_req=2 ttl=63 time=1.35 ms 64 bytes from 192.168.2.252: icmp_req=3 ttl=63 time=1.20 ms --- 192.168.2.252 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.208/4.723/11.611/4.871 ms 2026-01-05 04:07:33,743 - INFO - Ping to IP 192.168.2.252 from VM ctest-vn1_vm1-76914702 passed 2026-01-05 04:07:37,151 - INFO - 10.0.0.57 2026-01-05 04:07:37,155 - INFO - agent:initializing 2026-01-05 04:07:37,155 - DEBUG - defaultdict(, {'10.0.0.57': {'agent': {'status': 'initializing', 'description': None}}}) 2026-01-05 04:07:37,155 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-01-05 04:07:42,254 - INFO - 10.0.0.57 2026-01-05 04:07:42,260 - INFO - agent:active 2026-01-05 04:07:42,261 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.57 2026-01-05 04:07:45,673 - INFO - 10.0.0.56 2026-01-05 04:07:45,677 - INFO - agent:initializing 2026-01-05 04:07:45,677 - DEBUG - defaultdict(, {'10.0.0.56': {'agent': {'status': 'initializing', 'description': None}}}) 2026-01-05 04:07:45,677 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-01-05 04:07:50,778 - INFO - 10.0.0.56 2026-01-05 04:07:50,783 - INFO - agent:active 2026-01-05 04:07:50,784 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.56 2026-01-05 04:07:51,315 - INFO - 10.0.0.27 2026-01-05 04:07:51,319 - INFO - control:initializing 2026-01-05 04:07:51,319 - DEBUG - defaultdict(, {'10.0.0.27': {'control': {'status': 'initializing', 'description': None}}}) 2026-01-05 04:07:51,319 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-01-05 04:07:56,425 - INFO - 10.0.0.27 2026-01-05 04:07:56,430 - INFO - control:active 2026-01-05 04:07:56,431 - INFO - Contrail services ['control'] are up on nodes 10.0.0.27 2026-01-05 04:07:56,944 - INFO - 10.0.0.33 2026-01-05 04:07:56,948 - INFO - control:initializing 2026-01-05 04:07:56,948 - DEBUG - defaultdict(, {'10.0.0.33': {'control': {'status': 'initializing', 'description': None}}}) 2026-01-05 04:07:56,948 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-01-05 04:08:02,048 - INFO - 10.0.0.33 2026-01-05 04:08:02,053 - INFO - control:active 2026-01-05 04:08:02,053 - INFO - Contrail services ['control'] are up on nodes 10.0.0.33 2026-01-05 04:08:02,487 - INFO - 10.0.0.23 2026-01-05 04:08:02,492 - INFO - control:initializing 2026-01-05 04:08:02,492 - DEBUG - defaultdict(, {'10.0.0.23': {'control': {'status': 'initializing', 'description': None}}}) 2026-01-05 04:08:02,492 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-01-05 04:08:07,593 - INFO - 10.0.0.23 2026-01-05 04:08:07,598 - INFO - control:active 2026-01-05 04:08:07,598 - INFO - Contrail services ['control'] are up on nodes 10.0.0.23 2026-01-05 04:08:08,397 - INFO - 10.0.0.27 2026-01-05 04:08:08,400 - INFO - api-server:initializing 2026-01-05 04:08:08,400 - DEBUG - defaultdict(, {'10.0.0.27': {'api-server': {'status': 'initializing', 'description': None}}}) 2026-01-05 04:08:08,400 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-01-05 04:08:13,527 - INFO - 10.0.0.27 2026-01-05 04:08:13,534 - INFO - api-server:initializing 2026-01-05 04:08:13,534 - DEBUG - defaultdict(, {'10.0.0.27': {'api-server': {'status': 'initializing', 'description': None}}}) 2026-01-05 04:08:13,534 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2026-01-05 04:08:18,654 - INFO - 10.0.0.27 2026-01-05 04:08:18,660 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2026-01-05 04:08:18,660 - DEBUG - defaultdict(, {'10.0.0.27': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2026-01-05 04:08:18,660 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2026-01-05 04:08:23,764 - INFO - 10.0.0.27 2026-01-05 04:08:23,768 - INFO - api-server:active 2026-01-05 04:08:23,768 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.27 2026-01-05 04:08:24,448 - INFO - 10.0.0.33 2026-01-05 04:08:24,452 - INFO - api-server:initializing 2026-01-05 04:08:24,452 - DEBUG - defaultdict(, {'10.0.0.33': {'api-server': {'status': 'initializing', 'description': None}}}) 2026-01-05 04:08:24,452 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-01-05 04:08:29,553 - INFO - 10.0.0.33 2026-01-05 04:08:29,556 - INFO - api-server:initializing 2026-01-05 04:08:29,557 - DEBUG - defaultdict(, {'10.0.0.33': {'api-server': {'status': 'initializing', 'description': None}}}) 2026-01-05 04:08:29,557 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2026-01-05 04:08:34,648 - INFO - 10.0.0.33 2026-01-05 04:08:34,653 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2026-01-05 04:08:34,653 - DEBUG - defaultdict(, {'10.0.0.33': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2026-01-05 04:08:34,653 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2026-01-05 04:08:39,754 - INFO - 10.0.0.33 2026-01-05 04:08:39,759 - INFO - api-server:active 2026-01-05 04:08:39,759 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.33 2026-01-05 04:08:40,429 - INFO - 10.0.0.23 2026-01-05 04:08:40,434 - INFO - api-server:initializing 2026-01-05 04:08:40,434 - DEBUG - defaultdict(, {'10.0.0.23': {'api-server': {'status': 'initializing', 'description': None}}}) 2026-01-05 04:08:40,434 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-01-05 04:08:45,532 - INFO - 10.0.0.23 2026-01-05 04:08:45,536 - INFO - api-server:initializing 2026-01-05 04:08:45,537 - DEBUG - defaultdict(, {'10.0.0.23': {'api-server': {'status': 'initializing', 'description': None}}}) 2026-01-05 04:08:45,537 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2026-01-05 04:08:50,638 - INFO - 10.0.0.23 2026-01-05 04:08:50,645 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2026-01-05 04:08:50,645 - DEBUG - defaultdict(, {'10.0.0.23': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2026-01-05 04:08:50,645 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2026-01-05 04:08:55,745 - INFO - 10.0.0.23 2026-01-05 04:08:55,751 - INFO - api-server:active 2026-01-05 04:08:55,751 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.23 2026-01-05 04:08:55,850 - INFO - 10.0.0.27 2026-01-05 04:08:55,850 - INFO - config 2026-01-05 04:08:55,855 - INFO - config-nodemgr:active 2026-01-05 04:08:55,859 - INFO - api-server:active 2026-01-05 04:08:55,863 - INFO - schema:active 2026-01-05 04:08:55,867 - INFO - svc-monitor:active 2026-01-05 04:08:55,870 - INFO - device-manager:active 2026-01-05 04:08:55,871 - INFO - config-database 2026-01-05 04:08:55,871 - INFO - config-cassandra:active 2026-01-05 04:08:55,871 - INFO - config-zookeeper:active 2026-01-05 04:08:55,871 - INFO - config-rabbitmq:active 2026-01-05 04:08:55,871 - INFO - control 2026-01-05 04:08:55,875 - INFO - control-nodemgr:active 2026-01-05 04:08:55,879 - INFO - control:active 2026-01-05 04:08:55,879 - INFO - named:active 2026-01-05 04:08:55,884 - INFO - dns:active 2026-01-05 04:08:55,884 - INFO - analytics 2026-01-05 04:08:55,888 - INFO - analytics-nodemgr:active 2026-01-05 04:08:55,892 - INFO - analytics-api:active 2026-01-05 04:08:55,896 - INFO - collector:active 2026-01-05 04:08:55,896 - INFO - analytics-database 2026-01-05 04:08:55,896 - INFO - analytics-cassandra:active 2026-01-05 04:08:55,901 - INFO - analyticsdb-nodemgr:active 2026-01-05 04:08:55,905 - INFO - query-engine:active 2026-01-05 04:08:55,905 - INFO - webui 2026-01-05 04:08:55,905 - INFO - webui:active 2026-01-05 04:08:55,906 - INFO - webui-middleware:active 2026-01-05 04:08:55,906 - INFO - redis:active 2026-01-05 04:08:55,906 - INFO - analytics_snmp 2026-01-05 04:08:55,910 - INFO - snmp-collector:active 2026-01-05 04:08:55,910 - INFO - snmp-topology:active 2026-01-05 04:08:55,910 - INFO - analytics_alarm 2026-01-05 04:08:55,910 - INFO - alarmgen:active 2026-01-05 04:08:56,007 - INFO - 10.0.0.33 2026-01-05 04:08:56,007 - INFO - config 2026-01-05 04:08:56,014 - INFO - config-nodemgr:active 2026-01-05 04:08:56,018 - INFO - api-server:active 2026-01-05 04:08:56,022 - INFO - schema:backup 2026-01-05 04:08:56,026 - INFO - svc-monitor:backup 2026-01-05 04:08:56,029 - INFO - device-manager:backup 2026-01-05 04:08:56,029 - INFO - config-database 2026-01-05 04:08:56,029 - INFO - config-cassandra:active 2026-01-05 04:08:56,029 - INFO - config-zookeeper:active 2026-01-05 04:08:56,029 - INFO - config-rabbitmq:active 2026-01-05 04:08:56,029 - INFO - control 2026-01-05 04:08:56,034 - INFO - control-nodemgr:active 2026-01-05 04:08:56,040 - INFO - control:active 2026-01-05 04:08:56,040 - INFO - named:active 2026-01-05 04:08:56,045 - INFO - dns:active 2026-01-05 04:08:56,045 - INFO - analytics 2026-01-05 04:08:56,051 - INFO - analytics-nodemgr:active 2026-01-05 04:08:56,055 - INFO - analytics-api:active 2026-01-05 04:08:56,061 - INFO - collector:active 2026-01-05 04:08:56,061 - INFO - analytics-database 2026-01-05 04:08:56,061 - INFO - analytics-cassandra:active 2026-01-05 04:08:56,067 - INFO - analyticsdb-nodemgr:active 2026-01-05 04:08:56,071 - INFO - query-engine:active 2026-01-05 04:08:56,071 - INFO - webui 2026-01-05 04:08:56,071 - INFO - webui:active 2026-01-05 04:08:56,072 - INFO - webui-middleware:active 2026-01-05 04:08:56,072 - INFO - redis:active 2026-01-05 04:08:56,072 - INFO - analytics_snmp 2026-01-05 04:08:56,076 - INFO - snmp-collector:active 2026-01-05 04:08:56,076 - INFO - snmp-topology:active 2026-01-05 04:08:56,076 - INFO - analytics_alarm 2026-01-05 04:08:56,076 - INFO - alarmgen:active 2026-01-05 04:08:56,172 - INFO - 10.0.0.23 2026-01-05 04:08:56,172 - INFO - config 2026-01-05 04:08:56,178 - INFO - config-nodemgr:active 2026-01-05 04:08:56,182 - INFO - api-server:active 2026-01-05 04:08:56,185 - INFO - schema:backup 2026-01-05 04:08:56,188 - INFO - svc-monitor:backup 2026-01-05 04:08:56,192 - INFO - device-manager:backup 2026-01-05 04:08:56,192 - INFO - config-database 2026-01-05 04:08:56,192 - INFO - config-cassandra:active 2026-01-05 04:08:56,192 - INFO - config-zookeeper:active 2026-01-05 04:08:56,192 - INFO - config-rabbitmq:active 2026-01-05 04:08:56,192 - INFO - control 2026-01-05 04:08:56,198 - INFO - control-nodemgr:active 2026-01-05 04:08:56,202 - INFO - control:active 2026-01-05 04:08:56,202 - INFO - named:active 2026-01-05 04:08:56,207 - INFO - dns:active 2026-01-05 04:08:56,208 - INFO - analytics 2026-01-05 04:08:56,213 - INFO - analytics-nodemgr:active 2026-01-05 04:08:56,221 - INFO - analytics-api:active 2026-01-05 04:08:56,227 - INFO - collector:active 2026-01-05 04:08:56,228 - INFO - analytics-database 2026-01-05 04:08:56,228 - INFO - analytics-cassandra:active 2026-01-05 04:08:56,234 - INFO - analyticsdb-nodemgr:active 2026-01-05 04:08:56,242 - INFO - query-engine:active 2026-01-05 04:08:56,242 - INFO - webui 2026-01-05 04:08:56,242 - INFO - webui:active 2026-01-05 04:08:56,242 - INFO - webui-middleware:active 2026-01-05 04:08:56,243 - INFO - redis:active 2026-01-05 04:08:56,243 - INFO - analytics_snmp 2026-01-05 04:08:56,248 - INFO - snmp-collector:active 2026-01-05 04:08:56,248 - INFO - snmp-topology:active 2026-01-05 04:08:56,248 - INFO - analytics_alarm 2026-01-05 04:08:56,248 - INFO - alarmgen:active 2026-01-05 04:08:56,336 - INFO - 10.0.0.57 2026-01-05 04:08:56,336 - INFO - vrouter 2026-01-05 04:08:56,342 - INFO - vrouter-nodemgr:active 2026-01-05 04:08:56,347 - INFO - agent:active 2026-01-05 04:08:56,437 - INFO - 10.0.0.56 2026-01-05 04:08:56,437 - INFO - vrouter 2026-01-05 04:08:56,442 - INFO - vrouter-nodemgr:active 2026-01-05 04:08:56,447 - INFO - agent:active 2026-01-05 04:08:56,447 - INFO - All the contrail services are up on all nodes 2026-01-05 04:08:56,447 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4961-1?flat 2026-01-05 04:08:56,809 - INFO - an-jenkins-deploy-platform-ansible-os-4961-1:contrail-vrouter-agent module connection to ['10.20.0.25:5269'] servers UP 2026-01-05 04:08:56,809 - INFO - an-jenkins-deploy-platform-ansible-os-4961-1 connected to xmpp 10.20.0.25 2026-01-05 04:08:56,810 - INFO - an-jenkins-deploy-platform-ansible-os-4961-1:contrail-vrouter-agent module connection to ['10.20.0.254:5269'] servers UP 2026-01-05 04:08:56,810 - INFO - an-jenkins-deploy-platform-ansible-os-4961-1 connected to xmpp 10.20.0.254 2026-01-05 04:08:56,810 - WARNING - an-jenkins-deploy-platform-ansible-os-4961-1:contrail-vrouter-agent module connection to ['10.20.0.129:5269'] servers NOT UP 2026-01-05 04:08:56,810 - INFO - an-jenkins-deploy-platform-ansible-os-4961-1:contrail-vrouter-agent module connection to ['10.20.0.25:53'] servers UP 2026-01-05 04:08:56,810 - INFO - an-jenkins-deploy-platform-ansible-os-4961-1 connected to dns 10.20.0.25 2026-01-05 04:08:56,810 - INFO - an-jenkins-deploy-platform-ansible-os-4961-1:contrail-vrouter-agent module connection to ['10.20.0.254:53'] servers UP 2026-01-05 04:08:56,810 - INFO - an-jenkins-deploy-platform-ansible-os-4961-1 connected to dns 10.20.0.254 2026-01-05 04:08:56,810 - WARNING - an-jenkins-deploy-platform-ansible-os-4961-1:contrail-vrouter-agent module connection to ['10.20.0.129:53'] servers NOT UP 2026-01-05 04:08:56,811 - WARNING - an-jenkins-deploy-platform-ansible-os-4961-1:contrail-vrouter-agent module connection to ['10.0.0.27:8086'] servers NOT UP 2026-01-05 04:08:56,811 - WARNING - an-jenkins-deploy-platform-ansible-os-4961-1:contrail-vrouter-agent module connection to ['10.0.0.33:8086'] servers NOT UP 2026-01-05 04:08:56,811 - INFO - an-jenkins-deploy-platform-ansible-os-4961-1:contrail-vrouter-agent module connection to ['10.0.0.23:8086'] servers UP 2026-01-05 04:08:56,811 - INFO - an-jenkins-deploy-platform-ansible-os-4961-1 connected to collector 10.0.0.23 2026-01-05 04:08:56,811 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4961-1?flat 2026-01-05 04:08:57,186 - INFO - cn-jenkins-deploy-platform-ansible-os-4961-1:contrail-control module connection to 10.0.0.27:9041 servers UP 2026-01-05 04:08:57,186 - INFO - cn-jenkins-deploy-platform-ansible-os-4961-1:contrail-control module connection to 10.0.0.27:5673 servers UP 2026-01-05 04:08:57,186 - INFO - cn-jenkins-deploy-platform-ansible-os-4961-1:contrail-control module connection to ['10.0.0.27:8086'] servers UP 2026-01-05 04:08:57,187 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4961-2?flat 2026-01-05 04:08:57,567 - INFO - cn-jenkins-deploy-platform-ansible-os-4961-2:contrail-control module connection to 10.0.0.27:9041 servers UP 2026-01-05 04:08:57,567 - INFO - cn-jenkins-deploy-platform-ansible-os-4961-2:contrail-control module connection to 10.0.0.27:5673 servers UP 2026-01-05 04:08:57,567 - INFO - cn-jenkins-deploy-platform-ansible-os-4961-2:contrail-control module connection to ['10.0.0.27:8086'] servers UP 2026-01-05 04:08:57,567 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4961-3?flat 2026-01-05 04:08:57,588 - INFO - cn-jenkins-deploy-platform-ansible-os-4961-3:contrail-control module connection to 10.0.0.27:9041 servers UP 2026-01-05 04:08:57,588 - INFO - cn-jenkins-deploy-platform-ansible-os-4961-3:contrail-control module connection to 10.0.0.27:5673 servers UP 2026-01-05 04:08:57,588 - INFO - cn-jenkins-deploy-platform-ansible-os-4961-3:contrail-control module connection to ['10.0.0.27:8086'] servers UP 2026-01-05 04:08:57,589 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4961-1.?flat 2026-01-05 04:08:57,615 - DEBUG - contrail-api is running 2026-01-05 04:08:57,615 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4961-2.?flat 2026-01-05 04:08:57,638 - DEBUG - contrail-api is running 2026-01-05 04:08:57,638 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4961-3.?flat 2026-01-05 04:08:57,664 - DEBUG - contrail-api is running 2026-01-05 04:08:57,950 - INFO - Created VN ctest-vn3-38592111, UUID :5857264d-cbbb-40e5-9fa2-24eb69d44ba9 2026-01-05 04:08:58,401 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-05 04:08:58,433 - DEBUG - Requesting: http://10.0.0.27:8082/domain/182056ec-3382-44e1-b0fd-9a9d794f24a6 2026-01-05 04:08:58,445 - DEBUG - Requesting: http://10.0.0.27:8082/project/f5158d13-df10-419a-8547-25010c329027 2026-01-05 04:08:58,542 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f704f219-363b-4ecc-bfba-8ab00b5ddf8c 2026-01-05 04:08:58,559 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f704f219-363b-4ecc-bfba-8ab00b5ddf8c 2026-01-05 04:08:58,571 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/c094f679-de01-43c8-829b-44b6e80b8f2c 2026-01-05 04:08:58,581 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/c094f679-de01-43c8-829b-44b6e80b8f2c 2026-01-05 04:08:58,587 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/da57e858-e917-4811-aeb0-24c6906c14a3 2026-01-05 04:08:58,594 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-05 04:08:58,595 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f704f219-363b-4ecc-bfba-8ab00b5ddf8c 2026-01-05 04:08:58,614 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/c094f679-de01-43c8-829b-44b6e80b8f2c 2026-01-05 04:08:58,638 - INFO - Verified VN network id 14 for VN f704f219-363b-4ecc-bfba-8ab00b5ddf8c 2026-01-05 04:08:58,638 - INFO - Verifications in API Server for VN ctest-vn1-82287566 passed 2026-01-05 04:08:58,638 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f704f219-363b-4ecc-bfba-8ab00b5ddf8c 2026-01-05 04:08:58,647 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/c094f679-de01-43c8-829b-44b6e80b8f2c 2026-01-05 04:08:58,654 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/c094f679-de01-43c8-829b-44b6e80b8f2c 2026-01-05 04:08:58,662 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/da57e858-e917-4811-aeb0-24c6906c14a3 2026-01-05 04:08:58,673 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566', 'interests': '1', 'advertised': '1', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-57160223', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17799617817373003468', 'uuid-lslong': '13815507295621996428'}, 'enable': 'true', 'created': '2026-01-05T04:06:02', 'last-modified': '2026-01-05T04:06:04', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:06.917973'} 2026-01-05 04:08:58,678 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-05 04:08:58,684 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-57160223', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17799617817373003468', 'uuid-lslong': '13815507295621996428'}, 'enable': 'true', 'created': '2026-01-05T04:06:02', 'last-modified': '2026-01-05T04:06:04', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:01.366179'} 2026-01-05 04:08:58,694 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-05 04:08:58,699 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-57160223', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17799617817373003468', 'uuid-lslong': '13815507295621996428'}, 'enable': 'true', 'created': '2026-01-05T04:06:02', 'last-modified': '2026-01-05T04:06:04', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:55.858503'} 2026-01-05 04:08:58,706 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-05 04:08:58,706 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-82287566 passed 2026-01-05 04:08:58,706 - DEBUG - ====Verifying policy data for ctest-vn1-82287566 in API_Server ====== 2026-01-05 04:08:58,706 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-05 04:08:58,713 - DEBUG - Requesting: http://10.0.0.27:8082/domain/182056ec-3382-44e1-b0fd-9a9d794f24a6 2026-01-05 04:08:58,724 - DEBUG - Requesting: http://10.0.0.27:8082/project/f5158d13-df10-419a-8547-25010c329027 2026-01-05 04:08:58,810 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f704f219-363b-4ecc-bfba-8ab00b5ddf8c 2026-01-05 04:08:58,822 - DEBUG - ==>Verifying data for policy with id: 9d5eab95-eda7-4028-9bb9-ce42ec083647, fqn: ['default-domain', 'ctest-TestBasicVMVN0-57160223', 'policy1'] 2026-01-05 04:08:58,822 - INFO - VN ctest-vn1-82287566 Policy verification: verify_vn_policy_in_api_server, status: True 2026-01-05 04:08:58,822 - DEBUG - Verifying the vn in opserver 2026-01-05 04:08:58,823 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 virtual network link through opserver 10.0.0.27 2026-01-05 04:08:58,823 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2026-01-05 04:08:58,834 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566?flat'} 2026-01-05 04:08:58,834 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 virtual network link through opserver 10.0.0.33 2026-01-05 04:08:58,834 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2026-01-05 04:08:58,910 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566?flat'} 2026-01-05 04:08:58,910 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 virtual network link through opserver 10.0.0.23 2026-01-05 04:08:58,911 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2026-01-05 04:08:58,994 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566?flat'} 2026-01-05 04:08:58,994 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 is found in opserver 2026-01-05 04:08:59,004 - DEBUG - VRF ids for VN ctest-vn1-82287566: {'10.0.0.56': '2'} 2026-01-05 04:09:00,411 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4961-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4961-2) 2026-01-05 04:09:00,465 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2026-01-05 04:09:05,550 - DEBUG - VM is in ACTIVE state now 2026-01-05 04:09:05,550 - INFO - VM name : ctest-vn1_vm2-04238191 2026-01-05 04:09:05,632 - DEBUG - VM ctest-vn1_vm2-04238191 ID is b0a03ae2-32ce-41b4-be4a-a21deba0ce5d 2026-01-05 04:09:05,712 - DEBUG - VM is in ACTIVE state now 2026-01-05 04:09:05,712 - DEBUG - VM ctest-vn1_vm2-04238191 launched on Node an-jenkins-deploy-platform-ansible-os-4961-2 2026-01-05 04:09:05,789 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/b0a03ae2-32ce-41b4-be4a-a21deba0ce5d 2026-01-05 04:09:05,824 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/b0a03ae2-32ce-41b4-be4a-a21deba0ce5d 2026-01-05 04:09:05,831 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/d31f4df9-8a56-46a3-bcf2-f88dcdcdbd00 2026-01-05 04:09:05,840 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/b0a03ae2-32ce-41b4-be4a-a21deba0ce5d 2026-01-05 04:09:05,847 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/d31f4df9-8a56-46a3-bcf2-f88dcdcdbd00 2026-01-05 04:09:05,857 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/02ee6886-301e-4968-ad4a-ac9eb4c07585 2026-01-05 04:09:05,865 - DEBUG - Verifying in api server 10.0.0.23 2026-01-05 04:09:05,866 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/b0a03ae2-32ce-41b4-be4a-a21deba0ce5d 2026-01-05 04:09:05,872 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/d31f4df9-8a56-46a3-bcf2-f88dcdcdbd00 2026-01-05 04:09:05,883 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/02ee6886-301e-4968-ad4a-ac9eb4c07585 2026-01-05 04:09:05,890 - INFO - VM ctest-vn1_vm2-04238191 verfication in all API Servers passed 2026-01-05 04:09:05,909 - DEBUG - VM ctest-vn1_vm2-04238191 Tap interface: {'index': '4', 'name': 'tapd31f4df9-8a', 'uuid': 'd31f4df9-8a56-46a3-bcf2-f88dcdcdbd00', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566:ctest-vn1-82287566', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566', 'vm_uuid': 'b0a03ae2-32ce-41b4-be4a-a21deba0ce5d', 'vm_name': 'ctest-vn1_vm2-04238191', 'ip_addr': '192.168.1.251', 'mac_addr': '02:d3:1f:4d:f9:8a', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '165', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-57160223:d31f4df9-8a56-46a3-bcf2-f88dcdcdbd00', 'sg_uuid_list': ['5479f490-6e0d-425c-ab38-c03dcacd392b'], 'static_route_list': None, 'vm_project_uuid': 'f5158d13-df10-419a-8547-25010c329027', 'admin_state': 'Enabled', 'flow_key_idx': '28', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-01-05 04:09:05,909 - DEBUG - Agent 10.0.0.56 vrf name: default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566:ctest-vn1-82287566 2026-01-05 04:09:05,915 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566:ctest-vn1-82287566', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566', 'table_label': '-1', 'vxlan_id': '14', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.192:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2026-01-05 04:09:05,995 - DEBUG - VM ctest-vn1_vm2-04238191 : Tap interface tapd31f4df9-8a is set to Active 2026-01-05 04:09:05,995 - DEBUG - Tap interface tapd31f4df9-8a detail : {'index': '4', 'name': 'tapd31f4df9-8a', 'uuid': 'd31f4df9-8a56-46a3-bcf2-f88dcdcdbd00', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566:ctest-vn1-82287566', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566', 'vm_uuid': 'b0a03ae2-32ce-41b4-be4a-a21deba0ce5d', 'vm_name': 'ctest-vn1_vm2-04238191', 'ip_addr': '192.168.1.251', 'mac_addr': '02:d3:1f:4d:f9:8a', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '165', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-57160223:d31f4df9-8a56-46a3-bcf2-f88dcdcdbd00', 'sg_uuid_list': ['5479f490-6e0d-425c-ab38-c03dcacd392b'], 'static_route_list': None, 'vm_project_uuid': 'f5158d13-df10-419a-8547-25010c329027', 'admin_state': 'Enabled', 'flow_key_idx': '28', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-01-05 04:09:06,120 - DEBUG - Starting Layer 2 verification in Agent 2026-01-05 04:09:06,132 - DEBUG - Layer 2 path is seen for VM MAC 02:d3:1f:4d:f9:8a in agent 10.0.0.56 2026-01-05 04:09:06,132 - DEBUG - Active layer 2 route in agent is present for VMI tapd31f4df9-8a 2026-01-05 04:09:06,132 - DEBUG - L2 label(34) matches bw route table and interface table 2026-01-05 04:09:06,145 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 is found in Agent of node 10.0.0.56 2026-01-05 04:09:06,281 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 is consistent in agent 10.0.0.56 2026-01-05 04:09:06,282 - DEBUG - Route for VM IP ['192.168.1.251'] is consistent in agent 10.0.0.56 2026-01-05 04:09:06,282 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 verification for VM ctest-vn1_vm2-04238191 in Agent 10.0.0.56 passed 2026-01-05 04:09:06,395 - DEBUG - Starting all layer 2 verification in agent 10.0.0.56 2026-01-05 04:09:06,410 - DEBUG - Route for VM MAC 02:d3:1f:4d:f9:8a is consistent in agent 10.0.0.56 2026-01-05 04:09:09,611 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2026-01-05 04:09:09,612 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-04238191 failed! 2026-01-05 04:09:09,627 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 is 192.168.1.254 and allocation pool is NOT set 2026-01-05 04:09:13,709 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2026-01-05 04:09:13,709 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-04238191 failed! 2026-01-05 04:09:13,723 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 is 192.168.1.254 and allocation pool is NOT set 2026-01-05 04:09:17,799 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2026-01-05 04:09:17,799 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-04238191 failed! 2026-01-05 04:09:17,812 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 is 192.168.1.254 and allocation pool is NOT set 2026-01-05 04:09:21,902 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1027ms') 2026-01-05 04:09:21,903 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-04238191 failed! 2026-01-05 04:09:21,918 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 is 192.168.1.254 and allocation pool is NOT set 2026-01-05 04:09:25,996 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1008ms') 2026-01-05 04:09:25,996 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-04238191 failed! 2026-01-05 04:09:26,011 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 is 192.168.1.254 and allocation pool is NOT set 2026-01-05 04:09:30,093 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2026-01-05 04:09:30,093 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-04238191 failed! 2026-01-05 04:09:30,107 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 is 192.168.1.254 and allocation pool is NOT set 2026-01-05 04:09:34,186 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2026-01-05 04:09:34,186 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-04238191 failed! 2026-01-05 04:09:34,204 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 is 192.168.1.254 and allocation pool is NOT set 2026-01-05 04:09:38,278 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms') 2026-01-05 04:09:38,278 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-04238191 failed! 2026-01-05 04:09:38,291 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 is 192.168.1.254 and allocation pool is NOT set 2026-01-05 04:09:42,377 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=6.66 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 1 received, 50% packet loss, time 1023ms\r\nrtt min/avg/max/mdev = 6.660/6.660/6.660/0.000 ms') 2026-01-05 04:09:42,377 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-04238191 passed 2026-01-05 04:09:42,377 - INFO - VM ctest-vn1_vm2-04238191 verifications in Compute nodes passed 2026-01-05 04:09:42,586 - DEBUG - Validated VM route 192.168.1.251 in vrouter of 10.0.0.56 2026-01-05 04:09:42,592 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2026-01-05 04:09:42,592 - INFO - Validated routes of VM ctest-vn1_vm2-04238191 in all vrouters 2026-01-05 04:09:43,001 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2026-01-05 04:09:43,122 - DEBUG - Layer2 route found for VM MAC 02:d3:1f:4d:f9:8a in Control-node 10.0.0.23 2026-01-05 04:09:43,123 - DEBUG - L2 Label for VM ctest-vn1_vm2-04238191 same between Control-node 10.0.0.23 and Agent, Expected: 34, Seen: 34 2026-01-05 04:09:43,123 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2026-01-05 04:09:43,246 - DEBUG - Layer2 route found for VM MAC 02:d3:1f:4d:f9:8a in Control-node 10.0.0.27 2026-01-05 04:09:43,246 - DEBUG - L2 Label for VM ctest-vn1_vm2-04238191 same between Control-node 10.0.0.27 and Agent, Expected: 34, Seen: 34 2026-01-05 04:09:43,246 - DEBUG - Starting all layer2 verification in 10.0.0.33 Control Node 2026-01-05 04:09:43,365 - DEBUG - Layer2 route found for VM MAC 02:d3:1f:4d:f9:8a in Control-node 10.0.0.33 2026-01-05 04:09:43,365 - DEBUG - L2 Label for VM ctest-vn1_vm2-04238191 same between Control-node 10.0.0.33 and Agent, Expected: 34, Seen: 34 2026-01-05 04:09:43,365 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-04238191 passed 2026-01-05 04:09:43,365 - DEBUG - Verifying the vm in opserver 2026-01-05 04:09:43,365 - DEBUG - Verifying in collector 10.0.0.23 ... 2026-01-05 04:09:43,365 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/b0a03ae2-32ce-41b4-be4a-a21deba0ce5d?flat 2026-01-05 04:09:43,429 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-57160223:d31f4df9-8a56-46a3-bcf2-f88dcdcdbd00?flat 2026-01-05 04:09:43,447 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-57160223:d31f4df9-8a56-46a3-bcf2-f88dcdcdbd00?flat 2026-01-05 04:09:43,463 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/b0a03ae2-32ce-41b4-be4a-a21deba0ce5d?flat 2026-01-05 04:09:43,522 - DEBUG - Verifying vm in vn uve 2026-01-05 04:09:43,522 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-57160223:d31f4df9-8a56-46a3-bcf2-f88dcdcdbd00?flat 2026-01-05 04:09:43,541 - DEBUG - VM uve shows interface as {'mac_address': '02:d3:1f:4d:f9:8a', 'vn_uuid': 'f704f219-363b-4ecc-bfba-8ab00b5ddf8c', 'vhostuser_mode': 0, 'admin_state': True, 'rx_vlan': 65535, 'port_mirror_enabled': False, 'active': True, 'tx_vlan': 65535, 'ip6_active': False, 'ip_address': '192.168.1.251', 'vm_uuid': 'b0a03ae2-32ce-41b4-be4a-a21deba0ce5d', 'fixed_ip4_list': ['192.168.1.251'], 'l2_active': True, 'uuid': 'd31f4df9-8a56-46a3-bcf2-f88dcdcdbd00', 'label': 30, 'is_health_check_active': True, 'gateway': '192.168.1.254', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566', 'vm_name': 'ctest-vn1_vm2-04238191', '__T': 1767586156242412, 'ip6_address': '::', 'ip4_active': True} 2026-01-05 04:09:43,541 - DEBUG - VM uve shows ip address as ['192.168.1.251', '::'] 2026-01-05 04:09:43,541 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 2026-01-05 04:09:43,541 - INFO - Verifying through opserver in 10.0.0.27 2026-01-05 04:09:43,541 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566?flat 2026-01-05 04:09:43,564 - DEBUG - expected vm list b0a03ae2-32ce-41b4-be4a-a21deba0ce5d 2026-01-05 04:09:43,564 - DEBUG - Extracted vm list ['b0a03ae2-32ce-41b4-be4a-a21deba0ce5d'] 2026-01-05 04:09:43,564 - INFO - VM b0a03ae2-32ce-41b4-be4a-a21deba0ce5d is present in default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 2026-01-05 04:09:43,565 - DEBUG - Verifying vm in vrouter uve 2026-01-05 04:09:43,565 - DEBUG - Getting info from collector 10.0.0.23.. 2026-01-05 04:09:43,565 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/b0a03ae2-32ce-41b4-be4a-a21deba0ce5d?flat 2026-01-05 04:09:43,580 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4961-2'] 2026-01-05 04:09:43,580 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4961-2:Compute:contrail-vrouter-agent:0?flat 2026-01-05 04:09:43,615 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4961-2?flat 2026-01-05 04:09:43,641 - DEBUG - VM b0a03ae2-32ce-41b4-be4a-a21deba0ce5d is present in vrouter an-jenkins-deploy-platform-ansible-os-4961-2 uve 2026-01-05 04:09:43,641 - DEBUG - Validated that VM ctest-vn1_vm2-04238191 is in Vrouter an-jenkins-deploy-platform-ansible-os-4961-2 UVE 2026-01-05 04:09:43,641 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2026-01-05 04:09:43,648 - DEBUG - Expected tap interface of VM uuid b0a03ae2-32ce-41b4-be4a-a21deba0ce5d is default-domain:ctest-TestBasicVMVN0-57160223:d31f4df9-8a56-46a3-bcf2-f88dcdcdbd00 2026-01-05 04:09:43,648 - DEBUG - Expected VN of VM uuid b0a03ae2-32ce-41b4-be4a-a21deba0ce5d is default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 2026-01-05 04:09:43,648 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4961-2:Compute:contrail-vrouter-agent:0?flat 2026-01-05 04:09:43,680 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4961-2?flat 2026-01-05 04:09:43,707 - DEBUG - VM b0a03ae2-32ce-41b4-be4a-a21deba0ce5d is present in vrouter an-jenkins-deploy-platform-ansible-os-4961-2 uve 2026-01-05 04:09:43,707 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-57160223:d31f4df9-8a56-46a3-bcf2-f88dcdcdbd00 of vm b0a03ae2-32ce-41b4-be4a-a21deba0ce5d is present in vrouter an-jenkins-deploy-platform-ansible-os-4961-2 uve 2026-01-05 04:09:43,707 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 of vm b0a03ae2-32ce-41b4-be4a-a21deba0ce5d is present in vrouter an-jenkins-deploy-platform-ansible-os-4961-2 uve 2026-01-05 04:09:43,707 - DEBUG - Verifying the b0a03ae2-32ce-41b4-be4a-a21deba0ce5d virtual network link through opserver 10.0.0.27 2026-01-05 04:09:43,707 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-01-05 04:09:43,719 - DEBUG - VM link and name as {'name': 'b0a03ae2-32ce-41b4-be4a-a21deba0ce5d', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/b0a03ae2-32ce-41b4-be4a-a21deba0ce5d?flat'} 2026-01-05 04:09:43,719 - DEBUG - Verifying the b0a03ae2-32ce-41b4-be4a-a21deba0ce5d virtual network link through opserver 10.0.0.33 2026-01-05 04:09:43,719 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2026-01-05 04:09:43,781 - DEBUG - VM link and name as {'name': 'b0a03ae2-32ce-41b4-be4a-a21deba0ce5d', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-machine/b0a03ae2-32ce-41b4-be4a-a21deba0ce5d?flat'} 2026-01-05 04:09:43,781 - DEBUG - Verifying the b0a03ae2-32ce-41b4-be4a-a21deba0ce5d virtual network link through opserver 10.0.0.23 2026-01-05 04:09:43,781 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-01-05 04:09:43,795 - DEBUG - VM link and name as {'name': 'b0a03ae2-32ce-41b4-be4a-a21deba0ce5d', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/b0a03ae2-32ce-41b4-be4a-a21deba0ce5d?flat'} 2026-01-05 04:09:43,795 - INFO - VM ctest-vn1_vm2-04238191 validations in Opserver passed 2026-01-05 04:09:44,934 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4961-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4961-1) 2026-01-05 04:09:44,978 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2026-01-05 04:09:50,075 - DEBUG - VM is in ACTIVE state now 2026-01-05 04:09:50,075 - INFO - VM name : ctest-vn2_vm2-77715537 2026-01-05 04:09:50,166 - DEBUG - VM ctest-vn2_vm2-77715537 ID is 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d 2026-01-05 04:09:50,252 - DEBUG - VM is in ACTIVE state now 2026-01-05 04:09:50,252 - DEBUG - VM ctest-vn2_vm2-77715537 launched on Node an-jenkins-deploy-platform-ansible-os-4961-1 2026-01-05 04:09:50,339 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d 2026-01-05 04:09:50,355 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d 2026-01-05 04:09:50,367 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/ab9f74eb-a038-4b0c-9932-8ced0a87c40d 2026-01-05 04:09:50,389 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d 2026-01-05 04:09:50,398 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/ab9f74eb-a038-4b0c-9932-8ced0a87c40d 2026-01-05 04:09:50,408 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/a4b8a25c-f7fb-430c-bffd-a8d0d030010c 2026-01-05 04:09:50,418 - DEBUG - Verifying in api server 10.0.0.23 2026-01-05 04:09:50,418 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d 2026-01-05 04:09:50,426 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/ab9f74eb-a038-4b0c-9932-8ced0a87c40d 2026-01-05 04:09:50,437 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/a4b8a25c-f7fb-430c-bffd-a8d0d030010c 2026-01-05 04:09:50,446 - INFO - VM ctest-vn2_vm2-77715537 verfication in all API Servers passed 2026-01-05 04:09:50,466 - DEBUG - VM ctest-vn2_vm2-77715537 Tap interface: {'index': '4', 'name': 'tapab9f74eb-a0', 'uuid': 'ab9f74eb-a038-4b0c-9932-8ced0a87c40d', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408:ctest-vn2-71838408', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408', 'vm_uuid': '4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d', 'vm_name': 'ctest-vn2_vm2-77715537', 'ip_addr': '192.168.2.251', 'mac_addr': '02:ab:9f:74:eb:a0', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '131', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-57160223:ab9f74eb-a038-4b0c-9932-8ced0a87c40d', 'sg_uuid_list': ['5479f490-6e0d-425c-ab38-c03dcacd392b'], 'static_route_list': None, 'vm_project_uuid': 'f5158d13-df10-419a-8547-25010c329027', 'admin_state': 'Enabled', 'flow_key_idx': '34', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-01-05 04:09:50,466 - DEBUG - Agent 10.0.0.57 vrf name: default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408:ctest-vn2-71838408 2026-01-05 04:09:50,472 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408:ctest-vn2-71838408', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408', 'table_label': '-1', 'vxlan_id': '15', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.18:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2026-01-05 04:09:50,558 - DEBUG - VM ctest-vn2_vm2-77715537 : Tap interface tapab9f74eb-a0 is set to Active 2026-01-05 04:09:50,558 - DEBUG - Tap interface tapab9f74eb-a0 detail : {'index': '4', 'name': 'tapab9f74eb-a0', 'uuid': 'ab9f74eb-a038-4b0c-9932-8ced0a87c40d', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408:ctest-vn2-71838408', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408', 'vm_uuid': '4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d', 'vm_name': 'ctest-vn2_vm2-77715537', 'ip_addr': '192.168.2.251', 'mac_addr': '02:ab:9f:74:eb:a0', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '131', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-57160223:ab9f74eb-a038-4b0c-9932-8ced0a87c40d', 'sg_uuid_list': ['5479f490-6e0d-425c-ab38-c03dcacd392b'], 'static_route_list': None, 'vm_project_uuid': 'f5158d13-df10-419a-8547-25010c329027', 'admin_state': 'Enabled', 'flow_key_idx': '34', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-01-05 04:09:50,670 - DEBUG - Starting Layer 2 verification in Agent 2026-01-05 04:09:50,684 - DEBUG - Layer 2 path is seen for VM MAC 02:ab:9f:74:eb:a0 in agent 10.0.0.57 2026-01-05 04:09:50,684 - DEBUG - Active layer 2 route in agent is present for VMI tapab9f74eb-a0 2026-01-05 04:09:50,684 - DEBUG - L2 label(34) matches bw route table and interface table 2026-01-05 04:09:50,693 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 is found in Agent of node 10.0.0.57 2026-01-05 04:09:50,823 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 is consistent in agent 10.0.0.57 2026-01-05 04:09:50,824 - DEBUG - Route for VM IP ['192.168.2.251'] is consistent in agent 10.0.0.57 2026-01-05 04:09:50,824 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 verification for VM ctest-vn2_vm2-77715537 in Agent 10.0.0.57 passed 2026-01-05 04:09:50,935 - DEBUG - Starting all layer 2 verification in agent 10.0.0.57 2026-01-05 04:09:50,949 - DEBUG - Route for VM MAC 02:ab:9f:74:eb:a0 is consistent in agent 10.0.0.57 2026-01-05 04:09:54,163 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1031ms') 2026-01-05 04:09:54,163 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-77715537 failed! 2026-01-05 04:09:54,177 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 is 192.168.2.254 and allocation pool is NOT set 2026-01-05 04:09:58,263 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2026-01-05 04:09:58,263 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-77715537 failed! 2026-01-05 04:09:58,277 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 is 192.168.2.254 and allocation pool is NOT set 2026-01-05 04:10:02,354 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2026-01-05 04:10:02,354 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-77715537 failed! 2026-01-05 04:10:02,368 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 is 192.168.2.254 and allocation pool is NOT set 2026-01-05 04:10:06,451 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2026-01-05 04:10:06,451 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-77715537 failed! 2026-01-05 04:10:06,465 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 is 192.168.2.254 and allocation pool is NOT set 2026-01-05 04:10:10,551 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2026-01-05 04:10:10,551 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-77715537 failed! 2026-01-05 04:10:10,567 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 is 192.168.2.254 and allocation pool is NOT set 2026-01-05 04:10:14,645 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1008ms') 2026-01-05 04:10:14,645 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-77715537 failed! 2026-01-05 04:10:14,658 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 is 192.168.2.254 and allocation pool is NOT set 2026-01-05 04:10:18,741 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2026-01-05 04:10:18,741 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-77715537 failed! 2026-01-05 04:10:18,755 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 is 192.168.2.254 and allocation pool is NOT set 2026-01-05 04:10:22,834 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2026-01-05 04:10:22,834 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-77715537 failed! 2026-01-05 04:10:22,850 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 is 192.168.2.254 and allocation pool is NOT set 2026-01-05 04:10:26,935 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2026-01-05 04:10:26,935 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-77715537 failed! 2026-01-05 04:10:26,949 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 is 192.168.2.254 and allocation pool is NOT set 2026-01-05 04:10:29,016 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=7.61 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.530 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 0.530/4.069/7.609/3.539 ms') 2026-01-05 04:10:29,016 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-77715537 passed 2026-01-05 04:10:29,016 - INFO - VM ctest-vn2_vm2-77715537 verifications in Compute nodes passed 2026-01-05 04:10:29,249 - DEBUG - Validated VM route 192.168.2.251 in vrouter of 10.0.0.57 2026-01-05 04:10:29,255 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2026-01-05 04:10:29,255 - INFO - Validated routes of VM ctest-vn2_vm2-77715537 in all vrouters 2026-01-05 04:10:29,673 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2026-01-05 04:10:29,799 - DEBUG - Layer2 route found for VM MAC 02:ab:9f:74:eb:a0 in Control-node 10.0.0.23 2026-01-05 04:10:29,799 - DEBUG - L2 Label for VM ctest-vn2_vm2-77715537 same between Control-node 10.0.0.23 and Agent, Expected: 34, Seen: 34 2026-01-05 04:10:29,800 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2026-01-05 04:10:29,929 - DEBUG - Layer2 route found for VM MAC 02:ab:9f:74:eb:a0 in Control-node 10.0.0.27 2026-01-05 04:10:29,929 - DEBUG - L2 Label for VM ctest-vn2_vm2-77715537 same between Control-node 10.0.0.27 and Agent, Expected: 34, Seen: 34 2026-01-05 04:10:29,929 - DEBUG - Starting all layer2 verification in 10.0.0.33 Control Node 2026-01-05 04:10:30,061 - DEBUG - Layer2 route found for VM MAC 02:ab:9f:74:eb:a0 in Control-node 10.0.0.33 2026-01-05 04:10:30,062 - DEBUG - L2 Label for VM ctest-vn2_vm2-77715537 same between Control-node 10.0.0.33 and Agent, Expected: 34, Seen: 34 2026-01-05 04:10:30,062 - INFO - Verification in Control-nodes for VM ctest-vn2_vm2-77715537 passed 2026-01-05 04:10:30,062 - DEBUG - Verifying the vm in opserver 2026-01-05 04:10:30,062 - DEBUG - Verifying in collector 10.0.0.23 ... 2026-01-05 04:10:30,062 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d?flat 2026-01-05 04:10:30,081 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-57160223:ab9f74eb-a038-4b0c-9932-8ced0a87c40d?flat 2026-01-05 04:10:30,105 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-57160223:ab9f74eb-a038-4b0c-9932-8ced0a87c40d?flat 2026-01-05 04:10:30,126 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d?flat 2026-01-05 04:10:30,142 - DEBUG - Verifying vm in vn uve 2026-01-05 04:10:30,142 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-57160223:ab9f74eb-a038-4b0c-9932-8ced0a87c40d?flat 2026-01-05 04:10:30,165 - DEBUG - VM uve shows interface as {'gateway': '192.168.2.254', 'vhostuser_mode': 0, 'vm_name': 'ctest-vn2_vm2-77715537', 'active': True, 'fixed_ip4_list': ['192.168.2.251'], 'tx_vlan': 65535, '__T': 1767586207807261, 'ip6_active': False, 'mac_address': '02:ab:9f:74:eb:a0', 'l2_active': True, 'port_mirror_enabled': False, 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408', 'ip4_active': True, 'vn_uuid': 'e088200e-19f3-44cc-9ccd-270934d1a1e6', 'is_health_check_active': True, 'ip_address': '192.168.2.251', 'ip6_address': '::', 'label': 30, 'rx_vlan': 65535, 'uuid': 'ab9f74eb-a038-4b0c-9932-8ced0a87c40d', 'vm_uuid': '4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d', 'admin_state': True} 2026-01-05 04:10:30,165 - DEBUG - VM uve shows ip address as ['192.168.2.251', '::'] 2026-01-05 04:10:30,165 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 2026-01-05 04:10:30,165 - INFO - Verifying through opserver in 10.0.0.27 2026-01-05 04:10:30,165 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408?flat 2026-01-05 04:10:30,190 - DEBUG - expected vm list 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d 2026-01-05 04:10:30,190 - DEBUG - Extracted vm list ['4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d'] 2026-01-05 04:10:30,190 - INFO - VM 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d is present in default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 2026-01-05 04:10:30,190 - DEBUG - Verifying vm in vrouter uve 2026-01-05 04:10:30,190 - DEBUG - Getting info from collector 10.0.0.23.. 2026-01-05 04:10:30,190 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d?flat 2026-01-05 04:10:30,210 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4961-1'] 2026-01-05 04:10:30,210 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4961-1:Compute:contrail-vrouter-agent:0?flat 2026-01-05 04:10:30,254 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4961-1?flat 2026-01-05 04:10:30,281 - DEBUG - VM 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d is present in vrouter an-jenkins-deploy-platform-ansible-os-4961-1 uve 2026-01-05 04:10:30,281 - DEBUG - Validated that VM ctest-vn2_vm2-77715537 is in Vrouter an-jenkins-deploy-platform-ansible-os-4961-1 UVE 2026-01-05 04:10:30,281 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2026-01-05 04:10:30,289 - DEBUG - Expected tap interface of VM uuid 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d is default-domain:ctest-TestBasicVMVN0-57160223:ab9f74eb-a038-4b0c-9932-8ced0a87c40d 2026-01-05 04:10:30,289 - DEBUG - Expected VN of VM uuid 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d is default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 2026-01-05 04:10:30,289 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4961-1:Compute:contrail-vrouter-agent:0?flat 2026-01-05 04:10:30,337 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4961-1?flat 2026-01-05 04:10:30,374 - DEBUG - VM 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d is present in vrouter an-jenkins-deploy-platform-ansible-os-4961-1 uve 2026-01-05 04:10:30,374 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-57160223:ab9f74eb-a038-4b0c-9932-8ced0a87c40d of vm 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d is present in vrouter an-jenkins-deploy-platform-ansible-os-4961-1 uve 2026-01-05 04:10:30,374 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 of vm 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d is present in vrouter an-jenkins-deploy-platform-ansible-os-4961-1 uve 2026-01-05 04:10:30,374 - DEBUG - Verifying the 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d virtual network link through opserver 10.0.0.27 2026-01-05 04:10:30,374 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-01-05 04:10:30,390 - DEBUG - VM link and name as {'name': '4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d?flat'} 2026-01-05 04:10:30,390 - DEBUG - Verifying the 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d virtual network link through opserver 10.0.0.33 2026-01-05 04:10:30,390 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2026-01-05 04:10:30,404 - DEBUG - VM link and name as {'name': '4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-machine/4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d?flat'} 2026-01-05 04:10:30,404 - DEBUG - Verifying the 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d virtual network link through opserver 10.0.0.23 2026-01-05 04:10:30,404 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-01-05 04:10:30,419 - DEBUG - VM link and name as {'name': '4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d?flat'} 2026-01-05 04:10:30,419 - INFO - VM ctest-vn2_vm2-77715537 validations in Opserver passed 2026-01-05 04:10:30,419 - INFO - Waiting for VM ctest-vn1_vm2-04238191 to be up.. 2026-01-05 04:10:30,420 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:10:30,420 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/b0a03ae2-32ce-41b4-be4a-a21deba0ce5d 2026-01-05 04:10:30,429 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/d31f4df9-8a56-46a3-bcf2-f88dcdcdbd00 2026-01-05 04:10:31,585 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=3.26 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.457 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.457/1.857/3.257/1.400 ms') 2026-01-05 04:10:31,586 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-04238191 passed 2026-01-05 04:10:31,651 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:10:31,651 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-04238191, IP 192.168.1.251, Port 22 2026-01-05 04:10:31,819 - DEBUG - VM ctest-vn1_vm2-04238191 is ready for SSH connections 2026-01-05 04:10:31,819 - INFO - Waiting for VM ctest-vn2_vm2-77715537 to be up.. 2026-01-05 04:10:31,819 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:10:31,819 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d 2026-01-05 04:10:31,827 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/ab9f74eb-a038-4b0c-9932-8ced0a87c40d 2026-01-05 04:10:32,999 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=3.61 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=3.90 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 3.614/3.754/3.895/0.140 ms') 2026-01-05 04:10:32,999 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-77715537 passed 2026-01-05 04:10:33,071 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:10:33,071 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-77715537, IP 192.168.2.251, Port 22 2026-01-05 04:10:33,138 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-01-05 04:10:33,224 - DEBUG - VM ctest-vn2_vm2-77715537 is NOT ready for SSH connections, VM status: ACTIVE 2026-01-05 04:10:38,225 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:10:38,225 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-77715537, IP 192.168.2.251, Port 22 2026-01-05 04:10:38,290 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-01-05 04:10:38,372 - DEBUG - VM ctest-vn2_vm2-77715537 is NOT ready for SSH connections, VM status: ACTIVE 2026-01-05 04:10:43,373 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:10:43,373 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-77715537, IP 192.168.2.251, Port 22 2026-01-05 04:10:43,428 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-01-05 04:10:43,509 - DEBUG - VM ctest-vn2_vm2-77715537 is NOT ready for SSH connections, VM status: ACTIVE 2026-01-05 04:10:48,510 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:10:48,510 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-77715537, IP 192.168.2.251, Port 22 2026-01-05 04:10:48,678 - DEBUG - VM ctest-vn2_vm2-77715537 is ready for SSH connections 2026-01-05 04:10:48,678 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-05 04:10:48,678 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.251, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.56, gateway password: c0ntrail123 2026-01-05 04:10:48,678 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.251 2026-01-05 04:11:28,774 - DEBUG - PING 192.168.2.251 (192.168.2.251) 56(84) bytes of data. 64 bytes from 192.168.2.251: icmp_req=1 ttl=63 time=8.12 ms 64 bytes from 192.168.2.251: icmp_req=2 ttl=63 time=1.36 ms 64 bytes from 192.168.2.251: icmp_req=3 ttl=63 time=1.06 ms --- 192.168.2.251 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.061/3.518/8.126/3.260 ms 2026-01-05 04:11:28,774 - INFO - Ping to IP 192.168.2.251 from VM ctest-vn1_vm2-04238191 passed 2026-01-05 04:11:28,774 - INFO - Deleting VM ctest-vn2_vm2-77715537 2026-01-05 04:11:28,844 - DEBUG - Verifying in api server 10.0.0.23 2026-01-05 04:11:28,844 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d 2026-01-05 04:11:28,853 - DEBUG - VM ID 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d of VM ctest-vn2_vm2-77715537 is still found in API Server 2026-01-05 04:11:30,854 - DEBUG - Verifying in api server 10.0.0.23 2026-01-05 04:11:30,854 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d 2026-01-05 04:11:30,861 - DEBUG - Response Code: 404 2026-01-05 04:11:30,862 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d 2026-01-05 04:11:30,867 - DEBUG - Response Code: 404 2026-01-05 04:11:30,868 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d 2026-01-05 04:11:30,873 - DEBUG - Response Code: 404 2026-01-05 04:11:30,873 - INFO - VM ctest-vn2_vm2-77715537 is fully removed in API-Server 2026-01-05 04:11:31,026 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2026-01-05 04:11:31,187 - DEBUG - VM route 192.168.2.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 of compute 10.0.0.57 2026-01-05 04:11:31,193 - DEBUG - Validated that vrouter 10.0.0.57 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 2026-01-05 04:11:31,193 - DEBUG - Validated that vrouter 10.0.0.56 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 2026-01-05 04:11:31,193 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 2026-01-05 04:11:31,193 - INFO - VM ctest-vn2_vm2-77715537 is removed in Compute, and routes are removed in all compute nodes 2026-01-05 04:11:31,418 - INFO - Routes for VM ctest-vn2_vm2-77715537 is removed in all control-nodes 2026-01-05 04:11:31,418 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4961-1:Compute:contrail-vrouter-agent:0?flat 2026-01-05 04:11:31,464 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4961-1?flat 2026-01-05 04:11:31,493 - DEBUG - VM 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d is still present in vrouter an-jenkins-deploy-platform-ansible-os-4961-1 uve 2026-01-05 04:11:31,493 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-05 04:11:31,493 - DEBUG - interface for vm 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d deleted from vrouter uve ... 2026-01-05 04:11:31,493 - DEBUG - Deleted interface not in error interface list ... 2026-01-05 04:11:34,494 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4961-1:Compute:contrail-vrouter-agent:0?flat 2026-01-05 04:11:34,536 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4961-1?flat 2026-01-05 04:11:34,567 - DEBUG - VM 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d is still present in vrouter an-jenkins-deploy-platform-ansible-os-4961-1 uve 2026-01-05 04:11:34,567 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-05 04:11:34,568 - DEBUG - interface for vm 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d deleted from vrouter uve ... 2026-01-05 04:11:34,568 - DEBUG - Deleted interface not in error interface list ... 2026-01-05 04:11:37,568 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4961-1:Compute:contrail-vrouter-agent:0?flat 2026-01-05 04:11:37,618 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4961-1?flat 2026-01-05 04:11:37,648 - DEBUG - VM 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d is still present in vrouter an-jenkins-deploy-platform-ansible-os-4961-1 uve 2026-01-05 04:11:37,648 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-05 04:11:37,648 - DEBUG - interface for vm 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d deleted from vrouter uve ... 2026-01-05 04:11:37,649 - DEBUG - Deleted interface not in error interface list ... 2026-01-05 04:11:40,649 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4961-1:Compute:contrail-vrouter-agent:0?flat 2026-01-05 04:11:40,734 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4961-1?flat 2026-01-05 04:11:40,765 - DEBUG - VM 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d is not present in vrouter an-jenkins-deploy-platform-ansible-os-4961-1 uve 2026-01-05 04:11:40,765 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-05 04:11:40,765 - DEBUG - interface for vm 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d deleted from vrouter uve ... 2026-01-05 04:11:40,765 - DEBUG - Deleted interface not in error interface list ... 2026-01-05 04:11:40,765 - DEBUG - Validated that VM 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d is removed in Vrouter UVE 2026-01-05 04:11:40,765 - DEBUG - Verifying through opserver in 10.0.0.27 2026-01-05 04:11:40,765 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408?flat 2026-01-05 04:11:40,787 - DEBUG - 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d BM not in default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn2-71838408 uve 2026-01-05 04:11:40,787 - DEBUG - Validated that VM 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d is not present in the VN UVE 2026-01-05 04:11:40,787 - DEBUG - Verifying the 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d virtual network link through opserver 10.0.0.27 2026-01-05 04:11:40,787 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-01-05 04:11:40,801 - INFO - 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d vm uve deleted from opserver 2026-01-05 04:11:40,801 - DEBUG - Verifying the 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d virtual network link through opserver 10.0.0.33 2026-01-05 04:11:40,801 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2026-01-05 04:11:40,814 - INFO - 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d vm uve deleted from opserver 2026-01-05 04:11:40,814 - DEBUG - Verifying the 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d virtual network link through opserver 10.0.0.23 2026-01-05 04:11:40,815 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-01-05 04:11:40,828 - INFO - 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d vm uve deleted from opserver 2026-01-05 04:11:40,829 - INFO - Validated that VM 4c141e6c-f6ff-4ec8-8b53-669c01fc7d9d is removed from Opserver 2026-01-05 04:11:40,829 - INFO - Deleting VM ctest-vn1_vm2-04238191 2026-01-05 04:11:40,903 - DEBUG - Verifying in api server 10.0.0.23 2026-01-05 04:11:40,903 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/b0a03ae2-32ce-41b4-be4a-a21deba0ce5d 2026-01-05 04:11:40,911 - DEBUG - VM ID b0a03ae2-32ce-41b4-be4a-a21deba0ce5d of VM ctest-vn1_vm2-04238191 is still found in API Server 2026-01-05 04:11:42,911 - DEBUG - Verifying in api server 10.0.0.23 2026-01-05 04:11:42,911 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/b0a03ae2-32ce-41b4-be4a-a21deba0ce5d 2026-01-05 04:11:42,918 - DEBUG - Response Code: 404 2026-01-05 04:11:42,918 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/b0a03ae2-32ce-41b4-be4a-a21deba0ce5d 2026-01-05 04:11:42,923 - DEBUG - Response Code: 404 2026-01-05 04:11:42,923 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/b0a03ae2-32ce-41b4-be4a-a21deba0ce5d 2026-01-05 04:11:42,928 - DEBUG - Response Code: 404 2026-01-05 04:11:42,928 - INFO - VM ctest-vn1_vm2-04238191 is fully removed in API-Server 2026-01-05 04:11:43,082 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2026-01-05 04:11:43,113 - DEBUG - Validated that vrouter 10.0.0.57 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 2026-01-05 04:11:43,235 - DEBUG - VM route 192.168.1.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 of compute 10.0.0.56 2026-01-05 04:11:43,240 - DEBUG - Validated that vrouter 10.0.0.56 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 2026-01-05 04:11:43,240 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 2026-01-05 04:11:43,240 - INFO - VM ctest-vn1_vm2-04238191 is removed in Compute, and routes are removed in all compute nodes 2026-01-05 04:11:43,451 - INFO - Routes for VM ctest-vn1_vm2-04238191 is removed in all control-nodes 2026-01-05 04:11:43,451 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4961-2:Compute:contrail-vrouter-agent:0?flat 2026-01-05 04:11:43,487 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4961-2?flat 2026-01-05 04:11:43,515 - DEBUG - VM b0a03ae2-32ce-41b4-be4a-a21deba0ce5d is still present in vrouter an-jenkins-deploy-platform-ansible-os-4961-2 uve 2026-01-05 04:11:43,515 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-05 04:11:43,515 - DEBUG - interface for vm b0a03ae2-32ce-41b4-be4a-a21deba0ce5d deleted from vrouter uve ... 2026-01-05 04:11:43,515 - DEBUG - Deleted interface not in error interface list ... 2026-01-05 04:11:46,516 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4961-2:Compute:contrail-vrouter-agent:0?flat 2026-01-05 04:11:46,548 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4961-2?flat 2026-01-05 04:11:46,579 - DEBUG - VM b0a03ae2-32ce-41b4-be4a-a21deba0ce5d is not present in vrouter an-jenkins-deploy-platform-ansible-os-4961-2 uve 2026-01-05 04:11:46,579 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-05 04:11:46,579 - DEBUG - interface for vm b0a03ae2-32ce-41b4-be4a-a21deba0ce5d deleted from vrouter uve ... 2026-01-05 04:11:46,579 - DEBUG - Deleted interface not in error interface list ... 2026-01-05 04:11:46,579 - DEBUG - Validated that VM b0a03ae2-32ce-41b4-be4a-a21deba0ce5d is removed in Vrouter UVE 2026-01-05 04:11:46,579 - DEBUG - Verifying through opserver in 10.0.0.27 2026-01-05 04:11:46,580 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566?flat 2026-01-05 04:11:46,604 - DEBUG - b0a03ae2-32ce-41b4-be4a-a21deba0ce5d BM not in default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 uve 2026-01-05 04:11:46,604 - DEBUG - Validated that VM b0a03ae2-32ce-41b4-be4a-a21deba0ce5d is not present in the VN UVE 2026-01-05 04:11:46,604 - DEBUG - Verifying the b0a03ae2-32ce-41b4-be4a-a21deba0ce5d virtual network link through opserver 10.0.0.27 2026-01-05 04:11:46,604 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-01-05 04:11:46,617 - INFO - b0a03ae2-32ce-41b4-be4a-a21deba0ce5d vm uve deleted from opserver 2026-01-05 04:11:46,617 - DEBUG - Verifying the b0a03ae2-32ce-41b4-be4a-a21deba0ce5d virtual network link through opserver 10.0.0.33 2026-01-05 04:11:46,617 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2026-01-05 04:11:46,628 - INFO - b0a03ae2-32ce-41b4-be4a-a21deba0ce5d vm uve deleted from opserver 2026-01-05 04:11:46,628 - DEBUG - Verifying the b0a03ae2-32ce-41b4-be4a-a21deba0ce5d virtual network link through opserver 10.0.0.23 2026-01-05 04:11:46,628 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-01-05 04:11:46,640 - INFO - b0a03ae2-32ce-41b4-be4a-a21deba0ce5d vm uve deleted from opserver 2026-01-05 04:11:46,640 - INFO - Validated that VM b0a03ae2-32ce-41b4-be4a-a21deba0ce5d is removed from Opserver 2026-01-05 04:11:46,640 - INFO - Deleting VN ctest-vn3-38592111 2026-01-05 04:11:46,749 - INFO - Deleting VM ctest-vn2_vm1-00171585 2026-01-05 04:11:46,836 - INFO - Deleting VM ctest-vn1_vm1-76914702 2026-01-05 04:11:47,097 - INFO - Deleting VN ctest-vn2-71838408 2026-01-05 04:11:47,150 - DEBUG - RefsExistError Delete when resource still referred: ['http://10.0.0.23:8082/virtual-machine-interface/27c64578-bddf-48b4-b2cc-0d511db60dd4'] while deleting VN ctest-vn2-71838408..Will retry 2026-01-05 04:11:49,296 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/1159fc4a-f75e-43df-960f-95a3dd4096d8 2026-01-05 04:11:49,302 - DEBUG - Response Code: 404 2026-01-05 04:11:49,302 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-05 04:11:49,313 - DEBUG - Requesting: http://10.0.0.27:8082/domain/182056ec-3382-44e1-b0fd-9a9d794f24a6 2026-01-05 04:11:49,327 - DEBUG - Requesting: http://10.0.0.27:8082/project/f5158d13-df10-419a-8547-25010c329027 2026-01-05 04:11:49,412 - INFO - Validated that VN ctest-vn2-71838408 is not found in API Server 2026-01-05 04:11:49,425 - DEBUG - VN ctest-vn2-71838408 is not present in Agent 10.0.0.57 2026-01-05 04:11:49,436 - DEBUG - VN ctest-vn2-71838408 is not present in Agent 10.0.0.56 2026-01-05 04:11:49,436 - INFO - Validated that VN ctest-vn2-71838408 is not in any agent 2026-01-05 04:11:49,471 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn2-71838408 info 2026-01-05 04:11:49,628 - INFO - Deleting VN ctest-vn1-82287566 2026-01-05 04:11:49,751 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/c094f679-de01-43c8-829b-44b6e80b8f2c 2026-01-05 04:11:49,759 - DEBUG - Response Code: 404 2026-01-05 04:11:49,759 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-05 04:11:49,765 - DEBUG - Requesting: http://10.0.0.27:8082/domain/182056ec-3382-44e1-b0fd-9a9d794f24a6 2026-01-05 04:11:49,779 - DEBUG - Requesting: http://10.0.0.27:8082/project/f5158d13-df10-419a-8547-25010c329027 2026-01-05 04:11:49,891 - INFO - Validated that VN ctest-vn1-82287566 is not found in API Server 2026-01-05 04:11:49,904 - DEBUG - VN ctest-vn1-82287566 is not present in Agent 10.0.0.57 2026-01-05 04:11:49,916 - DEBUG - VN ctest-vn1-82287566 is not present in Agent 10.0.0.56 2026-01-05 04:11:49,916 - INFO - Validated that VN ctest-vn1-82287566 is not in any agent 2026-01-05 04:11:49,924 - DEBUG - VRF 2 is not seen in agent 10.0.0.56 2026-01-05 04:11:49,930 - DEBUG - Vrouter 10.0.0.56 does not have vrf 2 for VN ctest-vn1-82287566 2026-01-05 04:11:49,931 - INFO - Validated that all vrouters do not have the route table for VN default-domain:ctest-TestBasicVMVN0-57160223:ctest-vn1-82287566 2026-01-05 04:11:49,953 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-82287566 info 2026-01-05 04:11:49,999 - INFO - Deleted policy policy2 2026-01-05 04:11:50,040 - INFO - Deleted policy policy1 2026-01-05 04:11:51,309 - DEBUG - Skipping xmpp flap check
2026-01-05 04:11:51,309 - INFO - END TEST : test_process_restart_in_policy_between_vns : PASSED[0:05:51]
2026-01-05 04:11:51,309 - INFO - -------------------------------------------------------------------------------- 2026-01-05 04:11:51,312 - INFO - ================================================================================ 2026-01-05 04:11:51,312 - INFO - STARTING TEST : test_underlay_broadcast_traffic_handling 2026-01-05 04:11:51,312 - INFO - TEST DESCRIPTION : Test the underlay brocast traffic handling by vrouter. (Bug-1545229). 1. Send broadcast traffic from one compute node. 2. Other compute in same subnet should receive that traffic. 3. Receiving compute should treat this traffic as underlay. 4. Compute should not replicate the packet and send the copy back. Pass criteria: Step 3-4 should pass Maintainer : chhandak@juniper.net 2026-01-05 04:11:52,569 - DEBUG - Skipping xmpp flap check 2026-01-05 04:11:52,570 - INFO - Initial checks done. Running the testcase now 2026-01-05 04:11:52,570 - INFO - 2026-01-05 04:11:52,954 - INFO - Agent interface name: ens6 2026-01-05 04:11:53,036 - DEBUG - Executing command: sudo tcpdump -nni ens6 -U host 10.20.0.255 -w /tmp/ens6_ctest-random-89645679.pcap 2026-01-05 04:11:58,404 - INFO - Pinging broacast address 10.20.0.255 from compute 10.20.0.18 2026-01-05 04:12:12,689 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2026-01-05 04:12:14,828 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-89645679.pcap | grep -c length 2026-01-05 04:12:14,842 - DEBUG - STDOUT: 10 2026-01-05 04:12:14,842 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-89645679.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2026-01-05 04:12:14,842 - INFO - 10 packets are found in tcpdump output as expected 2026-01-05 04:12:14,842 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2026-01-05 04:12:16,891 - INFO - Packet count matched: Compute 10.0.0.56 has receive only 10 packet from source IP 10.20.0.255. No duplicate packet seen 2026-01-05 04:12:18,157 - DEBUG - Skipping xmpp flap check
2026-01-05 04:12:18,157 - INFO - END TEST : test_underlay_broadcast_traffic_handling : PASSED[0:00:27]
2026-01-05 04:12:18,158 - INFO - -------------------------------------------------------------------------------- 2026-01-05 04:12:18,877 - INFO - Deleted project: ctest-TestBasicVMVN0-57160223, ID : f5158d13-df10-419a-8547-25010c329027