2025-10-16 04:16:55,232 - INFO - Domain Default found not creating 2025-10-16 04:16:55,375 - INFO - Project ctest-TestBasicVMVN0-91100301 not found, creating it 2025-10-16 04:16:55,873 - INFO - Created Project:ctest-TestBasicVMVN0-91100301, ID : 03e29f65-50a8-4f9b-937e-9a83abd9c7d6 2025-10-16 04:16:57,347 - INFO - ================================================================================ 2025-10-16 04:16:57,347 - INFO - STARTING TEST : test_control_node_switchover 2025-10-16 04:16:57,347 - INFO - TEST DESCRIPTION : Stop the control node and check peering with agent fallback to other control node. 1. Pick one VN from respource pool which has 2 VM's in it 2. Verify ping between VM's 3. Find active control node in cluster by agent inspect 4. Stop control service on active control node 5. Verify agents are connected to new active control-node using xmpp connections 6. Bring back control service on previous active node 7. Verify ping between VM's again after bringing up control serveice Pass criteria: Step 2,5 and 7 should pass 2025-10-16 04:16:58,609 - DEBUG - Skipping xmpp flap check 2025-10-16 04:16:58,609 - INFO - Initial checks done. Running the testcase now 2025-10-16 04:16:58,609 - INFO - 2025-10-16 04:16:59,249 - DEBUG - Response for create_network : {'network': {'id': '30e1ddf2-af9d-43e1-8c79-10950ad554ba', 'name': 'ctest-vn1-36242444', 'tenant_id': '03e29f6550a84f9b937e9a83abd9c7d6', 'project_id': '03e29f6550a84f9b937e9a83abd9c7d6', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestBasicVMVN0-91100301', 'ctest-vn1-36242444'], 'port_security_enabled': True, 'description': ''}} 2025-10-16 04:16:59,482 - DEBUG - Response for create_subnet : {'subnet': {'id': '0d48dfda-3a3d-4525-a680-5e8d9c20c4e7', 'name': '', 'tenant_id': '03e29f6550a84f9b937e9a83abd9c7d6', 'network_id': '30e1ddf2-af9d-43e1-8c79-10950ad554ba', 'ip_version': 4, 'cidr': '192.168.1.0/24', 'allocation_pools': [{'start': '192.168.1.2', 'end': '192.168.1.254'}], 'gateway_ip': '192.168.1.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '192.168.1.2', 'tags': [], 'project_id': '03e29f6550a84f9b937e9a83abd9c7d6'}} 2025-10-16 04:16:59,506 - INFO - Created VN ctest-vn1-36242444 2025-10-16 04:16:59,517 - DEBUG - VN ctest-vn1-36242444 UUID is 30e1ddf2-af9d-43e1-8c79-10950ad554ba 2025-10-16 04:16:59,596 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-10-16 04:16:59,892 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-10-16 04:16:59,924 - DEBUG - Requesting: http://10.0.0.19:8082/domain/446ea2e6-9d96-4973-a55a-3f633ea4d0f9 2025-10-16 04:16:59,943 - DEBUG - Requesting: http://10.0.0.19:8082/project/03e29f65-50a8-4f9b-937e-9a83abd9c7d6 2025-10-16 04:17:00,046 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/30e1ddf2-af9d-43e1-8c79-10950ad554ba 2025-10-16 04:17:00,058 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/30e1ddf2-af9d-43e1-8c79-10950ad554ba 2025-10-16 04:17:00,067 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/3d2c9a8d-4be7-4561-9387-d7fdf5fa5de4 2025-10-16 04:17:00,079 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/3d2c9a8d-4be7-4561-9387-d7fdf5fa5de4 2025-10-16 04:17:00,087 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/7e824d25-618c-4b22-b31e-ef621a156a96 2025-10-16 04:17:00,094 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-16 04:17:00,094 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/30e1ddf2-af9d-43e1-8c79-10950ad554ba 2025-10-16 04:17:00,104 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/3d2c9a8d-4be7-4561-9387-d7fdf5fa5de4 2025-10-16 04:17:00,119 - INFO - Verified VN network id 12 for VN 30e1ddf2-af9d-43e1-8c79-10950ad554ba 2025-10-16 04:17:00,119 - INFO - Verifications in API Server for VN ctest-vn1-36242444 passed 2025-10-16 04:17:00,119 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/30e1ddf2-af9d-43e1-8c79-10950ad554ba 2025-10-16 04:17:00,129 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/3d2c9a8d-4be7-4561-9387-d7fdf5fa5de4 2025-10-16 04:17:00,139 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/3d2c9a8d-4be7-4561-9387-d7fdf5fa5de4 2025-10-16 04:17:00,146 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/7e824d25-618c-4b22-b31e-ef621a156a96 2025-10-16 04:17:00,160 - DEBUG - Control-node 10.0.0.19 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91100301', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3522340417978581985', 'uuid-lslong': '10122139869809824954'}, 'enable': 'true', 'created': '2025-10-16T04:16:59', 'last-modified': '2025-10-16T04:16:59', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.734588'} 2025-10-16 04:17:00,168 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-16 04:17:00,174 - DEBUG - Control-node 10.0.0.57 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91100301', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3522340417978581985', 'uuid-lslong': '10122139869809824954'}, 'enable': 'true', 'created': '2025-10-16T04:16:59', 'last-modified': '2025-10-16T04:16:59', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.746640'} 2025-10-16 04:17:00,180 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-16 04:17:00,186 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91100301', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3522340417978581985', 'uuid-lslong': '10122139869809824954'}, 'enable': 'true', 'created': '2025-10-16T04:16:59', 'last-modified': '2025-10-16T04:16:59', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.762469'} 2025-10-16 04:17:00,194 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-16 04:17:00,194 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-36242444 passed 2025-10-16 04:17:00,194 - DEBUG - ====Verifying policy data for ctest-vn1-36242444 in API_Server ====== 2025-10-16 04:17:00,194 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-10-16 04:17:00,201 - DEBUG - Requesting: http://10.0.0.19:8082/domain/446ea2e6-9d96-4973-a55a-3f633ea4d0f9 2025-10-16 04:17:00,211 - DEBUG - Requesting: http://10.0.0.19:8082/project/03e29f65-50a8-4f9b-937e-9a83abd9c7d6 2025-10-16 04:17:00,302 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/30e1ddf2-af9d-43e1-8c79-10950ad554ba 2025-10-16 04:17:00,312 - DEBUG - =>VN ctest-vn1-36242444 has no policy to be verified 2025-10-16 04:17:00,312 - DEBUG - Verifying the vn in opserver 2025-10-16 04:17:00,312 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 virtual network link through opserver 10.0.0.19 2025-10-16 04:17:00,313 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-networks 2025-10-16 04:17:00,614 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-networks 2025-10-16 04:17:00,677 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444', 'href': 'http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444?flat'} 2025-10-16 04:17:00,677 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 virtual network link through opserver 10.0.0.57 2025-10-16 04:17:00,677 - DEBUG - Requesting: http://10.0.0.57:8081/analytics/uves/virtual-networks 2025-10-16 04:17:00,972 - DEBUG - Requesting: http://10.0.0.57:8081/analytics/uves/virtual-networks 2025-10-16 04:17:01,028 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444', 'href': 'http://10.0.0.57:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444?flat'} 2025-10-16 04:17:01,028 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 virtual network link through opserver 10.0.0.23 2025-10-16 04:17:01,028 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-10-16 04:17:01,341 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-10-16 04:17:01,406 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444?flat'} 2025-10-16 04:17:01,406 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 is found in opserver 2025-10-16 04:17:01,428 - DEBUG - Do not have enough data to verify VN in agent 2025-10-16 04:17:01,438 - DEBUG - VRF ids for VN ctest-vn1-36242444: {} 2025-10-16 04:17:01,501 - DEBUG - Services list from nova: [, , , ] 2025-10-16 04:17:02,550 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4339-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4339-1) 2025-10-16 04:17:04,001 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4339-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4339-2) 2025-10-16 04:17:04,001 - INFO - Waiting for VM ctest-vn1_vm1-38862421 to be up.. 2025-10-16 04:17:04,089 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-10-16 04:17:09,173 - DEBUG - VM is in ACTIVE state now 2025-10-16 04:17:09,173 - INFO - VM name : ctest-vn1_vm1-38862421 2025-10-16 04:17:09,256 - DEBUG - VM ctest-vn1_vm1-38862421 ID is e648138f-ddc6-4577-932b-ecbff466899b 2025-10-16 04:17:09,281 - DEBUG - VM ctest-vn1_vm1-38862421 launched on Node an-jenkins-deploy-platform-ansible-os-4339-1 2025-10-16 04:17:09,361 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/e648138f-ddc6-4577-932b-ecbff466899b 2025-10-16 04:17:09,675 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/e648138f-ddc6-4577-932b-ecbff466899b 2025-10-16 04:17:09,714 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/5fe3521f-072b-4936-9383-00e5809f22f6 2025-10-16 04:17:12,869 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1007ms') 2025-10-16 04:17:12,869 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-38862421 failed! 2025-10-16 04:17:12,884 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 is 192.168.1.1 and allocation pool is NOT set 2025-10-16 04:17:16,972 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2025-10-16 04:17:16,972 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-38862421 failed! 2025-10-16 04:17:16,987 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 is 192.168.1.1 and allocation pool is NOT set 2025-10-16 04:17:21,056 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2025-10-16 04:17:21,056 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-38862421 failed! 2025-10-16 04:17:21,068 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 is 192.168.1.1 and allocation pool is NOT set 2025-10-16 04:17:25,165 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1029ms') 2025-10-16 04:17:25,165 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-38862421 failed! 2025-10-16 04:17:25,178 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 is 192.168.1.1 and allocation pool is NOT set 2025-10-16 04:17:29,256 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2025-10-16 04:17:29,257 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-38862421 failed! 2025-10-16 04:17:29,270 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 is 192.168.1.1 and allocation pool is NOT set 2025-10-16 04:17:33,354 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-10-16 04:17:33,354 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-38862421 failed! 2025-10-16 04:17:33,372 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 is 192.168.1.1 and allocation pool is NOT set 2025-10-16 04:17:37,444 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms') 2025-10-16 04:17:37,444 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-38862421 failed! 2025-10-16 04:17:37,457 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 is 192.168.1.1 and allocation pool is NOT set 2025-10-16 04:17:41,543 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=14.0 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 1 received, 50% packet loss, time 1020ms\r\nrtt min/avg/max/mdev = 14.014/14.014/14.014/0.000 ms') 2025-10-16 04:17:41,543 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-38862421 passed 2025-10-16 04:17:41,622 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:17:41,623 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-38862421, IP 192.168.1.3, Port 22 2025-10-16 04:17:41,691 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-16 04:17:41,818 - DEBUG - VM ctest-vn1_vm1-38862421 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-16 04:17:46,818 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:17:46,819 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-38862421, IP 192.168.1.3, Port 22 2025-10-16 04:17:46,878 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-16 04:17:46,968 - DEBUG - VM ctest-vn1_vm1-38862421 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-16 04:17:51,969 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:17:51,969 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-38862421, IP 192.168.1.3, Port 22 2025-10-16 04:17:52,040 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-16 04:17:52,125 - DEBUG - VM ctest-vn1_vm1-38862421 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-16 04:17:57,126 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:17:57,126 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-38862421, IP 192.168.1.3, Port 22 2025-10-16 04:17:57,181 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-16 04:17:57,266 - DEBUG - VM ctest-vn1_vm1-38862421 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-16 04:18:02,267 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:18:02,267 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-38862421, IP 192.168.1.3, Port 22 2025-10-16 04:18:02,437 - DEBUG - VM ctest-vn1_vm1-38862421 is ready for SSH connections 2025-10-16 04:18:02,437 - INFO - Waiting for VM ctest-vn1_vm2-11265942 to be up.. 2025-10-16 04:18:02,527 - DEBUG - VM is in ACTIVE state now 2025-10-16 04:18:02,527 - INFO - VM name : ctest-vn1_vm2-11265942 2025-10-16 04:18:02,611 - DEBUG - VM ctest-vn1_vm2-11265942 ID is cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 2025-10-16 04:18:02,612 - DEBUG - VM ctest-vn1_vm2-11265942 launched on Node an-jenkins-deploy-platform-ansible-os-4339-2 2025-10-16 04:18:02,698 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 2025-10-16 04:18:02,709 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/69242d8e-a338-4f55-b538-ad5d0123414b 2025-10-16 04:18:03,869 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=7.15 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.937 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.937/4.042/7.147/3.105 ms') 2025-10-16 04:18:03,869 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm2-11265942 passed 2025-10-16 04:18:03,949 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:18:03,949 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-11265942, IP 192.168.1.4, Port 22 2025-10-16 04:18:04,016 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-16 04:18:04,104 - DEBUG - VM ctest-vn1_vm2-11265942 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-16 04:18:09,105 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:18:09,105 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-11265942, IP 192.168.1.4, Port 22 2025-10-16 04:18:09,171 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-16 04:18:09,256 - DEBUG - VM ctest-vn1_vm2-11265942 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-16 04:18:14,257 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:18:14,257 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-11265942, IP 192.168.1.4, Port 22 2025-10-16 04:18:14,423 - DEBUG - VM ctest-vn1_vm2-11265942 is ready for SSH connections 2025-10-16 04:18:14,423 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:18:14,423 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.38, gateway password: c0ntrail123 2025-10-16 04:18:14,423 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4 2025-10-16 04:18:55,109 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data. 64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=11.5 ms 64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.06 ms 64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.24 ms --- 192.168.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 1.065/4.614/11.533/4.893 ms 2025-10-16 04:18:55,109 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-38862421 passed 2025-10-16 04:18:55,109 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:18:55,109 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.129, gateway password: c0ntrail123 2025-10-16 04:18:55,109 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3 2025-10-16 04:19:42,365 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. 64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=7.81 ms 64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.16 ms 64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.63 ms --- 192.168.1.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.166/3.539/7.818/3.032 ms 2025-10-16 04:19:42,365 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-11265942 passed 2025-10-16 04:19:42,371 - INFO - Active control node from the Agent 10.0.0.38 is 10.0.0.23 2025-10-16 04:19:42,372 - INFO - Stoping the Control service in 10.0.0.23 2025-10-16 04:19:42,751 - INFO - 10.0.0.23 2025-10-16 04:19:42,751 - INFO - control:inactive 2025-10-16 04:19:42,751 - INFO - Contrail services control are down on nodes 10.0.0.23 2025-10-16 04:19:47,759 - INFO - Active control node from the Agent 10.0.0.38 is 10.0.0.57 2025-10-16 04:19:47,759 - INFO - Starting the Control service in 10.0.0.23 2025-10-16 04:19:48,063 - INFO - 10.0.0.23 2025-10-16 04:19:48,075 - INFO - control:initializing 2025-10-16 04:19:48,075 - DEBUG - defaultdict(, {'10.0.0.23': {'control': {'status': 'initializing', 'description': None}}}) 2025-10-16 04:19:48,075 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-16 04:19:53,167 - INFO - 10.0.0.23 2025-10-16 04:19:53,173 - INFO - control:active 2025-10-16 04:19:53,173 - INFO - Contrail services ['control'] are up on nodes 10.0.0.23 2025-10-16 04:19:58,275 - DEBUG - VM is in ACTIVE state now 2025-10-16 04:19:58,275 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:19:58,275 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/e648138f-ddc6-4577-932b-ecbff466899b 2025-10-16 04:19:58,284 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/5fe3521f-072b-4936-9383-00e5809f22f6 2025-10-16 04:19:58,294 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/e648138f-ddc6-4577-932b-ecbff466899b 2025-10-16 04:19:58,301 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/5fe3521f-072b-4936-9383-00e5809f22f6 2025-10-16 04:19:58,310 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/2aaf0a1e-e4ca-40a8-af90-6b13732b12b6 2025-10-16 04:19:58,321 - DEBUG - Verifying in api server 10.0.0.23 2025-10-16 04:19:58,321 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/e648138f-ddc6-4577-932b-ecbff466899b 2025-10-16 04:19:58,328 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/5fe3521f-072b-4936-9383-00e5809f22f6 2025-10-16 04:19:58,337 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/2aaf0a1e-e4ca-40a8-af90-6b13732b12b6 2025-10-16 04:19:58,345 - INFO - VM ctest-vn1_vm1-38862421 verfication in all API Servers passed 2025-10-16 04:19:58,364 - DEBUG - VM ctest-vn1_vm1-38862421 Tap interface: {'index': '3', 'name': 'tap5fe3521f-07', 'uuid': '5fe3521f-072b-4936-9383-00e5809f22f6', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444:ctest-vn1-36242444', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '12', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444', 'vm_uuid': 'e648138f-ddc6-4577-932b-ecbff466899b', 'vm_name': 'ctest-vn1_vm1-38862421', 'ip_addr': '192.168.1.3', 'mac_addr': '02:5f:e3:52:1f:07', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '161', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-91100301:5fe3521f-072b-4936-9383-00e5809f22f6', 'sg_uuid_list': ['d00d1c1c-903e-4107-af98-1b6cf220bd38'], 'static_route_list': None, 'vm_project_uuid': '03e29f65-50a8-4f9b-937e-9a83abd9c7d6', 'admin_state': 'Enabled', 'flow_key_idx': '23', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-16 04:19:58,364 - DEBUG - Agent 10.0.0.38 vrf name: default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444:ctest-vn1-36242444 2025-10-16 04:19:58,369 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444:ctest-vn1-36242444', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444', 'table_label': '-1', 'vxlan_id': '12', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.145:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-10-16 04:19:58,449 - DEBUG - VM ctest-vn1_vm1-38862421 : Tap interface tap5fe3521f-07 is set to Active 2025-10-16 04:19:58,449 - DEBUG - Tap interface tap5fe3521f-07 detail : {'index': '3', 'name': 'tap5fe3521f-07', 'uuid': '5fe3521f-072b-4936-9383-00e5809f22f6', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444:ctest-vn1-36242444', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '12', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444', 'vm_uuid': 'e648138f-ddc6-4577-932b-ecbff466899b', 'vm_name': 'ctest-vn1_vm1-38862421', 'ip_addr': '192.168.1.3', 'mac_addr': '02:5f:e3:52:1f:07', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '161', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-91100301:5fe3521f-072b-4936-9383-00e5809f22f6', 'sg_uuid_list': ['d00d1c1c-903e-4107-af98-1b6cf220bd38'], 'static_route_list': None, 'vm_project_uuid': '03e29f65-50a8-4f9b-937e-9a83abd9c7d6', 'admin_state': 'Enabled', 'flow_key_idx': '23', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-16 04:19:58,559 - DEBUG - Starting Layer 2 verification in Agent 2025-10-16 04:19:58,571 - DEBUG - Layer 2 path is seen for VM MAC 02:5f:e3:52:1f:07 in agent 10.0.0.38 2025-10-16 04:19:58,572 - DEBUG - Active layer 2 route in agent is present for VMI tap5fe3521f-07 2025-10-16 04:19:58,572 - DEBUG - L2 label(29) matches bw route table and interface table 2025-10-16 04:19:58,580 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 is found in Agent of node 10.0.0.38 2025-10-16 04:19:58,708 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 is consistent in agent 10.0.0.38 2025-10-16 04:19:58,708 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.38 2025-10-16 04:19:58,708 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 verification for VM ctest-vn1_vm1-38862421 in Agent 10.0.0.38 passed 2025-10-16 04:19:58,816 - DEBUG - Starting all layer 2 verification in agent 10.0.0.38 2025-10-16 04:19:58,827 - DEBUG - Route for VM MAC 02:5f:e3:52:1f:07 is consistent in agent 10.0.0.38 2025-10-16 04:19:58,832 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 is found in Agent of node 10.0.0.129 2025-10-16 04:19:58,957 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 is consistent in agent 10.0.0.129 2025-10-16 04:19:58,958 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.129 2025-10-16 04:19:58,958 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 verification for VM ctest-vn1_vm1-38862421 in Agent 10.0.0.129 passed 2025-10-16 04:19:59,071 - DEBUG - Starting all layer 2 verification in agent 10.0.0.129 2025-10-16 04:19:59,081 - DEBUG - Route for VM MAC 02:5f:e3:52:1f:07 is consistent in agent 10.0.0.129 2025-10-16 04:19:59,081 - INFO - VM ctest-vn1_vm1-38862421 verifications in Compute nodes passed 2025-10-16 04:19:59,316 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.38 2025-10-16 04:19:59,322 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-10-16 04:19:59,437 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.129 2025-10-16 04:19:59,441 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.145, 24; Got 10.20.0.145,24 2025-10-16 04:19:59,442 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.129 2025-10-16 04:19:59,442 - INFO - Validated routes of VM ctest-vn1_vm1-38862421 in all vrouters 2025-10-16 04:19:59,948 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2025-10-16 04:20:00,072 - DEBUG - Layer2 route found for VM MAC 02:5f:e3:52:1f:07 in Control-node 10.0.0.23 2025-10-16 04:20:00,072 - DEBUG - L2 Label for VM ctest-vn1_vm1-38862421 same between Control-node 10.0.0.23 and Agent, Expected: 29, Seen: 29 2025-10-16 04:20:00,072 - DEBUG - Starting all layer2 verification in 10.0.0.57 Control Node 2025-10-16 04:20:00,203 - DEBUG - Layer2 route found for VM MAC 02:5f:e3:52:1f:07 in Control-node 10.0.0.57 2025-10-16 04:20:00,203 - DEBUG - L2 Label for VM ctest-vn1_vm1-38862421 same between Control-node 10.0.0.57 and Agent, Expected: 29, Seen: 29 2025-10-16 04:20:00,203 - INFO - Verification in Control-nodes for VM ctest-vn1_vm1-38862421 passed 2025-10-16 04:20:00,204 - DEBUG - Verifying the vm in opserver 2025-10-16 04:20:00,204 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-10-16 04:20:00,204 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/e648138f-ddc6-4577-932b-ecbff466899b?flat 2025-10-16 04:20:00,225 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91100301:5fe3521f-072b-4936-9383-00e5809f22f6?flat 2025-10-16 04:20:00,246 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91100301:5fe3521f-072b-4936-9383-00e5809f22f6?flat 2025-10-16 04:20:00,306 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/e648138f-ddc6-4577-932b-ecbff466899b?flat 2025-10-16 04:20:00,365 - DEBUG - Verifying vm in vn uve 2025-10-16 04:20:00,365 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91100301:5fe3521f-072b-4936-9383-00e5809f22f6?flat 2025-10-16 04:20:00,454 - DEBUG - VM uve shows interface as {'ip_address': '192.168.1.3', 'ip4_active': True, 'admin_state': True, 'gateway': '192.168.1.1', 'uuid': '5fe3521f-072b-4936-9383-00e5809f22f6', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444', 'vm_uuid': 'e648138f-ddc6-4577-932b-ecbff466899b', 'vm_name': 'ctest-vn1_vm1-38862421', '__T': 1760588225492941, 'port_mirror_enabled': False, 'tx_vlan': 65535, 'rx_vlan': 65535, 'fixed_ip4_list': ['192.168.1.3'], 'label': 24, 'is_health_check_active': True, 'mac_address': '02:5f:e3:52:1f:07', 'vhostuser_mode': 0, 'active': True, 'ip6_active': False, 'vn_uuid': '30e1ddf2-af9d-43e1-8c79-10950ad554ba', 'ip6_address': '::', 'l2_active': True} 2025-10-16 04:20:00,454 - DEBUG - VM uve shows ip address as ['192.168.1.3', '::'] 2025-10-16 04:20:00,454 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 2025-10-16 04:20:00,454 - INFO - Verifying through opserver in 10.0.0.19 2025-10-16 04:20:00,454 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444?flat 2025-10-16 04:20:00,483 - DEBUG - expected vm list e648138f-ddc6-4577-932b-ecbff466899b 2025-10-16 04:20:00,484 - DEBUG - Extracted vm list ['e648138f-ddc6-4577-932b-ecbff466899b'] 2025-10-16 04:20:00,484 - INFO - VM e648138f-ddc6-4577-932b-ecbff466899b is present in default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 2025-10-16 04:20:00,484 - DEBUG - Verifying vm in vrouter uve 2025-10-16 04:20:00,484 - DEBUG - Getting info from collector 10.0.0.23.. 2025-10-16 04:20:00,484 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/e648138f-ddc6-4577-932b-ecbff466899b?flat 2025-10-16 04:20:00,560 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4339-1'] 2025-10-16 04:20:00,560 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-1:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:20:00,605 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-1?flat 2025-10-16 04:20:00,635 - DEBUG - VM e648138f-ddc6-4577-932b-ecbff466899b is present in vrouter an-jenkins-deploy-platform-ansible-os-4339-1 uve 2025-10-16 04:20:00,635 - DEBUG - Validated that VM ctest-vn1_vm1-38862421 is in Vrouter an-jenkins-deploy-platform-ansible-os-4339-1 UVE 2025-10-16 04:20:00,635 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-10-16 04:20:00,644 - DEBUG - Expected tap interface of VM uuid e648138f-ddc6-4577-932b-ecbff466899b is default-domain:ctest-TestBasicVMVN0-91100301:5fe3521f-072b-4936-9383-00e5809f22f6 2025-10-16 04:20:00,644 - DEBUG - Expected VN of VM uuid e648138f-ddc6-4577-932b-ecbff466899b is default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 2025-10-16 04:20:00,644 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-1:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:20:00,695 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-1?flat 2025-10-16 04:20:00,727 - DEBUG - VM e648138f-ddc6-4577-932b-ecbff466899b is present in vrouter an-jenkins-deploy-platform-ansible-os-4339-1 uve 2025-10-16 04:20:00,727 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-91100301:5fe3521f-072b-4936-9383-00e5809f22f6 of vm e648138f-ddc6-4577-932b-ecbff466899b is present in vrouter an-jenkins-deploy-platform-ansible-os-4339-1 uve 2025-10-16 04:20:00,728 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 of vm e648138f-ddc6-4577-932b-ecbff466899b is present in vrouter an-jenkins-deploy-platform-ansible-os-4339-1 uve 2025-10-16 04:20:00,728 - DEBUG - Verifying the e648138f-ddc6-4577-932b-ecbff466899b virtual network link through opserver 10.0.0.19 2025-10-16 04:20:00,728 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machines 2025-10-16 04:20:00,745 - DEBUG - VM link and name as {'name': 'e648138f-ddc6-4577-932b-ecbff466899b', 'href': 'http://10.0.0.19:8081/analytics/uves/virtual-machine/e648138f-ddc6-4577-932b-ecbff466899b?flat'} 2025-10-16 04:20:00,745 - DEBUG - Verifying the e648138f-ddc6-4577-932b-ecbff466899b virtual network link through opserver 10.0.0.57 2025-10-16 04:20:00,745 - DEBUG - Requesting: http://10.0.0.57:8081/analytics/uves/virtual-machines 2025-10-16 04:20:00,762 - DEBUG - VM link and name as {'name': 'e648138f-ddc6-4577-932b-ecbff466899b', 'href': 'http://10.0.0.57:8081/analytics/uves/virtual-machine/e648138f-ddc6-4577-932b-ecbff466899b?flat'} 2025-10-16 04:20:00,762 - DEBUG - Verifying the e648138f-ddc6-4577-932b-ecbff466899b virtual network link through opserver 10.0.0.23 2025-10-16 04:20:00,762 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-10-16 04:20:00,775 - DEBUG - VM link and name as {'name': 'e648138f-ddc6-4577-932b-ecbff466899b', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/e648138f-ddc6-4577-932b-ecbff466899b?flat'} 2025-10-16 04:20:00,775 - INFO - VM ctest-vn1_vm1-38862421 validations in Opserver passed 2025-10-16 04:20:00,878 - DEBUG - VM is in ACTIVE state now 2025-10-16 04:20:00,878 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:20:00,878 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 2025-10-16 04:20:00,886 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/69242d8e-a338-4f55-b538-ad5d0123414b 2025-10-16 04:20:00,897 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 2025-10-16 04:20:00,906 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/69242d8e-a338-4f55-b538-ad5d0123414b 2025-10-16 04:20:00,916 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/734a38af-d352-418d-945c-a725d0414041 2025-10-16 04:20:00,925 - DEBUG - Verifying in api server 10.0.0.23 2025-10-16 04:20:00,926 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 2025-10-16 04:20:00,934 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/69242d8e-a338-4f55-b538-ad5d0123414b 2025-10-16 04:20:00,945 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/734a38af-d352-418d-945c-a725d0414041 2025-10-16 04:20:00,952 - INFO - VM ctest-vn1_vm2-11265942 verfication in all API Servers passed 2025-10-16 04:20:00,975 - DEBUG - VM ctest-vn1_vm2-11265942 Tap interface: {'index': '3', 'name': 'tap69242d8e-a3', 'uuid': '69242d8e-a338-4f55-b538-ad5d0123414b', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444:ctest-vn1-36242444', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '12', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444', 'vm_uuid': 'cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16', 'vm_name': 'ctest-vn1_vm2-11265942', 'ip_addr': '192.168.1.4', 'mac_addr': '02:69:24:2d:8e:a3', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '127', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-91100301:69242d8e-a338-4f55-b538-ad5d0123414b', 'sg_uuid_list': ['d00d1c1c-903e-4107-af98-1b6cf220bd38'], 'static_route_list': None, 'vm_project_uuid': '03e29f65-50a8-4f9b-937e-9a83abd9c7d6', 'admin_state': 'Enabled', 'flow_key_idx': '29', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-16 04:20:00,975 - DEBUG - Agent 10.0.0.129 vrf name: default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444:ctest-vn1-36242444 2025-10-16 04:20:00,983 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444:ctest-vn1-36242444', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444', 'table_label': '-1', 'vxlan_id': '12', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.19:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-10-16 04:20:01,070 - DEBUG - VM ctest-vn1_vm2-11265942 : Tap interface tap69242d8e-a3 is set to Active 2025-10-16 04:20:01,071 - DEBUG - Tap interface tap69242d8e-a3 detail : {'index': '3', 'name': 'tap69242d8e-a3', 'uuid': '69242d8e-a338-4f55-b538-ad5d0123414b', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444:ctest-vn1-36242444', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '12', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444', 'vm_uuid': 'cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16', 'vm_name': 'ctest-vn1_vm2-11265942', 'ip_addr': '192.168.1.4', 'mac_addr': '02:69:24:2d:8e:a3', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '127', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-91100301:69242d8e-a338-4f55-b538-ad5d0123414b', 'sg_uuid_list': ['d00d1c1c-903e-4107-af98-1b6cf220bd38'], 'static_route_list': None, 'vm_project_uuid': '03e29f65-50a8-4f9b-937e-9a83abd9c7d6', 'admin_state': 'Enabled', 'flow_key_idx': '29', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-16 04:20:01,192 - DEBUG - Starting Layer 2 verification in Agent 2025-10-16 04:20:01,206 - DEBUG - Layer 2 path is seen for VM MAC 02:69:24:2d:8e:a3 in agent 10.0.0.129 2025-10-16 04:20:01,207 - DEBUG - Active layer 2 route in agent is present for VMI tap69242d8e-a3 2025-10-16 04:20:01,207 - DEBUG - L2 label(29) matches bw route table and interface table 2025-10-16 04:20:01,217 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 is found in Agent of node 10.0.0.38 2025-10-16 04:20:01,362 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 is consistent in agent 10.0.0.38 2025-10-16 04:20:01,362 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.38 2025-10-16 04:20:01,362 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 verification for VM ctest-vn1_vm2-11265942 in Agent 10.0.0.38 passed 2025-10-16 04:20:01,487 - DEBUG - Starting all layer 2 verification in agent 10.0.0.38 2025-10-16 04:20:01,498 - DEBUG - Route for VM MAC 02:69:24:2d:8e:a3 is consistent in agent 10.0.0.38 2025-10-16 04:20:01,504 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 is found in Agent of node 10.0.0.129 2025-10-16 04:20:01,631 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 is consistent in agent 10.0.0.129 2025-10-16 04:20:01,632 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.129 2025-10-16 04:20:01,632 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 verification for VM ctest-vn1_vm2-11265942 in Agent 10.0.0.129 passed 2025-10-16 04:20:01,744 - DEBUG - Starting all layer 2 verification in agent 10.0.0.129 2025-10-16 04:20:01,757 - DEBUG - Route for VM MAC 02:69:24:2d:8e:a3 is consistent in agent 10.0.0.129 2025-10-16 04:20:01,757 - INFO - VM ctest-vn1_vm2-11265942 verifications in Compute nodes passed 2025-10-16 04:20:02,011 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.38 2025-10-16 04:20:02,016 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.19, 25; Got 10.20.0.19,25 2025-10-16 04:20:02,016 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.38 2025-10-16 04:20:02,130 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.129 2025-10-16 04:20:02,135 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-10-16 04:20:02,135 - INFO - Validated routes of VM ctest-vn1_vm2-11265942 in all vrouters 2025-10-16 04:20:02,627 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2025-10-16 04:20:02,760 - DEBUG - Layer2 route found for VM MAC 02:69:24:2d:8e:a3 in Control-node 10.0.0.23 2025-10-16 04:20:02,761 - DEBUG - L2 Label for VM ctest-vn1_vm2-11265942 same between Control-node 10.0.0.23 and Agent, Expected: 29, Seen: 29 2025-10-16 04:20:02,761 - DEBUG - Starting all layer2 verification in 10.0.0.57 Control Node 2025-10-16 04:20:02,888 - DEBUG - Layer2 route found for VM MAC 02:69:24:2d:8e:a3 in Control-node 10.0.0.57 2025-10-16 04:20:02,888 - DEBUG - L2 Label for VM ctest-vn1_vm2-11265942 same between Control-node 10.0.0.57 and Agent, Expected: 29, Seen: 29 2025-10-16 04:20:02,888 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-11265942 passed 2025-10-16 04:20:02,888 - DEBUG - Verifying the vm in opserver 2025-10-16 04:20:02,888 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-10-16 04:20:02,888 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16?flat 2025-10-16 04:20:02,907 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91100301:69242d8e-a338-4f55-b538-ad5d0123414b?flat 2025-10-16 04:20:02,928 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91100301:69242d8e-a338-4f55-b538-ad5d0123414b?flat 2025-10-16 04:20:02,947 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16?flat 2025-10-16 04:20:02,963 - DEBUG - Verifying vm in vn uve 2025-10-16 04:20:02,963 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91100301:69242d8e-a338-4f55-b538-ad5d0123414b?flat 2025-10-16 04:20:02,980 - DEBUG - VM uve shows interface as {'is_health_check_active': True, 'vn_uuid': '30e1ddf2-af9d-43e1-8c79-10950ad554ba', 'mac_address': '02:69:24:2d:8e:a3', 'gateway': '192.168.1.1', 'ip_address': '192.168.1.4', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444', 'ip6_address': '::', 'l2_active': True, 'rx_vlan': 65535, 'label': 25, 'tx_vlan': 65535, 'admin_state': True, 'active': True, 'ip4_active': True, 'port_mirror_enabled': False, 'vm_name': 'ctest-vn1_vm2-11265942', 'vm_uuid': 'cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16', '__T': 1760588384637577, 'vhostuser_mode': 0, 'uuid': '69242d8e-a338-4f55-b538-ad5d0123414b', 'fixed_ip4_list': ['192.168.1.4'], 'ip6_active': False} 2025-10-16 04:20:02,981 - DEBUG - VM uve shows ip address as ['192.168.1.4', '::'] 2025-10-16 04:20:02,981 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 2025-10-16 04:20:02,981 - INFO - Verifying through opserver in 10.0.0.19 2025-10-16 04:20:02,981 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444?flat 2025-10-16 04:20:03,008 - DEBUG - expected vm list cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 2025-10-16 04:20:03,008 - DEBUG - Extracted vm list ['cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16'] 2025-10-16 04:20:03,008 - INFO - VM cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 is present in default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 2025-10-16 04:20:03,008 - DEBUG - Verifying vm in vrouter uve 2025-10-16 04:20:03,009 - DEBUG - Getting info from collector 10.0.0.23.. 2025-10-16 04:20:03,009 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16?flat 2025-10-16 04:20:03,024 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4339-2'] 2025-10-16 04:20:03,025 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-2:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:20:03,060 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-2?flat 2025-10-16 04:20:03,090 - DEBUG - VM cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 is present in vrouter an-jenkins-deploy-platform-ansible-os-4339-2 uve 2025-10-16 04:20:03,090 - DEBUG - Validated that VM ctest-vn1_vm2-11265942 is in Vrouter an-jenkins-deploy-platform-ansible-os-4339-2 UVE 2025-10-16 04:20:03,090 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-10-16 04:20:03,098 - DEBUG - Expected tap interface of VM uuid cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 is default-domain:ctest-TestBasicVMVN0-91100301:69242d8e-a338-4f55-b538-ad5d0123414b 2025-10-16 04:20:03,098 - DEBUG - Expected VN of VM uuid cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 is default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 2025-10-16 04:20:03,098 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-2:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:20:03,134 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-2?flat 2025-10-16 04:20:03,164 - DEBUG - VM cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 is present in vrouter an-jenkins-deploy-platform-ansible-os-4339-2 uve 2025-10-16 04:20:03,164 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-91100301:69242d8e-a338-4f55-b538-ad5d0123414b of vm cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 is present in vrouter an-jenkins-deploy-platform-ansible-os-4339-2 uve 2025-10-16 04:20:03,164 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 of vm cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 is present in vrouter an-jenkins-deploy-platform-ansible-os-4339-2 uve 2025-10-16 04:20:03,164 - DEBUG - Verifying the cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 virtual network link through opserver 10.0.0.19 2025-10-16 04:20:03,164 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machines 2025-10-16 04:20:03,178 - DEBUG - VM link and name as {'name': 'cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16', 'href': 'http://10.0.0.19:8081/analytics/uves/virtual-machine/cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16?flat'} 2025-10-16 04:20:03,178 - DEBUG - Verifying the cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 virtual network link through opserver 10.0.0.57 2025-10-16 04:20:03,178 - DEBUG - Requesting: http://10.0.0.57:8081/analytics/uves/virtual-machines 2025-10-16 04:20:03,243 - DEBUG - VM link and name as {'name': 'cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16', 'href': 'http://10.0.0.57:8081/analytics/uves/virtual-machine/cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16?flat'} 2025-10-16 04:20:03,243 - DEBUG - Verifying the cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 virtual network link through opserver 10.0.0.23 2025-10-16 04:20:03,244 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-10-16 04:20:03,258 - DEBUG - VM link and name as {'name': 'cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16?flat'} 2025-10-16 04:20:03,258 - INFO - VM ctest-vn1_vm2-11265942 validations in Opserver passed 2025-10-16 04:20:03,258 - INFO - Checking the ping between the VM again 2025-10-16 04:20:03,258 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:20:03,258 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.38, gateway password: c0ntrail123 2025-10-16 04:20:03,258 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4 2025-10-16 04:20:05,736 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data. 64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=3.33 ms 64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.38 ms 64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.09 ms --- 192.168.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 1.098/1.941/3.339/0.996 ms 2025-10-16 04:20:05,736 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-38862421 passed 2025-10-16 04:20:05,736 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:20:05,736 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.129, gateway password: c0ntrail123 2025-10-16 04:20:05,736 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3 2025-10-16 04:20:08,292 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. 64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=2.55 ms 64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.24 ms 64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.10 ms --- 192.168.1.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.100/1.635/2.559/0.656 ms 2025-10-16 04:20:08,292 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-11265942 passed 2025-10-16 04:20:08,292 - INFO - Deleting VM ctest-vn1_vm2-11265942 2025-10-16 04:20:08,367 - DEBUG - Verifying in api server 10.0.0.23 2025-10-16 04:20:08,368 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 2025-10-16 04:20:08,378 - DEBUG - VM ID cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 of VM ctest-vn1_vm2-11265942 is still found in API Server 2025-10-16 04:20:10,378 - DEBUG - Verifying in api server 10.0.0.23 2025-10-16 04:20:10,379 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 2025-10-16 04:20:10,387 - DEBUG - Response Code: 404 2025-10-16 04:20:10,387 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 2025-10-16 04:20:10,394 - DEBUG - Response Code: 404 2025-10-16 04:20:10,394 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 2025-10-16 04:20:10,400 - DEBUG - Response Code: 404 2025-10-16 04:20:10,401 - INFO - VM ctest-vn1_vm2-11265942 is fully removed in API-Server 2025-10-16 04:20:10,569 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-10-16 04:20:10,737 - DEBUG - VM route 192.168.1.4 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 of compute 10.0.0.38 2025-10-16 04:20:10,742 - DEBUG - Validated that vrouter 10.0.0.38 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 2025-10-16 04:20:10,742 - DEBUG - Validated that vrouter 10.0.0.129 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 2025-10-16 04:20:10,742 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 2025-10-16 04:20:10,743 - INFO - VM ctest-vn1_vm2-11265942 is removed in Compute, and routes are removed in all compute nodes 2025-10-16 04:20:10,994 - INFO - Routes for VM ctest-vn1_vm2-11265942 is removed in all control-nodes 2025-10-16 04:20:10,995 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-2:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:20:11,042 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-2?flat 2025-10-16 04:20:11,069 - DEBUG - VM cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4339-2 uve 2025-10-16 04:20:11,069 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-16 04:20:11,069 - DEBUG - interface for vm cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 deleted from vrouter uve ... 2025-10-16 04:20:11,069 - DEBUG - Deleted interface not in error interface list ... 2025-10-16 04:20:14,070 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-2:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:20:14,102 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-2?flat 2025-10-16 04:20:14,129 - DEBUG - VM cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4339-2 uve 2025-10-16 04:20:14,129 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-16 04:20:14,129 - DEBUG - interface for vm cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 deleted from vrouter uve ... 2025-10-16 04:20:14,129 - DEBUG - Deleted interface not in error interface list ... 2025-10-16 04:20:17,130 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-2:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:20:17,164 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-2?flat 2025-10-16 04:20:17,194 - DEBUG - VM cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 is not present in vrouter an-jenkins-deploy-platform-ansible-os-4339-2 uve 2025-10-16 04:20:17,194 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-16 04:20:17,194 - DEBUG - interface for vm cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 deleted from vrouter uve ... 2025-10-16 04:20:17,194 - DEBUG - Deleted interface not in error interface list ... 2025-10-16 04:20:17,194 - DEBUG - Validated that VM cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 is removed in Vrouter UVE 2025-10-16 04:20:17,194 - DEBUG - Verifying through opserver in 10.0.0.19 2025-10-16 04:20:17,194 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444?flat 2025-10-16 04:20:17,217 - DEBUG - cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 BM not in default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 uve 2025-10-16 04:20:17,217 - DEBUG - Validated that VM cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 is not present in the VN UVE 2025-10-16 04:20:17,217 - DEBUG - Verifying the cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 virtual network link through opserver 10.0.0.19 2025-10-16 04:20:17,217 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machines 2025-10-16 04:20:17,228 - INFO - cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 vm uve deleted from opserver 2025-10-16 04:20:17,228 - DEBUG - Verifying the cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 virtual network link through opserver 10.0.0.57 2025-10-16 04:20:17,228 - DEBUG - Requesting: http://10.0.0.57:8081/analytics/uves/virtual-machines 2025-10-16 04:20:17,281 - INFO - cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 vm uve deleted from opserver 2025-10-16 04:20:17,281 - DEBUG - Verifying the cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 virtual network link through opserver 10.0.0.23 2025-10-16 04:20:17,281 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-10-16 04:20:17,294 - INFO - cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 vm uve deleted from opserver 2025-10-16 04:20:17,294 - INFO - Validated that VM cbeedfc0-c6e0-47b4-81fa-5338d1bdbd16 is removed from Opserver 2025-10-16 04:20:17,294 - INFO - Deleting VM ctest-vn1_vm1-38862421 2025-10-16 04:20:17,368 - DEBUG - Verifying in api server 10.0.0.23 2025-10-16 04:20:17,368 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/e648138f-ddc6-4577-932b-ecbff466899b 2025-10-16 04:20:17,377 - DEBUG - VM ID e648138f-ddc6-4577-932b-ecbff466899b of VM ctest-vn1_vm1-38862421 is still found in API Server 2025-10-16 04:20:19,377 - DEBUG - Verifying in api server 10.0.0.23 2025-10-16 04:20:19,378 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/e648138f-ddc6-4577-932b-ecbff466899b 2025-10-16 04:20:19,385 - DEBUG - Response Code: 404 2025-10-16 04:20:19,386 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/e648138f-ddc6-4577-932b-ecbff466899b 2025-10-16 04:20:19,392 - DEBUG - Response Code: 404 2025-10-16 04:20:19,392 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/e648138f-ddc6-4577-932b-ecbff466899b 2025-10-16 04:20:19,397 - DEBUG - Response Code: 404 2025-10-16 04:20:19,397 - INFO - VM ctest-vn1_vm1-38862421 is fully removed in API-Server 2025-10-16 04:20:19,554 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-10-16 04:20:19,596 - DEBUG - Validated that vrouter 10.0.0.38 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 2025-10-16 04:20:19,606 - DEBUG - Validated that vrouter 10.0.0.129 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 2025-10-16 04:20:19,607 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 2025-10-16 04:20:19,607 - INFO - VM ctest-vn1_vm1-38862421 is removed in Compute, and routes are removed in all compute nodes 2025-10-16 04:20:19,838 - INFO - Routes for VM ctest-vn1_vm1-38862421 is removed in all control-nodes 2025-10-16 04:20:19,838 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-1:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:20:19,883 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-1?flat 2025-10-16 04:20:19,918 - DEBUG - VM e648138f-ddc6-4577-932b-ecbff466899b is still present in vrouter an-jenkins-deploy-platform-ansible-os-4339-1 uve 2025-10-16 04:20:19,919 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-16 04:20:19,919 - DEBUG - interface for vm e648138f-ddc6-4577-932b-ecbff466899b deleted from vrouter uve ... 2025-10-16 04:20:19,919 - DEBUG - Deleted interface not in error interface list ... 2025-10-16 04:20:22,919 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-1:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:20:22,971 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-1?flat 2025-10-16 04:20:23,005 - DEBUG - VM e648138f-ddc6-4577-932b-ecbff466899b is still present in vrouter an-jenkins-deploy-platform-ansible-os-4339-1 uve 2025-10-16 04:20:23,005 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-16 04:20:23,005 - DEBUG - interface for vm e648138f-ddc6-4577-932b-ecbff466899b deleted from vrouter uve ... 2025-10-16 04:20:23,005 - DEBUG - Deleted interface not in error interface list ... 2025-10-16 04:20:26,006 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-1:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:20:26,054 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-1?flat 2025-10-16 04:20:26,086 - DEBUG - VM e648138f-ddc6-4577-932b-ecbff466899b is still present in vrouter an-jenkins-deploy-platform-ansible-os-4339-1 uve 2025-10-16 04:20:26,086 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-16 04:20:26,086 - DEBUG - interface for vm e648138f-ddc6-4577-932b-ecbff466899b deleted from vrouter uve ... 2025-10-16 04:20:26,087 - DEBUG - Deleted interface not in error interface list ... 2025-10-16 04:20:29,087 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-1:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:20:29,137 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-1?flat 2025-10-16 04:20:29,170 - DEBUG - VM e648138f-ddc6-4577-932b-ecbff466899b is still present in vrouter an-jenkins-deploy-platform-ansible-os-4339-1 uve 2025-10-16 04:20:29,170 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-16 04:20:29,170 - DEBUG - interface for vm e648138f-ddc6-4577-932b-ecbff466899b deleted from vrouter uve ... 2025-10-16 04:20:29,170 - DEBUG - Deleted interface not in error interface list ... 2025-10-16 04:20:32,171 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-1:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:20:32,218 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-1?flat 2025-10-16 04:20:32,250 - DEBUG - VM e648138f-ddc6-4577-932b-ecbff466899b is still present in vrouter an-jenkins-deploy-platform-ansible-os-4339-1 uve 2025-10-16 04:20:32,250 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-16 04:20:32,250 - DEBUG - interface for vm e648138f-ddc6-4577-932b-ecbff466899b deleted from vrouter uve ... 2025-10-16 04:20:32,250 - DEBUG - Deleted interface not in error interface list ... 2025-10-16 04:20:35,250 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-1:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:20:35,299 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-1?flat 2025-10-16 04:20:35,333 - DEBUG - VM e648138f-ddc6-4577-932b-ecbff466899b is still present in vrouter an-jenkins-deploy-platform-ansible-os-4339-1 uve 2025-10-16 04:20:35,333 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-16 04:20:35,333 - DEBUG - interface for vm e648138f-ddc6-4577-932b-ecbff466899b deleted from vrouter uve ... 2025-10-16 04:20:35,333 - DEBUG - Deleted interface not in error interface list ... 2025-10-16 04:20:38,334 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-1:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:20:38,385 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-1?flat 2025-10-16 04:20:38,413 - DEBUG - VM e648138f-ddc6-4577-932b-ecbff466899b is not present in vrouter an-jenkins-deploy-platform-ansible-os-4339-1 uve 2025-10-16 04:20:38,413 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-16 04:20:38,413 - DEBUG - interface for vm e648138f-ddc6-4577-932b-ecbff466899b deleted from vrouter uve ... 2025-10-16 04:20:38,413 - DEBUG - Deleted interface not in error interface list ... 2025-10-16 04:20:38,413 - DEBUG - Validated that VM e648138f-ddc6-4577-932b-ecbff466899b is removed in Vrouter UVE 2025-10-16 04:20:38,413 - DEBUG - Verifying through opserver in 10.0.0.19 2025-10-16 04:20:38,413 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444?flat 2025-10-16 04:20:38,430 - DEBUG - e648138f-ddc6-4577-932b-ecbff466899b BM not in default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-36242444 uve 2025-10-16 04:20:38,430 - DEBUG - Validated that VM e648138f-ddc6-4577-932b-ecbff466899b is not present in the VN UVE 2025-10-16 04:20:38,430 - DEBUG - Verifying the e648138f-ddc6-4577-932b-ecbff466899b virtual network link through opserver 10.0.0.19 2025-10-16 04:20:38,430 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machines 2025-10-16 04:20:38,444 - INFO - e648138f-ddc6-4577-932b-ecbff466899b vm uve deleted from opserver 2025-10-16 04:20:38,444 - DEBUG - Verifying the e648138f-ddc6-4577-932b-ecbff466899b virtual network link through opserver 10.0.0.57 2025-10-16 04:20:38,444 - DEBUG - Requesting: http://10.0.0.57:8081/analytics/uves/virtual-machines 2025-10-16 04:20:38,456 - INFO - e648138f-ddc6-4577-932b-ecbff466899b vm uve deleted from opserver 2025-10-16 04:20:38,457 - DEBUG - Verifying the e648138f-ddc6-4577-932b-ecbff466899b virtual network link through opserver 10.0.0.23 2025-10-16 04:20:38,457 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-10-16 04:20:38,470 - INFO - e648138f-ddc6-4577-932b-ecbff466899b vm uve deleted from opserver 2025-10-16 04:20:38,470 - INFO - Validated that VM e648138f-ddc6-4577-932b-ecbff466899b is removed from Opserver 2025-10-16 04:20:38,471 - INFO - Deleting VN ctest-vn1-36242444 2025-10-16 04:20:38,706 - DEBUG - Response for deleting network () 2025-10-16 04:20:38,706 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/3d2c9a8d-4be7-4561-9387-d7fdf5fa5de4 2025-10-16 04:20:38,714 - DEBUG - Response Code: 404 2025-10-16 04:20:38,714 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-10-16 04:20:38,721 - DEBUG - Requesting: http://10.0.0.19:8082/domain/446ea2e6-9d96-4973-a55a-3f633ea4d0f9 2025-10-16 04:20:38,783 - DEBUG - Requesting: http://10.0.0.19:8082/project/03e29f65-50a8-4f9b-937e-9a83abd9c7d6 2025-10-16 04:20:38,883 - INFO - Validated that VN ctest-vn1-36242444 is not found in API Server 2025-10-16 04:20:38,892 - DEBUG - VN ctest-vn1-36242444 is not present in Agent 10.0.0.38 2025-10-16 04:20:38,903 - DEBUG - VN ctest-vn1-36242444 is not present in Agent 10.0.0.129 2025-10-16 04:20:38,903 - INFO - Validated that VN ctest-vn1-36242444 is not in any agent 2025-10-16 04:20:38,932 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-36242444 info 2025-10-16 04:20:40,216 - DEBUG - Skipping xmpp flap check 2025-10-16 04:20:40,216 - INFO - END TEST : test_control_node_switchover : PASSED[0:03:43] 2025-10-16 04:20:40,216 - INFO - -------------------------------------------------------------------------------- 2025-10-16 04:20:40,219 - INFO - ================================================================================ 2025-10-16 04:20:40,219 - INFO - STARTING TEST : test_process_restart_in_policy_between_vns 2025-10-16 04:20:40,219 - INFO - TEST DESCRIPTION : Test to validate that with policy having rule to check icmp fwding between VMs on different VNs , ping between VMs should pass with process restarts 1. Pick 2 VN's from resource pool which has one VM each 2. Create policy with icmp allow rule between those VN's and bind it networks 3. Ping from one VM to another VM 4. Restart process 'vrouter' and 'control' on setup 5. Ping again between VM's after process restart Pass criteria: Step 2,3,4 and 5 should pass 2025-10-16 04:20:41,486 - DEBUG - Skipping xmpp flap check 2025-10-16 04:20:41,486 - INFO - Initial checks done. Running the testcase now 2025-10-16 04:20:41,486 - INFO - 2025-10-16 04:20:41,492 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-10-16 04:20:41,806 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-10-16 04:20:42,342 - INFO - Created VN ctest-vn1-59270179, UUID :12d682ad-d9f7-41cf-89cf-045cfbd1f623 2025-10-16 04:20:42,665 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-10-16 04:20:42,673 - DEBUG - Requesting: http://10.0.0.19:8082/domain/446ea2e6-9d96-4973-a55a-3f633ea4d0f9 2025-10-16 04:20:42,685 - DEBUG - Requesting: http://10.0.0.19:8082/project/03e29f65-50a8-4f9b-937e-9a83abd9c7d6 2025-10-16 04:20:42,789 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/12d682ad-d9f7-41cf-89cf-045cfbd1f623 2025-10-16 04:20:42,801 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/12d682ad-d9f7-41cf-89cf-045cfbd1f623 2025-10-16 04:20:42,810 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/5abac593-02f5-480b-b1f1-dc2d65495173 2025-10-16 04:20:42,819 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/5abac593-02f5-480b-b1f1-dc2d65495173 2025-10-16 04:20:42,826 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/71d9c50f-1864-49f8-9fd2-03be48bacb8a 2025-10-16 04:20:42,834 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-16 04:20:42,834 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/12d682ad-d9f7-41cf-89cf-045cfbd1f623 2025-10-16 04:20:42,844 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/5abac593-02f5-480b-b1f1-dc2d65495173 2025-10-16 04:20:42,860 - INFO - Verified VN network id 12 for VN 12d682ad-d9f7-41cf-89cf-045cfbd1f623 2025-10-16 04:20:42,860 - INFO - Verifications in API Server for VN ctest-vn1-59270179 passed 2025-10-16 04:20:42,861 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/12d682ad-d9f7-41cf-89cf-045cfbd1f623 2025-10-16 04:20:42,874 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/5abac593-02f5-480b-b1f1-dc2d65495173 2025-10-16 04:20:42,882 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/5abac593-02f5-480b-b1f1-dc2d65495173 2025-10-16 04:20:42,888 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/71d9c50f-1864-49f8-9fd2-03be48bacb8a 2025-10-16 04:20:42,900 - DEBUG - Control-node 10.0.0.19 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91100301', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '1357416020896596431', 'uuid-lslong': '9930160500783576611'}, 'enable': 'true', 'created': '2025-10-16T04:20:42', 'last-modified': '2025-10-16T04:20:42', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.422398'} 2025-10-16 04:20:42,907 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-16 04:20:42,913 - DEBUG - Control-node 10.0.0.57 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91100301', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '1357416020896596431', 'uuid-lslong': '9930160500783576611'}, 'enable': 'true', 'created': '2025-10-16T04:20:42', 'last-modified': '2025-10-16T04:20:42', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.433215'} 2025-10-16 04:20:42,918 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-16 04:20:42,923 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91100301', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '1357416020896596431', 'uuid-lslong': '9930160500783576611'}, 'enable': 'true', 'created': '2025-10-16T04:20:42', 'last-modified': '2025-10-16T04:20:42', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.446162'} 2025-10-16 04:20:42,928 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-16 04:20:42,928 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-59270179 passed 2025-10-16 04:20:42,929 - DEBUG - ====Verifying policy data for ctest-vn1-59270179 in API_Server ====== 2025-10-16 04:20:42,929 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-10-16 04:20:42,934 - DEBUG - Requesting: http://10.0.0.19:8082/domain/446ea2e6-9d96-4973-a55a-3f633ea4d0f9 2025-10-16 04:20:42,946 - DEBUG - Requesting: http://10.0.0.19:8082/project/03e29f65-50a8-4f9b-937e-9a83abd9c7d6 2025-10-16 04:20:43,038 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/12d682ad-d9f7-41cf-89cf-045cfbd1f623 2025-10-16 04:20:43,049 - DEBUG - =>VN ctest-vn1-59270179 has no policy to be verified 2025-10-16 04:20:43,050 - DEBUG - Verifying the vn in opserver 2025-10-16 04:20:43,050 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 virtual network link through opserver 10.0.0.19 2025-10-16 04:20:43,050 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-networks 2025-10-16 04:20:43,062 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179', 'href': 'http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179?flat'} 2025-10-16 04:20:43,062 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 virtual network link through opserver 10.0.0.57 2025-10-16 04:20:43,062 - DEBUG - Requesting: http://10.0.0.57:8081/analytics/uves/virtual-networks 2025-10-16 04:20:43,075 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179', 'href': 'http://10.0.0.57:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179?flat'} 2025-10-16 04:20:43,075 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 virtual network link through opserver 10.0.0.23 2025-10-16 04:20:43,075 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-10-16 04:20:43,086 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179?flat'} 2025-10-16 04:20:43,086 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 is found in opserver 2025-10-16 04:20:43,107 - DEBUG - Do not have enough data to verify VN in agent 2025-10-16 04:20:43,118 - DEBUG - VRF ids for VN ctest-vn1-59270179: {} 2025-10-16 04:20:43,746 - INFO - Created VN ctest-vn2-58198678, UUID :e6c28100-ef36-401e-b3dd-23ef1008c619 2025-10-16 04:20:44,068 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-10-16 04:20:44,074 - DEBUG - Requesting: http://10.0.0.19:8082/domain/446ea2e6-9d96-4973-a55a-3f633ea4d0f9 2025-10-16 04:20:44,085 - DEBUG - Requesting: http://10.0.0.19:8082/project/03e29f65-50a8-4f9b-937e-9a83abd9c7d6 2025-10-16 04:20:44,183 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/e6c28100-ef36-401e-b3dd-23ef1008c619 2025-10-16 04:20:44,207 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/e6c28100-ef36-401e-b3dd-23ef1008c619 2025-10-16 04:20:44,222 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/e64ce128-57bf-4ddf-b2c5-d5cfe5c76a7a 2025-10-16 04:20:44,229 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/e64ce128-57bf-4ddf-b2c5-d5cfe5c76a7a 2025-10-16 04:20:44,238 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/de8e0421-cf2f-4407-8d8a-b781290e92d2 2025-10-16 04:20:44,246 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-10-16 04:20:44,246 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/e6c28100-ef36-401e-b3dd-23ef1008c619 2025-10-16 04:20:44,257 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/e64ce128-57bf-4ddf-b2c5-d5cfe5c76a7a 2025-10-16 04:20:44,273 - INFO - Verified VN network id 13 for VN e6c28100-ef36-401e-b3dd-23ef1008c619 2025-10-16 04:20:44,273 - INFO - Verifications in API Server for VN ctest-vn2-58198678 passed 2025-10-16 04:20:44,273 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/e6c28100-ef36-401e-b3dd-23ef1008c619 2025-10-16 04:20:44,283 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/e64ce128-57bf-4ddf-b2c5-d5cfe5c76a7a 2025-10-16 04:20:44,290 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/e64ce128-57bf-4ddf-b2c5-d5cfe5c76a7a 2025-10-16 04:20:44,299 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/de8e0421-cf2f-4407-8d8a-b781290e92d2 2025-10-16 04:20:44,311 - DEBUG - Control-node 10.0.0.19 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91100301', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '16627994615218585630', 'uuid-lslong': '12960554812315321881'}, 'enable': 'true', 'created': '2025-10-16T04:20:43', 'last-modified': '2025-10-16T04:20:43', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.416072'} 2025-10-16 04:20:44,317 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-10-16 04:20:44,322 - DEBUG - Control-node 10.0.0.57 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91100301', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '16627994615218585630', 'uuid-lslong': '12960554812315321881'}, 'enable': 'true', 'created': '2025-10-16T04:20:43', 'last-modified': '2025-10-16T04:20:43', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.426788'} 2025-10-16 04:20:44,328 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-10-16 04:20:44,333 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91100301', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '16627994615218585630', 'uuid-lslong': '12960554812315321881'}, 'enable': 'true', 'created': '2025-10-16T04:20:43', 'last-modified': '2025-10-16T04:20:43', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.437842'} 2025-10-16 04:20:44,338 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-10-16 04:20:44,338 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn2-58198678 passed 2025-10-16 04:20:44,338 - DEBUG - ====Verifying policy data for ctest-vn2-58198678 in API_Server ====== 2025-10-16 04:20:44,339 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-10-16 04:20:44,345 - DEBUG - Requesting: http://10.0.0.19:8082/domain/446ea2e6-9d96-4973-a55a-3f633ea4d0f9 2025-10-16 04:20:44,355 - DEBUG - Requesting: http://10.0.0.19:8082/project/03e29f65-50a8-4f9b-937e-9a83abd9c7d6 2025-10-16 04:20:44,470 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/e6c28100-ef36-401e-b3dd-23ef1008c619 2025-10-16 04:20:44,481 - DEBUG - =>VN ctest-vn2-58198678 has no policy to be verified 2025-10-16 04:20:44,481 - DEBUG - Verifying the vn in opserver 2025-10-16 04:20:44,481 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 virtual network link through opserver 10.0.0.19 2025-10-16 04:20:44,482 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-networks 2025-10-16 04:20:44,495 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678', 'href': 'http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678?flat'} 2025-10-16 04:20:44,495 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 virtual network link through opserver 10.0.0.57 2025-10-16 04:20:44,495 - DEBUG - Requesting: http://10.0.0.57:8081/analytics/uves/virtual-networks 2025-10-16 04:20:44,507 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678', 'href': 'http://10.0.0.57:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678?flat'} 2025-10-16 04:20:44,508 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 virtual network link through opserver 10.0.0.23 2025-10-16 04:20:44,508 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-10-16 04:20:44,520 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678?flat'} 2025-10-16 04:20:44,520 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 is found in opserver 2025-10-16 04:20:44,542 - DEBUG - Do not have enough data to verify VN in agent 2025-10-16 04:20:44,555 - DEBUG - VRF ids for VN ctest-vn2-58198678: {} 2025-10-16 04:20:46,053 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4339-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4339-1) 2025-10-16 04:20:47,478 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4339-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4339-2) 2025-10-16 04:20:47,478 - INFO - Waiting for VM ctest-vn1_vm1-67798974 to be up.. 2025-10-16 04:20:47,573 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-10-16 04:20:52,650 - DEBUG - VM is in ACTIVE state now 2025-10-16 04:20:52,650 - INFO - VM name : ctest-vn1_vm1-67798974 2025-10-16 04:20:52,727 - DEBUG - VM ctest-vn1_vm1-67798974 ID is 4df706fb-91e9-47d7-bd59-bf4a8c8494e9 2025-10-16 04:20:52,727 - DEBUG - VM ctest-vn1_vm1-67798974 launched on Node an-jenkins-deploy-platform-ansible-os-4339-1 2025-10-16 04:20:52,811 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/4df706fb-91e9-47d7-bd59-bf4a8c8494e9 2025-10-16 04:20:52,821 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/13aee4ef-22fd-4c32-a490-2d026525953e 2025-10-16 04:20:56,005 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-10-16 04:20:56,005 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67798974 failed! 2025-10-16 04:20:56,021 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 is 192.168.1.254 and allocation pool is NOT set 2025-10-16 04:21:00,098 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-10-16 04:21:00,098 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67798974 failed! 2025-10-16 04:21:00,111 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 is 192.168.1.254 and allocation pool is NOT set 2025-10-16 04:21:04,193 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2025-10-16 04:21:04,193 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67798974 failed! 2025-10-16 04:21:04,212 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 is 192.168.1.254 and allocation pool is NOT set 2025-10-16 04:21:08,292 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-10-16 04:21:08,292 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67798974 failed! 2025-10-16 04:21:08,307 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 is 192.168.1.254 and allocation pool is NOT set 2025-10-16 04:21:12,393 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2025-10-16 04:21:12,393 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67798974 failed! 2025-10-16 04:21:12,409 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 is 192.168.1.254 and allocation pool is NOT set 2025-10-16 04:21:16,479 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-10-16 04:21:16,479 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67798974 failed! 2025-10-16 04:21:16,493 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 is 192.168.1.254 and allocation pool is NOT set 2025-10-16 04:21:20,582 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1024ms') 2025-10-16 04:21:20,582 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67798974 failed! 2025-10-16 04:21:20,598 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 is 192.168.1.254 and allocation pool is NOT set 2025-10-16 04:21:24,680 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2025-10-16 04:21:24,680 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67798974 failed! 2025-10-16 04:21:24,695 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 is 192.168.1.254 and allocation pool is NOT set 2025-10-16 04:21:26,756 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=7.03 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=3.64 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 3.643/5.337/7.032/1.694 ms') 2025-10-16 04:21:26,756 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67798974 passed 2025-10-16 04:21:26,828 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:21:26,829 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-67798974, IP 192.168.1.252, Port 22 2025-10-16 04:21:26,896 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-16 04:21:26,979 - DEBUG - VM ctest-vn1_vm1-67798974 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-16 04:21:31,980 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:21:31,980 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-67798974, IP 192.168.1.252, Port 22 2025-10-16 04:21:32,038 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-16 04:21:32,130 - DEBUG - VM ctest-vn1_vm1-67798974 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-16 04:21:37,130 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:21:37,130 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-67798974, IP 192.168.1.252, Port 22 2025-10-16 04:21:37,186 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-16 04:21:37,289 - DEBUG - VM ctest-vn1_vm1-67798974 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-16 04:21:42,290 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:21:42,290 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-67798974, IP 192.168.1.252, Port 22 2025-10-16 04:21:42,355 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-16 04:21:42,438 - DEBUG - VM ctest-vn1_vm1-67798974 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-16 04:21:47,438 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:21:47,439 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-67798974, IP 192.168.1.252, Port 22 2025-10-16 04:21:47,602 - DEBUG - VM ctest-vn1_vm1-67798974 is ready for SSH connections 2025-10-16 04:21:47,602 - INFO - Waiting for VM ctest-vn2_vm1-98937197 to be up.. 2025-10-16 04:21:47,701 - DEBUG - VM is in ACTIVE state now 2025-10-16 04:21:47,701 - INFO - VM name : ctest-vn2_vm1-98937197 2025-10-16 04:21:47,785 - DEBUG - VM ctest-vn2_vm1-98937197 ID is af9d4edc-43b3-4a81-a3d4-ef8fe6dff8b9 2025-10-16 04:21:47,785 - DEBUG - VM ctest-vn2_vm1-98937197 launched on Node an-jenkins-deploy-platform-ansible-os-4339-2 2025-10-16 04:21:47,866 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/af9d4edc-43b3-4a81-a3d4-ef8fe6dff8b9 2025-10-16 04:21:47,876 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/74a143f2-29c4-4841-973a-5980770490bb 2025-10-16 04:21:49,045 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=6.75 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.725 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 0.725/3.738/6.752/3.013 ms') 2025-10-16 04:21:49,045 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn2_vm1-98937197 passed 2025-10-16 04:21:49,111 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:21:49,111 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-98937197, IP 192.168.2.252, Port 22 2025-10-16 04:21:49,173 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-16 04:21:49,259 - DEBUG - VM ctest-vn2_vm1-98937197 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-16 04:21:54,260 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:21:54,260 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-98937197, IP 192.168.2.252, Port 22 2025-10-16 04:21:54,328 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-16 04:21:54,408 - DEBUG - VM ctest-vn2_vm1-98937197 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-16 04:21:59,408 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:21:59,408 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-98937197, IP 192.168.2.252, Port 22 2025-10-16 04:21:59,581 - DEBUG - VM ctest-vn2_vm1-98937197 is ready for SSH connections 2025-10-16 04:21:59,581 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:21:59,581 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.252, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.38, gateway password: c0ntrail123 2025-10-16 04:21:59,581 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.252 2025-10-16 04:22:40,798 - DEBUG - PING 192.168.2.252 (192.168.2.252) 56(84) bytes of data. 64 bytes from 192.168.2.252: icmp_req=1 ttl=63 time=7.19 ms 64 bytes from 192.168.2.252: icmp_req=2 ttl=63 time=1.27 ms 64 bytes from 192.168.2.252: icmp_req=3 ttl=63 time=1.43 ms --- 192.168.2.252 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 1.270/3.299/7.199/2.758 ms 2025-10-16 04:22:40,798 - INFO - Ping to IP 192.168.2.252 from VM ctest-vn1_vm1-67798974 passed 2025-10-16 04:22:44,221 - INFO - 10.0.0.38 2025-10-16 04:22:44,225 - INFO - agent:initializing 2025-10-16 04:22:44,225 - DEBUG - defaultdict(, {'10.0.0.38': {'agent': {'status': 'initializing', 'description': None}}}) 2025-10-16 04:22:44,225 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-16 04:22:49,319 - INFO - 10.0.0.38 2025-10-16 04:22:49,326 - INFO - agent:active 2025-10-16 04:22:49,326 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.38 2025-10-16 04:22:52,772 - INFO - 10.0.0.129 2025-10-16 04:22:52,778 - INFO - agent:initializing 2025-10-16 04:22:52,778 - DEBUG - defaultdict(, {'10.0.0.129': {'agent': {'status': 'initializing', 'description': None}}}) 2025-10-16 04:22:52,778 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-16 04:22:57,879 - INFO - 10.0.0.129 2025-10-16 04:22:57,885 - INFO - agent:active 2025-10-16 04:22:57,885 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.129 2025-10-16 04:22:58,440 - INFO - 10.0.0.19 2025-10-16 04:22:58,445 - INFO - control:initializing 2025-10-16 04:22:58,445 - DEBUG - defaultdict(, {'10.0.0.19': {'control': {'status': 'initializing', 'description': None}}}) 2025-10-16 04:22:58,446 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-16 04:23:03,557 - INFO - 10.0.0.19 2025-10-16 04:23:03,568 - INFO - control:active 2025-10-16 04:23:03,568 - INFO - Contrail services ['control'] are up on nodes 10.0.0.19 2025-10-16 04:23:03,964 - INFO - 10.0.0.57 2025-10-16 04:23:03,967 - INFO - control:initializing 2025-10-16 04:23:03,967 - DEBUG - defaultdict(, {'10.0.0.57': {'control': {'status': 'initializing', 'description': None}}}) 2025-10-16 04:23:03,967 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-16 04:23:09,076 - INFO - 10.0.0.57 2025-10-16 04:23:09,085 - INFO - control:active 2025-10-16 04:23:09,085 - INFO - Contrail services ['control'] are up on nodes 10.0.0.57 2025-10-16 04:23:09,548 - INFO - 10.0.0.23 2025-10-16 04:23:09,551 - INFO - control:initializing 2025-10-16 04:23:09,551 - DEBUG - defaultdict(, {'10.0.0.23': {'control': {'status': 'initializing', 'description': None}}}) 2025-10-16 04:23:09,551 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-16 04:23:14,643 - INFO - 10.0.0.23 2025-10-16 04:23:14,648 - INFO - control:active 2025-10-16 04:23:14,649 - INFO - Contrail services ['control'] are up on nodes 10.0.0.23 2025-10-16 04:23:15,508 - INFO - 10.0.0.19 2025-10-16 04:23:15,511 - INFO - api-server:initializing 2025-10-16 04:23:15,511 - DEBUG - defaultdict(, {'10.0.0.19': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-16 04:23:15,511 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-16 04:23:20,610 - INFO - 10.0.0.19 2025-10-16 04:23:20,614 - INFO - api-server:initializing 2025-10-16 04:23:20,614 - DEBUG - defaultdict(, {'10.0.0.19': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-16 04:23:20,614 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-10-16 04:23:25,746 - INFO - 10.0.0.19 2025-10-16 04:23:25,751 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2025-10-16 04:23:25,752 - DEBUG - defaultdict(, {'10.0.0.19': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2025-10-16 04:23:25,752 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-10-16 04:23:30,882 - INFO - 10.0.0.19 2025-10-16 04:23:30,886 - INFO - api-server:active 2025-10-16 04:23:30,886 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.19 2025-10-16 04:23:31,598 - INFO - 10.0.0.57 2025-10-16 04:23:31,602 - INFO - api-server:initializing 2025-10-16 04:23:31,602 - DEBUG - defaultdict(, {'10.0.0.57': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-16 04:23:31,602 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-16 04:23:36,703 - INFO - 10.0.0.57 2025-10-16 04:23:36,708 - INFO - api-server:initializing 2025-10-16 04:23:36,708 - DEBUG - defaultdict(, {'10.0.0.57': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-16 04:23:36,708 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-10-16 04:23:41,799 - INFO - 10.0.0.57 2025-10-16 04:23:41,805 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2025-10-16 04:23:41,805 - DEBUG - defaultdict(, {'10.0.0.57': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2025-10-16 04:23:41,805 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-10-16 04:23:46,897 - INFO - 10.0.0.57 2025-10-16 04:23:46,903 - INFO - api-server:active 2025-10-16 04:23:46,904 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.57 2025-10-16 04:23:47,530 - INFO - 10.0.0.23 2025-10-16 04:23:47,534 - INFO - api-server:initializing 2025-10-16 04:23:47,534 - DEBUG - defaultdict(, {'10.0.0.23': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-16 04:23:47,534 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-16 04:23:52,634 - INFO - 10.0.0.23 2025-10-16 04:23:52,638 - INFO - api-server:initializing 2025-10-16 04:23:52,638 - DEBUG - defaultdict(, {'10.0.0.23': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-16 04:23:52,638 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-10-16 04:23:57,736 - INFO - 10.0.0.23 2025-10-16 04:23:57,743 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2025-10-16 04:23:57,743 - DEBUG - defaultdict(, {'10.0.0.23': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2025-10-16 04:23:57,743 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-10-16 04:24:02,831 - INFO - 10.0.0.23 2025-10-16 04:24:02,836 - INFO - api-server:active 2025-10-16 04:24:02,836 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.23 2025-10-16 04:24:02,949 - INFO - 10.0.0.19 2025-10-16 04:24:02,949 - INFO - config 2025-10-16 04:24:02,955 - INFO - config-nodemgr:active 2025-10-16 04:24:02,960 - INFO - api-server:active 2025-10-16 04:24:02,965 - INFO - schema:active 2025-10-16 04:24:02,970 - INFO - svc-monitor:active 2025-10-16 04:24:02,974 - INFO - device-manager:active 2025-10-16 04:24:02,974 - INFO - config-database 2025-10-16 04:24:02,975 - INFO - config-cassandra:active 2025-10-16 04:24:02,975 - INFO - config-zookeeper:active 2025-10-16 04:24:02,975 - INFO - config-rabbitmq:active 2025-10-16 04:24:02,975 - INFO - control 2025-10-16 04:24:02,980 - INFO - control-nodemgr:active 2025-10-16 04:24:02,985 - INFO - control:active 2025-10-16 04:24:02,985 - INFO - named:active 2025-10-16 04:24:02,991 - INFO - dns:active 2025-10-16 04:24:02,991 - INFO - analytics 2025-10-16 04:24:02,997 - INFO - analytics-nodemgr:active 2025-10-16 04:24:03,001 - INFO - analytics-api:active 2025-10-16 04:24:03,006 - INFO - collector:active 2025-10-16 04:24:03,006 - INFO - analytics-database 2025-10-16 04:24:03,006 - INFO - analytics-cassandra:active 2025-10-16 04:24:03,012 - INFO - analyticsdb-nodemgr:active 2025-10-16 04:24:03,017 - INFO - query-engine:active 2025-10-16 04:24:03,017 - INFO - webui 2025-10-16 04:24:03,017 - INFO - webui:active 2025-10-16 04:24:03,017 - INFO - webui-middleware:active 2025-10-16 04:24:03,017 - INFO - redis:active 2025-10-16 04:24:03,017 - INFO - analytics_snmp 2025-10-16 04:24:03,022 - INFO - snmp-collector:active 2025-10-16 04:24:03,022 - INFO - snmp-topology:active 2025-10-16 04:24:03,022 - INFO - analytics_alarm 2025-10-16 04:24:03,022 - INFO - alarmgen:active 2025-10-16 04:24:03,124 - INFO - 10.0.0.57 2025-10-16 04:24:03,125 - INFO - config 2025-10-16 04:24:03,132 - INFO - config-nodemgr:active 2025-10-16 04:24:03,137 - INFO - api-server:active 2025-10-16 04:24:03,140 - INFO - schema:backup 2025-10-16 04:24:03,143 - INFO - svc-monitor:backup 2025-10-16 04:24:03,147 - INFO - device-manager:backup 2025-10-16 04:24:03,147 - INFO - config-database 2025-10-16 04:24:03,147 - INFO - config-cassandra:active 2025-10-16 04:24:03,148 - INFO - config-zookeeper:active 2025-10-16 04:24:03,148 - INFO - config-rabbitmq:active 2025-10-16 04:24:03,148 - INFO - control 2025-10-16 04:24:03,153 - INFO - control-nodemgr:active 2025-10-16 04:24:03,159 - INFO - control:active 2025-10-16 04:24:03,159 - INFO - named:active 2025-10-16 04:24:03,165 - INFO - dns:active 2025-10-16 04:24:03,165 - INFO - analytics 2025-10-16 04:24:03,171 - INFO - analytics-nodemgr:active 2025-10-16 04:24:03,177 - INFO - analytics-api:active 2025-10-16 04:24:03,183 - INFO - collector:active 2025-10-16 04:24:03,184 - INFO - analytics-database 2025-10-16 04:24:03,184 - INFO - analytics-cassandra:active 2025-10-16 04:24:03,190 - INFO - analyticsdb-nodemgr:active 2025-10-16 04:24:03,197 - INFO - query-engine:active 2025-10-16 04:24:03,197 - INFO - webui 2025-10-16 04:24:03,197 - INFO - webui:active 2025-10-16 04:24:03,197 - INFO - webui-middleware:active 2025-10-16 04:24:03,197 - INFO - redis:active 2025-10-16 04:24:03,197 - INFO - analytics_snmp 2025-10-16 04:24:03,203 - INFO - snmp-collector:active 2025-10-16 04:24:03,203 - INFO - snmp-topology:active 2025-10-16 04:24:03,203 - INFO - analytics_alarm 2025-10-16 04:24:03,203 - INFO - alarmgen:active 2025-10-16 04:24:03,316 - INFO - 10.0.0.23 2025-10-16 04:24:03,316 - INFO - config 2025-10-16 04:24:03,323 - INFO - config-nodemgr:active 2025-10-16 04:24:03,329 - INFO - api-server:active 2025-10-16 04:24:03,333 - INFO - schema:backup 2025-10-16 04:24:03,337 - INFO - svc-monitor:backup 2025-10-16 04:24:03,341 - INFO - device-manager:backup 2025-10-16 04:24:03,341 - INFO - config-database 2025-10-16 04:24:03,341 - INFO - config-cassandra:active 2025-10-16 04:24:03,341 - INFO - config-zookeeper:active 2025-10-16 04:24:03,341 - INFO - config-rabbitmq:active 2025-10-16 04:24:03,341 - INFO - control 2025-10-16 04:24:03,347 - INFO - control-nodemgr:active 2025-10-16 04:24:03,351 - INFO - control:active 2025-10-16 04:24:03,352 - INFO - named:active 2025-10-16 04:24:03,357 - INFO - dns:active 2025-10-16 04:24:03,357 - INFO - analytics 2025-10-16 04:24:03,361 - INFO - analytics-nodemgr:active 2025-10-16 04:24:03,366 - INFO - analytics-api:active 2025-10-16 04:24:03,371 - INFO - collector:active 2025-10-16 04:24:03,371 - INFO - analytics-database 2025-10-16 04:24:03,371 - INFO - analytics-cassandra:active 2025-10-16 04:24:03,377 - INFO - analyticsdb-nodemgr:active 2025-10-16 04:24:03,385 - INFO - query-engine:active 2025-10-16 04:24:03,385 - INFO - webui 2025-10-16 04:24:03,385 - INFO - webui:active 2025-10-16 04:24:03,385 - INFO - webui-middleware:active 2025-10-16 04:24:03,385 - INFO - redis:active 2025-10-16 04:24:03,385 - INFO - analytics_snmp 2025-10-16 04:24:03,390 - INFO - snmp-collector:active 2025-10-16 04:24:03,391 - INFO - snmp-topology:active 2025-10-16 04:24:03,391 - INFO - analytics_alarm 2025-10-16 04:24:03,391 - INFO - alarmgen:active 2025-10-16 04:24:03,481 - INFO - 10.0.0.38 2025-10-16 04:24:03,481 - INFO - vrouter 2025-10-16 04:24:03,487 - INFO - vrouter-nodemgr:active 2025-10-16 04:24:03,492 - INFO - agent:active 2025-10-16 04:24:03,580 - INFO - 10.0.0.129 2025-10-16 04:24:03,581 - INFO - vrouter 2025-10-16 04:24:03,587 - INFO - vrouter-nodemgr:active 2025-10-16 04:24:03,594 - INFO - agent:active 2025-10-16 04:24:03,594 - INFO - All the contrail services are up on all nodes 2025-10-16 04:24:03,595 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-1?flat 2025-10-16 04:24:04,044 - WARNING - an-jenkins-deploy-platform-ansible-os-4339-1:contrail-vrouter-agent module connection to ['10.20.0.5:5269'] servers NOT UP 2025-10-16 04:24:04,044 - INFO - an-jenkins-deploy-platform-ansible-os-4339-1:contrail-vrouter-agent module connection to ['10.20.0.14:5269'] servers UP 2025-10-16 04:24:04,045 - INFO - an-jenkins-deploy-platform-ansible-os-4339-1 connected to xmpp 10.20.0.14 2025-10-16 04:24:04,045 - INFO - an-jenkins-deploy-platform-ansible-os-4339-1:contrail-vrouter-agent module connection to ['10.20.0.252:5269'] servers UP 2025-10-16 04:24:04,045 - INFO - an-jenkins-deploy-platform-ansible-os-4339-1 connected to xmpp 10.20.0.252 2025-10-16 04:24:04,045 - INFO - an-jenkins-deploy-platform-ansible-os-4339-1:contrail-vrouter-agent module connection to ['10.20.0.5:53'] servers UP 2025-10-16 04:24:04,045 - INFO - an-jenkins-deploy-platform-ansible-os-4339-1 connected to dns 10.20.0.5 2025-10-16 04:24:04,045 - INFO - an-jenkins-deploy-platform-ansible-os-4339-1:contrail-vrouter-agent module connection to ['10.20.0.14:53'] servers UP 2025-10-16 04:24:04,045 - INFO - an-jenkins-deploy-platform-ansible-os-4339-1 connected to dns 10.20.0.14 2025-10-16 04:24:04,045 - WARNING - an-jenkins-deploy-platform-ansible-os-4339-1:contrail-vrouter-agent module connection to ['10.20.0.252:53'] servers NOT UP 2025-10-16 04:24:04,046 - WARNING - an-jenkins-deploy-platform-ansible-os-4339-1:contrail-vrouter-agent module connection to ['10.0.0.19:8086'] servers NOT UP 2025-10-16 04:24:04,046 - INFO - an-jenkins-deploy-platform-ansible-os-4339-1:contrail-vrouter-agent module connection to ['10.0.0.57:8086'] servers UP 2025-10-16 04:24:04,046 - INFO - an-jenkins-deploy-platform-ansible-os-4339-1 connected to collector 10.0.0.57 2025-10-16 04:24:04,046 - WARNING - an-jenkins-deploy-platform-ansible-os-4339-1:contrail-vrouter-agent module connection to ['10.0.0.23:8086'] servers NOT UP 2025-10-16 04:24:04,046 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4339-1?flat 2025-10-16 04:24:04,464 - INFO - cn-jenkins-deploy-platform-ansible-os-4339-1:contrail-control module connection to 10.0.0.19:9041 servers UP 2025-10-16 04:24:04,465 - INFO - cn-jenkins-deploy-platform-ansible-os-4339-1:contrail-control module connection to 10.0.0.19:5673 servers UP 2025-10-16 04:24:04,465 - INFO - cn-jenkins-deploy-platform-ansible-os-4339-1:contrail-control module connection to ['10.0.0.19:8086'] servers UP 2025-10-16 04:24:04,465 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4339-2?flat 2025-10-16 04:24:04,529 - INFO - cn-jenkins-deploy-platform-ansible-os-4339-2:contrail-control module connection to 10.0.0.19:9041 servers UP 2025-10-16 04:24:04,529 - INFO - cn-jenkins-deploy-platform-ansible-os-4339-2:contrail-control module connection to 10.0.0.19:5673 servers UP 2025-10-16 04:24:04,529 - INFO - cn-jenkins-deploy-platform-ansible-os-4339-2:contrail-control module connection to ['10.0.0.19:8086'] servers UP 2025-10-16 04:24:04,529 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4339-3?flat 2025-10-16 04:24:04,553 - INFO - cn-jenkins-deploy-platform-ansible-os-4339-3:contrail-control module connection to 10.0.0.19:9041 servers UP 2025-10-16 04:24:04,553 - INFO - cn-jenkins-deploy-platform-ansible-os-4339-3:contrail-control module connection to 10.0.0.19:5673 servers UP 2025-10-16 04:24:04,553 - INFO - cn-jenkins-deploy-platform-ansible-os-4339-3:contrail-control module connection to ['10.0.0.19:8086'] servers UP 2025-10-16 04:24:04,553 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4339-1.?flat 2025-10-16 04:24:04,583 - DEBUG - contrail-api is running 2025-10-16 04:24:04,583 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4339-2.?flat 2025-10-16 04:24:04,611 - DEBUG - contrail-api is running 2025-10-16 04:24:04,611 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4339-3.?flat 2025-10-16 04:24:04,632 - DEBUG - contrail-api is running 2025-10-16 04:24:04,942 - INFO - Created VN ctest-vn3-24509381, UUID :20548a9b-640d-4766-812e-ff33a597a470 2025-10-16 04:24:05,355 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-10-16 04:24:05,384 - DEBUG - Requesting: http://10.0.0.19:8082/domain/446ea2e6-9d96-4973-a55a-3f633ea4d0f9 2025-10-16 04:24:05,400 - DEBUG - Requesting: http://10.0.0.19:8082/project/03e29f65-50a8-4f9b-937e-9a83abd9c7d6 2025-10-16 04:24:05,499 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/12d682ad-d9f7-41cf-89cf-045cfbd1f623 2025-10-16 04:24:05,526 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/12d682ad-d9f7-41cf-89cf-045cfbd1f623 2025-10-16 04:24:05,538 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/5abac593-02f5-480b-b1f1-dc2d65495173 2025-10-16 04:24:05,546 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/5abac593-02f5-480b-b1f1-dc2d65495173 2025-10-16 04:24:05,554 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/71d9c50f-1864-49f8-9fd2-03be48bacb8a 2025-10-16 04:24:05,564 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-16 04:24:05,564 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/12d682ad-d9f7-41cf-89cf-045cfbd1f623 2025-10-16 04:24:05,577 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/5abac593-02f5-480b-b1f1-dc2d65495173 2025-10-16 04:24:05,597 - INFO - Verified VN network id 12 for VN 12d682ad-d9f7-41cf-89cf-045cfbd1f623 2025-10-16 04:24:05,597 - INFO - Verifications in API Server for VN ctest-vn1-59270179 passed 2025-10-16 04:24:05,597 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/12d682ad-d9f7-41cf-89cf-045cfbd1f623 2025-10-16 04:24:05,611 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/5abac593-02f5-480b-b1f1-dc2d65495173 2025-10-16 04:24:05,625 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/5abac593-02f5-480b-b1f1-dc2d65495173 2025-10-16 04:24:05,634 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/71d9c50f-1864-49f8-9fd2-03be48bacb8a 2025-10-16 04:24:05,646 - DEBUG - Control-node 10.0.0.19 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91100301', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '1357416020896596431', 'uuid-lslong': '9930160500783576611'}, 'enable': 'true', 'created': '2025-10-16T04:20:42', 'last-modified': '2025-10-16T04:20:43', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:06.753834'} 2025-10-16 04:24:05,655 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-16 04:24:05,662 - DEBUG - Control-node 10.0.0.57 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91100301', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '1357416020896596431', 'uuid-lslong': '9930160500783576611'}, 'enable': 'true', 'created': '2025-10-16T04:20:42', 'last-modified': '2025-10-16T04:20:43', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:01.302244'} 2025-10-16 04:24:05,674 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-16 04:24:05,680 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91100301', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '1357416020896596431', 'uuid-lslong': '9930160500783576611'}, 'enable': 'true', 'created': '2025-10-16T04:20:42', 'last-modified': '2025-10-16T04:20:43', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:55.805543'} 2025-10-16 04:24:05,687 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-16 04:24:05,687 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-59270179 passed 2025-10-16 04:24:05,687 - DEBUG - ====Verifying policy data for ctest-vn1-59270179 in API_Server ====== 2025-10-16 04:24:05,687 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-10-16 04:24:05,696 - DEBUG - Requesting: http://10.0.0.19:8082/domain/446ea2e6-9d96-4973-a55a-3f633ea4d0f9 2025-10-16 04:24:05,709 - DEBUG - Requesting: http://10.0.0.19:8082/project/03e29f65-50a8-4f9b-937e-9a83abd9c7d6 2025-10-16 04:24:05,803 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/12d682ad-d9f7-41cf-89cf-045cfbd1f623 2025-10-16 04:24:05,815 - DEBUG - ==>Verifying data for policy with id: 92042949-c2ae-4eb6-bca4-62cf5a2ebe5a, fqn: ['default-domain', 'ctest-TestBasicVMVN0-91100301', 'policy1'] 2025-10-16 04:24:05,815 - INFO - VN ctest-vn1-59270179 Policy verification: verify_vn_policy_in_api_server, status: True 2025-10-16 04:24:05,815 - DEBUG - Verifying the vn in opserver 2025-10-16 04:24:05,815 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 virtual network link through opserver 10.0.0.19 2025-10-16 04:24:05,815 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-networks 2025-10-16 04:24:05,834 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179', 'href': 'http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179?flat'} 2025-10-16 04:24:05,834 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 virtual network link through opserver 10.0.0.57 2025-10-16 04:24:05,834 - DEBUG - Requesting: http://10.0.0.57:8081/analytics/uves/virtual-networks 2025-10-16 04:24:05,910 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179', 'href': 'http://10.0.0.57:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179?flat'} 2025-10-16 04:24:05,910 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 virtual network link through opserver 10.0.0.23 2025-10-16 04:24:05,910 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-10-16 04:24:05,967 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179?flat'} 2025-10-16 04:24:05,967 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 is found in opserver 2025-10-16 04:24:05,978 - DEBUG - VRF ids for VN ctest-vn1-59270179: {'10.0.0.38': '2'} 2025-10-16 04:24:07,239 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4339-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4339-1) 2025-10-16 04:24:07,286 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-10-16 04:24:12,367 - DEBUG - VM is in ACTIVE state now 2025-10-16 04:24:12,367 - INFO - VM name : ctest-vn1_vm2-23587683 2025-10-16 04:24:12,453 - DEBUG - VM ctest-vn1_vm2-23587683 ID is 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f 2025-10-16 04:24:12,535 - DEBUG - VM is in ACTIVE state now 2025-10-16 04:24:12,536 - DEBUG - VM ctest-vn1_vm2-23587683 launched on Node an-jenkins-deploy-platform-ansible-os-4339-1 2025-10-16 04:24:12,626 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/012d19a3-853f-4c9e-b74b-a7e5e7c6a84f 2025-10-16 04:24:12,656 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/012d19a3-853f-4c9e-b74b-a7e5e7c6a84f 2025-10-16 04:24:12,663 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/8986748e-3656-4445-bd8e-718fd493c180 2025-10-16 04:24:12,676 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/012d19a3-853f-4c9e-b74b-a7e5e7c6a84f 2025-10-16 04:24:12,684 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/8986748e-3656-4445-bd8e-718fd493c180 2025-10-16 04:24:12,693 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/b4a17ee4-6805-4d74-adab-11beb8338d5b 2025-10-16 04:24:12,706 - DEBUG - Verifying in api server 10.0.0.23 2025-10-16 04:24:12,706 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/012d19a3-853f-4c9e-b74b-a7e5e7c6a84f 2025-10-16 04:24:12,713 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/8986748e-3656-4445-bd8e-718fd493c180 2025-10-16 04:24:12,726 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/b4a17ee4-6805-4d74-adab-11beb8338d5b 2025-10-16 04:24:12,733 - INFO - VM ctest-vn1_vm2-23587683 verfication in all API Servers passed 2025-10-16 04:24:12,751 - DEBUG - VM ctest-vn1_vm2-23587683 Tap interface: {'index': '4', 'name': 'tap8986748e-36', 'uuid': '8986748e-3656-4445-bd8e-718fd493c180', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179:ctest-vn1-59270179', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '12', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179', 'vm_uuid': '012d19a3-853f-4c9e-b74b-a7e5e7c6a84f', 'vm_name': 'ctest-vn1_vm2-23587683', 'ip_addr': '192.168.1.251', 'mac_addr': '02:89:86:74:8e:36', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '165', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-91100301:8986748e-3656-4445-bd8e-718fd493c180', 'sg_uuid_list': ['d00d1c1c-903e-4107-af98-1b6cf220bd38'], 'static_route_list': None, 'vm_project_uuid': '03e29f65-50a8-4f9b-937e-9a83abd9c7d6', 'admin_state': 'Enabled', 'flow_key_idx': '27', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-16 04:24:12,752 - DEBUG - Agent 10.0.0.38 vrf name: default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179:ctest-vn1-59270179 2025-10-16 04:24:12,757 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179:ctest-vn1-59270179', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179', 'table_label': '-1', 'vxlan_id': '12', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.145:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-10-16 04:24:12,844 - DEBUG - VM ctest-vn1_vm2-23587683 : Tap interface tap8986748e-36 is set to Active 2025-10-16 04:24:12,844 - DEBUG - Tap interface tap8986748e-36 detail : {'index': '4', 'name': 'tap8986748e-36', 'uuid': '8986748e-3656-4445-bd8e-718fd493c180', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179:ctest-vn1-59270179', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '12', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179', 'vm_uuid': '012d19a3-853f-4c9e-b74b-a7e5e7c6a84f', 'vm_name': 'ctest-vn1_vm2-23587683', 'ip_addr': '192.168.1.251', 'mac_addr': '02:89:86:74:8e:36', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '165', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-91100301:8986748e-3656-4445-bd8e-718fd493c180', 'sg_uuid_list': ['d00d1c1c-903e-4107-af98-1b6cf220bd38'], 'static_route_list': None, 'vm_project_uuid': '03e29f65-50a8-4f9b-937e-9a83abd9c7d6', 'admin_state': 'Enabled', 'flow_key_idx': '27', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-16 04:24:12,958 - DEBUG - Starting Layer 2 verification in Agent 2025-10-16 04:24:12,975 - DEBUG - Layer 2 path is seen for VM MAC 02:89:86:74:8e:36 in agent 10.0.0.38 2025-10-16 04:24:12,975 - DEBUG - Active layer 2 route in agent is present for VMI tap8986748e-36 2025-10-16 04:24:12,975 - DEBUG - L2 label(34) matches bw route table and interface table 2025-10-16 04:24:12,983 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 is found in Agent of node 10.0.0.38 2025-10-16 04:24:13,115 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 is consistent in agent 10.0.0.38 2025-10-16 04:24:13,116 - DEBUG - Route for VM IP ['192.168.1.251'] is consistent in agent 10.0.0.38 2025-10-16 04:24:13,116 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 verification for VM ctest-vn1_vm2-23587683 in Agent 10.0.0.38 passed 2025-10-16 04:24:13,224 - DEBUG - Starting all layer 2 verification in agent 10.0.0.38 2025-10-16 04:24:13,243 - DEBUG - Route for VM MAC 02:89:86:74:8e:36 is consistent in agent 10.0.0.38 2025-10-16 04:24:16,453 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1025ms') 2025-10-16 04:24:16,453 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-23587683 failed! 2025-10-16 04:24:16,466 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 is 192.168.1.254 and allocation pool is NOT set 2025-10-16 04:24:20,552 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-10-16 04:24:20,552 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-23587683 failed! 2025-10-16 04:24:20,566 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 is 192.168.1.254 and allocation pool is NOT set 2025-10-16 04:24:24,650 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1012ms') 2025-10-16 04:24:24,650 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-23587683 failed! 2025-10-16 04:24:24,663 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 is 192.168.1.254 and allocation pool is NOT set 2025-10-16 04:24:28,743 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-10-16 04:24:28,743 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-23587683 failed! 2025-10-16 04:24:28,756 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 is 192.168.1.254 and allocation pool is NOT set 2025-10-16 04:24:32,840 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2025-10-16 04:24:32,840 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-23587683 failed! 2025-10-16 04:24:32,854 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 is 192.168.1.254 and allocation pool is NOT set 2025-10-16 04:24:36,937 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-10-16 04:24:36,937 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-23587683 failed! 2025-10-16 04:24:36,953 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 is 192.168.1.254 and allocation pool is NOT set 2025-10-16 04:24:41,031 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-10-16 04:24:41,031 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-23587683 failed! 2025-10-16 04:24:41,044 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 is 192.168.1.254 and allocation pool is NOT set 2025-10-16 04:24:45,119 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-10-16 04:24:45,120 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-23587683 failed! 2025-10-16 04:24:45,133 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 is 192.168.1.254 and allocation pool is NOT set 2025-10-16 04:24:47,200 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=12.7 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=3.27 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 3.273/7.979/12.685/4.706 ms') 2025-10-16 04:24:47,200 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-23587683 passed 2025-10-16 04:24:47,200 - INFO - VM ctest-vn1_vm2-23587683 verifications in Compute nodes passed 2025-10-16 04:24:47,404 - DEBUG - Validated VM route 192.168.1.251 in vrouter of 10.0.0.38 2025-10-16 04:24:47,411 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-10-16 04:24:47,411 - INFO - Validated routes of VM ctest-vn1_vm2-23587683 in all vrouters 2025-10-16 04:24:47,839 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2025-10-16 04:24:47,961 - DEBUG - Layer2 route found for VM MAC 02:89:86:74:8e:36 in Control-node 10.0.0.23 2025-10-16 04:24:47,961 - DEBUG - L2 Label for VM ctest-vn1_vm2-23587683 same between Control-node 10.0.0.23 and Agent, Expected: 34, Seen: 34 2025-10-16 04:24:47,961 - DEBUG - Starting all layer2 verification in 10.0.0.19 Control Node 2025-10-16 04:24:48,093 - DEBUG - Layer2 route found for VM MAC 02:89:86:74:8e:36 in Control-node 10.0.0.19 2025-10-16 04:24:48,093 - DEBUG - L2 Label for VM ctest-vn1_vm2-23587683 same between Control-node 10.0.0.19 and Agent, Expected: 34, Seen: 34 2025-10-16 04:24:48,093 - DEBUG - Starting all layer2 verification in 10.0.0.57 Control Node 2025-10-16 04:24:48,219 - DEBUG - Layer2 route found for VM MAC 02:89:86:74:8e:36 in Control-node 10.0.0.57 2025-10-16 04:24:48,219 - DEBUG - L2 Label for VM ctest-vn1_vm2-23587683 same between Control-node 10.0.0.57 and Agent, Expected: 34, Seen: 34 2025-10-16 04:24:48,219 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-23587683 passed 2025-10-16 04:24:48,219 - DEBUG - Verifying the vm in opserver 2025-10-16 04:24:48,219 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-10-16 04:24:48,219 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/012d19a3-853f-4c9e-b74b-a7e5e7c6a84f?flat 2025-10-16 04:24:48,283 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91100301:8986748e-3656-4445-bd8e-718fd493c180?flat 2025-10-16 04:24:48,300 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91100301:8986748e-3656-4445-bd8e-718fd493c180?flat 2025-10-16 04:24:48,320 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/012d19a3-853f-4c9e-b74b-a7e5e7c6a84f?flat 2025-10-16 04:24:48,336 - DEBUG - Verifying vm in vn uve 2025-10-16 04:24:48,336 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91100301:8986748e-3656-4445-bd8e-718fd493c180?flat 2025-10-16 04:24:48,354 - DEBUG - VM uve shows interface as {'mac_address': '02:89:86:74:8e:36', 'l2_active': True, 'vm_name': 'ctest-vn1_vm2-23587683', 'admin_state': True, 'label': 30, 'vn_uuid': '12d682ad-d9f7-41cf-89cf-045cfbd1f623', 'uuid': '8986748e-3656-4445-bd8e-718fd493c180', 'ip4_active': True, 'port_mirror_enabled': False, 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179', 'is_health_check_active': True, 'ip_address': '192.168.1.251', 'rx_vlan': 65535, 'tx_vlan': 65535, 'active': True, 'ip6_address': '::', '__T': 1760588654812741, 'ip6_active': False, 'gateway': '192.168.1.254', 'vhostuser_mode': 0, 'vm_uuid': '012d19a3-853f-4c9e-b74b-a7e5e7c6a84f', 'fixed_ip4_list': ['192.168.1.251']} 2025-10-16 04:24:48,354 - DEBUG - VM uve shows ip address as ['192.168.1.251', '::'] 2025-10-16 04:24:48,354 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 2025-10-16 04:24:48,354 - INFO - Verifying through opserver in 10.0.0.19 2025-10-16 04:24:48,354 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179?flat 2025-10-16 04:24:48,378 - DEBUG - expected vm list 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f 2025-10-16 04:24:48,378 - DEBUG - Extracted vm list ['012d19a3-853f-4c9e-b74b-a7e5e7c6a84f'] 2025-10-16 04:24:48,378 - INFO - VM 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f is present in default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 2025-10-16 04:24:48,378 - DEBUG - Verifying vm in vrouter uve 2025-10-16 04:24:48,378 - DEBUG - Getting info from collector 10.0.0.23.. 2025-10-16 04:24:48,378 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/012d19a3-853f-4c9e-b74b-a7e5e7c6a84f?flat 2025-10-16 04:24:48,440 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4339-1'] 2025-10-16 04:24:48,440 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-1:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:24:48,486 - DEBUG - Requesting: http://10.0.0.57:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-1?flat 2025-10-16 04:24:48,562 - DEBUG - VM 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f is present in vrouter an-jenkins-deploy-platform-ansible-os-4339-1 uve 2025-10-16 04:24:48,562 - DEBUG - Validated that VM ctest-vn1_vm2-23587683 is in Vrouter an-jenkins-deploy-platform-ansible-os-4339-1 UVE 2025-10-16 04:24:48,562 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-10-16 04:24:48,569 - DEBUG - Expected tap interface of VM uuid 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f is default-domain:ctest-TestBasicVMVN0-91100301:8986748e-3656-4445-bd8e-718fd493c180 2025-10-16 04:24:48,569 - DEBUG - Expected VN of VM uuid 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f is default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 2025-10-16 04:24:48,569 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-1:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:24:48,612 - DEBUG - Requesting: http://10.0.0.57:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-1?flat 2025-10-16 04:24:48,641 - DEBUG - VM 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f is present in vrouter an-jenkins-deploy-platform-ansible-os-4339-1 uve 2025-10-16 04:24:48,641 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-91100301:8986748e-3656-4445-bd8e-718fd493c180 of vm 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f is present in vrouter an-jenkins-deploy-platform-ansible-os-4339-1 uve 2025-10-16 04:24:48,641 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 of vm 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f is present in vrouter an-jenkins-deploy-platform-ansible-os-4339-1 uve 2025-10-16 04:24:48,641 - DEBUG - Verifying the 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f virtual network link through opserver 10.0.0.19 2025-10-16 04:24:48,641 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machines 2025-10-16 04:24:48,655 - DEBUG - VM link and name as {'name': '012d19a3-853f-4c9e-b74b-a7e5e7c6a84f', 'href': 'http://10.0.0.19:8081/analytics/uves/virtual-machine/012d19a3-853f-4c9e-b74b-a7e5e7c6a84f?flat'} 2025-10-16 04:24:48,655 - DEBUG - Verifying the 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f virtual network link through opserver 10.0.0.57 2025-10-16 04:24:48,655 - DEBUG - Requesting: http://10.0.0.57:8081/analytics/uves/virtual-machines 2025-10-16 04:24:48,705 - DEBUG - VM link and name as {'name': '012d19a3-853f-4c9e-b74b-a7e5e7c6a84f', 'href': 'http://10.0.0.57:8081/analytics/uves/virtual-machine/012d19a3-853f-4c9e-b74b-a7e5e7c6a84f?flat'} 2025-10-16 04:24:48,705 - DEBUG - Verifying the 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f virtual network link through opserver 10.0.0.23 2025-10-16 04:24:48,705 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-10-16 04:24:48,717 - DEBUG - VM link and name as {'name': '012d19a3-853f-4c9e-b74b-a7e5e7c6a84f', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/012d19a3-853f-4c9e-b74b-a7e5e7c6a84f?flat'} 2025-10-16 04:24:48,717 - INFO - VM ctest-vn1_vm2-23587683 validations in Opserver passed 2025-10-16 04:24:49,941 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4339-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4339-2) 2025-10-16 04:24:49,986 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-10-16 04:24:55,079 - DEBUG - VM is in ACTIVE state now 2025-10-16 04:24:55,079 - INFO - VM name : ctest-vn2_vm2-45800712 2025-10-16 04:24:55,162 - DEBUG - VM ctest-vn2_vm2-45800712 ID is b63488fc-79b4-48c6-857f-07e2535a468a 2025-10-16 04:24:55,241 - DEBUG - VM is in ACTIVE state now 2025-10-16 04:24:55,241 - DEBUG - VM ctest-vn2_vm2-45800712 launched on Node an-jenkins-deploy-platform-ansible-os-4339-2 2025-10-16 04:24:55,330 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/b63488fc-79b4-48c6-857f-07e2535a468a 2025-10-16 04:24:55,340 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/b63488fc-79b4-48c6-857f-07e2535a468a 2025-10-16 04:24:55,347 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/da14cb89-862d-4385-a634-d88898a1b6ec 2025-10-16 04:24:55,356 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/b63488fc-79b4-48c6-857f-07e2535a468a 2025-10-16 04:24:55,364 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/da14cb89-862d-4385-a634-d88898a1b6ec 2025-10-16 04:24:55,373 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/79028493-8e7c-464b-8cf1-d67477c1fd5f 2025-10-16 04:24:55,382 - DEBUG - Verifying in api server 10.0.0.23 2025-10-16 04:24:55,382 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/b63488fc-79b4-48c6-857f-07e2535a468a 2025-10-16 04:24:55,389 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/da14cb89-862d-4385-a634-d88898a1b6ec 2025-10-16 04:24:55,397 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/79028493-8e7c-464b-8cf1-d67477c1fd5f 2025-10-16 04:24:55,405 - INFO - VM ctest-vn2_vm2-45800712 verfication in all API Servers passed 2025-10-16 04:24:55,436 - DEBUG - VM ctest-vn2_vm2-45800712 Tap interface: {'index': '4', 'name': 'tapda14cb89-86', 'uuid': 'da14cb89-862d-4385-a634-d88898a1b6ec', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678:ctest-vn2-58198678', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678', 'vm_uuid': 'b63488fc-79b4-48c6-857f-07e2535a468a', 'vm_name': 'ctest-vn2_vm2-45800712', 'ip_addr': '192.168.2.251', 'mac_addr': '02:da:14:cb:89:86', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '131', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-91100301:da14cb89-862d-4385-a634-d88898a1b6ec', 'sg_uuid_list': ['d00d1c1c-903e-4107-af98-1b6cf220bd38'], 'static_route_list': None, 'vm_project_uuid': '03e29f65-50a8-4f9b-937e-9a83abd9c7d6', 'admin_state': 'Enabled', 'flow_key_idx': '26', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-16 04:24:55,436 - DEBUG - Agent 10.0.0.129 vrf name: default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678:ctest-vn2-58198678 2025-10-16 04:24:55,443 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678:ctest-vn2-58198678', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678', 'table_label': '-1', 'vxlan_id': '13', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.19:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-10-16 04:24:55,532 - DEBUG - VM ctest-vn2_vm2-45800712 : Tap interface tapda14cb89-86 is set to Active 2025-10-16 04:24:55,532 - DEBUG - Tap interface tapda14cb89-86 detail : {'index': '4', 'name': 'tapda14cb89-86', 'uuid': 'da14cb89-862d-4385-a634-d88898a1b6ec', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678:ctest-vn2-58198678', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678', 'vm_uuid': 'b63488fc-79b4-48c6-857f-07e2535a468a', 'vm_name': 'ctest-vn2_vm2-45800712', 'ip_addr': '192.168.2.251', 'mac_addr': '02:da:14:cb:89:86', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '131', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-91100301:da14cb89-862d-4385-a634-d88898a1b6ec', 'sg_uuid_list': ['d00d1c1c-903e-4107-af98-1b6cf220bd38'], 'static_route_list': None, 'vm_project_uuid': '03e29f65-50a8-4f9b-937e-9a83abd9c7d6', 'admin_state': 'Enabled', 'flow_key_idx': '26', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-16 04:24:55,642 - DEBUG - Starting Layer 2 verification in Agent 2025-10-16 04:24:55,655 - DEBUG - Layer 2 path is seen for VM MAC 02:da:14:cb:89:86 in agent 10.0.0.129 2025-10-16 04:24:55,655 - DEBUG - Active layer 2 route in agent is present for VMI tapda14cb89-86 2025-10-16 04:24:55,655 - DEBUG - L2 label(34) matches bw route table and interface table 2025-10-16 04:24:55,694 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 is found in Agent of node 10.0.0.129 2025-10-16 04:24:55,807 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 is consistent in agent 10.0.0.129 2025-10-16 04:24:55,808 - DEBUG - Route for VM IP ['192.168.2.251'] is consistent in agent 10.0.0.129 2025-10-16 04:24:55,808 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 verification for VM ctest-vn2_vm2-45800712 in Agent 10.0.0.129 passed 2025-10-16 04:24:55,919 - DEBUG - Starting all layer 2 verification in agent 10.0.0.129 2025-10-16 04:24:55,937 - DEBUG - Route for VM MAC 02:da:14:cb:89:86 is consistent in agent 10.0.0.129 2025-10-16 04:24:59,124 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms') 2025-10-16 04:24:59,125 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-45800712 failed! 2025-10-16 04:24:59,138 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 is 192.168.2.254 and allocation pool is NOT set 2025-10-16 04:25:03,227 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2025-10-16 04:25:03,227 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-45800712 failed! 2025-10-16 04:25:03,242 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 is 192.168.2.254 and allocation pool is NOT set 2025-10-16 04:25:07,322 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2025-10-16 04:25:07,322 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-45800712 failed! 2025-10-16 04:25:07,335 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 is 192.168.2.254 and allocation pool is NOT set 2025-10-16 04:25:11,424 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1021ms') 2025-10-16 04:25:11,425 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-45800712 failed! 2025-10-16 04:25:11,440 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 is 192.168.2.254 and allocation pool is NOT set 2025-10-16 04:25:15,516 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms') 2025-10-16 04:25:15,517 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-45800712 failed! 2025-10-16 04:25:15,531 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 is 192.168.2.254 and allocation pool is NOT set 2025-10-16 04:25:19,605 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms') 2025-10-16 04:25:19,605 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-45800712 failed! 2025-10-16 04:25:19,618 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 is 192.168.2.254 and allocation pool is NOT set 2025-10-16 04:25:23,704 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1021ms') 2025-10-16 04:25:23,705 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-45800712 failed! 2025-10-16 04:25:23,718 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 is 192.168.2.254 and allocation pool is NOT set 2025-10-16 04:25:27,797 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-10-16 04:25:27,797 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-45800712 failed! 2025-10-16 04:25:27,810 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 is 192.168.2.254 and allocation pool is NOT set 2025-10-16 04:25:31,897 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2025-10-16 04:25:31,897 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-45800712 failed! 2025-10-16 04:25:31,912 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 is 192.168.2.254 and allocation pool is NOT set 2025-10-16 04:25:35,990 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-10-16 04:25:35,991 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-45800712 failed! 2025-10-16 04:25:36,005 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 is 192.168.2.254 and allocation pool is NOT set 2025-10-16 04:25:38,079 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=9.95 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=2.03 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 2.034/5.992/9.950/3.958 ms') 2025-10-16 04:25:38,079 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-45800712 passed 2025-10-16 04:25:38,079 - INFO - VM ctest-vn2_vm2-45800712 verifications in Compute nodes passed 2025-10-16 04:25:38,299 - DEBUG - Validated VM route 192.168.2.251 in vrouter of 10.0.0.129 2025-10-16 04:25:38,306 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-10-16 04:25:38,306 - INFO - Validated routes of VM ctest-vn2_vm2-45800712 in all vrouters 2025-10-16 04:25:38,730 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2025-10-16 04:25:38,864 - DEBUG - Layer2 route found for VM MAC 02:da:14:cb:89:86 in Control-node 10.0.0.23 2025-10-16 04:25:38,865 - DEBUG - L2 Label for VM ctest-vn2_vm2-45800712 same between Control-node 10.0.0.23 and Agent, Expected: 34, Seen: 34 2025-10-16 04:25:38,865 - DEBUG - Starting all layer2 verification in 10.0.0.19 Control Node 2025-10-16 04:25:38,995 - DEBUG - Layer2 route found for VM MAC 02:da:14:cb:89:86 in Control-node 10.0.0.19 2025-10-16 04:25:38,995 - DEBUG - L2 Label for VM ctest-vn2_vm2-45800712 same between Control-node 10.0.0.19 and Agent, Expected: 34, Seen: 34 2025-10-16 04:25:38,995 - DEBUG - Starting all layer2 verification in 10.0.0.57 Control Node 2025-10-16 04:25:39,121 - DEBUG - Layer2 route found for VM MAC 02:da:14:cb:89:86 in Control-node 10.0.0.57 2025-10-16 04:25:39,122 - DEBUG - L2 Label for VM ctest-vn2_vm2-45800712 same between Control-node 10.0.0.57 and Agent, Expected: 34, Seen: 34 2025-10-16 04:25:39,122 - INFO - Verification in Control-nodes for VM ctest-vn2_vm2-45800712 passed 2025-10-16 04:25:39,122 - DEBUG - Verifying the vm in opserver 2025-10-16 04:25:39,122 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-10-16 04:25:39,122 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/b63488fc-79b4-48c6-857f-07e2535a468a?flat 2025-10-16 04:25:39,140 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91100301:da14cb89-862d-4385-a634-d88898a1b6ec?flat 2025-10-16 04:25:39,159 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91100301:da14cb89-862d-4385-a634-d88898a1b6ec?flat 2025-10-16 04:25:39,180 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/b63488fc-79b4-48c6-857f-07e2535a468a?flat 2025-10-16 04:25:39,195 - DEBUG - Verifying vm in vn uve 2025-10-16 04:25:39,195 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91100301:da14cb89-862d-4385-a634-d88898a1b6ec?flat 2025-10-16 04:25:39,212 - DEBUG - VM uve shows interface as {'ip_address': '192.168.2.251', 'ip4_active': True, 'admin_state': True, 'gateway': '192.168.2.254', 'uuid': 'da14cb89-862d-4385-a634-d88898a1b6ec', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678', 'vm_uuid': 'b63488fc-79b4-48c6-857f-07e2535a468a', 'vm_name': 'ctest-vn2_vm2-45800712', '__T': 1760588693517458, 'port_mirror_enabled': False, 'tx_vlan': 65535, 'rx_vlan': 65535, 'fixed_ip4_list': ['192.168.2.251'], 'label': 30, 'is_health_check_active': True, 'mac_address': '02:da:14:cb:89:86', 'vhostuser_mode': 0, 'active': True, 'ip6_active': False, 'vn_uuid': 'e6c28100-ef36-401e-b3dd-23ef1008c619', 'ip6_address': '::', 'l2_active': True} 2025-10-16 04:25:39,212 - DEBUG - VM uve shows ip address as ['192.168.2.251', '::'] 2025-10-16 04:25:39,212 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 2025-10-16 04:25:39,212 - INFO - Verifying through opserver in 10.0.0.19 2025-10-16 04:25:39,212 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678?flat 2025-10-16 04:25:39,235 - DEBUG - expected vm list b63488fc-79b4-48c6-857f-07e2535a468a 2025-10-16 04:25:39,235 - DEBUG - Extracted vm list ['b63488fc-79b4-48c6-857f-07e2535a468a'] 2025-10-16 04:25:39,235 - INFO - VM b63488fc-79b4-48c6-857f-07e2535a468a is present in default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 2025-10-16 04:25:39,235 - DEBUG - Verifying vm in vrouter uve 2025-10-16 04:25:39,236 - DEBUG - Getting info from collector 10.0.0.23.. 2025-10-16 04:25:39,236 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/b63488fc-79b4-48c6-857f-07e2535a468a?flat 2025-10-16 04:25:39,251 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4339-2'] 2025-10-16 04:25:39,251 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-2:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:25:39,294 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-2?flat 2025-10-16 04:25:39,325 - DEBUG - VM b63488fc-79b4-48c6-857f-07e2535a468a is present in vrouter an-jenkins-deploy-platform-ansible-os-4339-2 uve 2025-10-16 04:25:39,326 - DEBUG - Validated that VM ctest-vn2_vm2-45800712 is in Vrouter an-jenkins-deploy-platform-ansible-os-4339-2 UVE 2025-10-16 04:25:39,326 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-10-16 04:25:39,334 - DEBUG - Expected tap interface of VM uuid b63488fc-79b4-48c6-857f-07e2535a468a is default-domain:ctest-TestBasicVMVN0-91100301:da14cb89-862d-4385-a634-d88898a1b6ec 2025-10-16 04:25:39,334 - DEBUG - Expected VN of VM uuid b63488fc-79b4-48c6-857f-07e2535a468a is default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 2025-10-16 04:25:39,334 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-2:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:25:39,383 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-2?flat 2025-10-16 04:25:39,415 - DEBUG - VM b63488fc-79b4-48c6-857f-07e2535a468a is present in vrouter an-jenkins-deploy-platform-ansible-os-4339-2 uve 2025-10-16 04:25:39,415 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-91100301:da14cb89-862d-4385-a634-d88898a1b6ec of vm b63488fc-79b4-48c6-857f-07e2535a468a is present in vrouter an-jenkins-deploy-platform-ansible-os-4339-2 uve 2025-10-16 04:25:39,415 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 of vm b63488fc-79b4-48c6-857f-07e2535a468a is present in vrouter an-jenkins-deploy-platform-ansible-os-4339-2 uve 2025-10-16 04:25:39,415 - DEBUG - Verifying the b63488fc-79b4-48c6-857f-07e2535a468a virtual network link through opserver 10.0.0.19 2025-10-16 04:25:39,415 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machines 2025-10-16 04:25:39,428 - DEBUG - VM link and name as {'name': 'b63488fc-79b4-48c6-857f-07e2535a468a', 'href': 'http://10.0.0.19:8081/analytics/uves/virtual-machine/b63488fc-79b4-48c6-857f-07e2535a468a?flat'} 2025-10-16 04:25:39,428 - DEBUG - Verifying the b63488fc-79b4-48c6-857f-07e2535a468a virtual network link through opserver 10.0.0.57 2025-10-16 04:25:39,428 - DEBUG - Requesting: http://10.0.0.57:8081/analytics/uves/virtual-machines 2025-10-16 04:25:39,443 - DEBUG - VM link and name as {'name': 'b63488fc-79b4-48c6-857f-07e2535a468a', 'href': 'http://10.0.0.57:8081/analytics/uves/virtual-machine/b63488fc-79b4-48c6-857f-07e2535a468a?flat'} 2025-10-16 04:25:39,443 - DEBUG - Verifying the b63488fc-79b4-48c6-857f-07e2535a468a virtual network link through opserver 10.0.0.23 2025-10-16 04:25:39,443 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-10-16 04:25:39,457 - DEBUG - VM link and name as {'name': 'b63488fc-79b4-48c6-857f-07e2535a468a', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/b63488fc-79b4-48c6-857f-07e2535a468a?flat'} 2025-10-16 04:25:39,457 - INFO - VM ctest-vn2_vm2-45800712 validations in Opserver passed 2025-10-16 04:25:39,457 - INFO - Waiting for VM ctest-vn1_vm2-23587683 to be up.. 2025-10-16 04:25:39,457 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:25:39,457 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/012d19a3-853f-4c9e-b74b-a7e5e7c6a84f 2025-10-16 04:25:39,466 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/8986748e-3656-4445-bd8e-718fd493c180 2025-10-16 04:25:40,626 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=3.03 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.541 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.541/1.784/3.028/1.243 ms') 2025-10-16 04:25:40,626 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-23587683 passed 2025-10-16 04:25:40,695 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:25:40,695 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-23587683, IP 192.168.1.251, Port 22 2025-10-16 04:25:40,869 - DEBUG - VM ctest-vn1_vm2-23587683 is ready for SSH connections 2025-10-16 04:25:40,869 - INFO - Waiting for VM ctest-vn2_vm2-45800712 to be up.. 2025-10-16 04:25:40,869 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:25:40,869 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/b63488fc-79b4-48c6-857f-07e2535a468a 2025-10-16 04:25:40,878 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/da14cb89-862d-4385-a634-d88898a1b6ec 2025-10-16 04:25:42,061 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=1.76 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=5.48 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 1.760/3.622/5.484/1.862 ms') 2025-10-16 04:25:42,061 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-45800712 passed 2025-10-16 04:25:42,131 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:25:42,131 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-45800712, IP 192.168.2.251, Port 22 2025-10-16 04:25:42,200 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-16 04:25:42,321 - DEBUG - VM ctest-vn2_vm2-45800712 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-16 04:25:47,322 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:25:47,322 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-45800712, IP 192.168.2.251, Port 22 2025-10-16 04:25:47,388 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-16 04:25:47,469 - DEBUG - VM ctest-vn2_vm2-45800712 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-16 04:25:52,470 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:25:52,470 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-45800712, IP 192.168.2.251, Port 22 2025-10-16 04:25:52,536 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-16 04:25:52,637 - DEBUG - VM ctest-vn2_vm2-45800712 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-16 04:25:57,637 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:25:57,637 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-45800712, IP 192.168.2.251, Port 22 2025-10-16 04:25:57,704 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-16 04:25:57,783 - DEBUG - VM ctest-vn2_vm2-45800712 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-16 04:26:02,784 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:26:02,784 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-45800712, IP 192.168.2.251, Port 22 2025-10-16 04:26:02,959 - DEBUG - VM ctest-vn2_vm2-45800712 is ready for SSH connections 2025-10-16 04:26:02,959 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-16 04:26:02,960 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.251, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.38, gateway password: c0ntrail123 2025-10-16 04:26:02,960 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.251 2025-10-16 04:26:42,958 - DEBUG - PING 192.168.2.251 (192.168.2.251) 56(84) bytes of data. 64 bytes from 192.168.2.251: icmp_req=1 ttl=63 time=8.56 ms 64 bytes from 192.168.2.251: icmp_req=2 ttl=63 time=1.32 ms 64 bytes from 192.168.2.251: icmp_req=3 ttl=63 time=1.28 ms --- 192.168.2.251 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 1.284/3.722/8.562/3.422 ms 2025-10-16 04:26:42,959 - INFO - Ping to IP 192.168.2.251 from VM ctest-vn1_vm2-23587683 passed 2025-10-16 04:26:42,959 - INFO - Deleting VM ctest-vn2_vm2-45800712 2025-10-16 04:26:43,037 - DEBUG - Verifying in api server 10.0.0.23 2025-10-16 04:26:43,037 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/b63488fc-79b4-48c6-857f-07e2535a468a 2025-10-16 04:26:43,047 - DEBUG - VM ID b63488fc-79b4-48c6-857f-07e2535a468a of VM ctest-vn2_vm2-45800712 is still found in API Server 2025-10-16 04:26:45,048 - DEBUG - Verifying in api server 10.0.0.23 2025-10-16 04:26:45,048 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/b63488fc-79b4-48c6-857f-07e2535a468a 2025-10-16 04:26:45,055 - DEBUG - Response Code: 404 2025-10-16 04:26:45,055 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/b63488fc-79b4-48c6-857f-07e2535a468a 2025-10-16 04:26:45,060 - DEBUG - Response Code: 404 2025-10-16 04:26:45,060 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/b63488fc-79b4-48c6-857f-07e2535a468a 2025-10-16 04:26:45,066 - DEBUG - Response Code: 404 2025-10-16 04:26:45,066 - INFO - VM ctest-vn2_vm2-45800712 is fully removed in API-Server 2025-10-16 04:26:45,229 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-10-16 04:26:45,273 - DEBUG - Validated that vrouter 10.0.0.38 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 2025-10-16 04:26:45,439 - DEBUG - VM route 192.168.2.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 of compute 10.0.0.129 2025-10-16 04:26:45,448 - DEBUG - Validated that vrouter 10.0.0.129 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 2025-10-16 04:26:45,448 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 2025-10-16 04:26:45,448 - INFO - VM ctest-vn2_vm2-45800712 is removed in Compute, and routes are removed in all compute nodes 2025-10-16 04:26:45,695 - INFO - Routes for VM ctest-vn2_vm2-45800712 is removed in all control-nodes 2025-10-16 04:26:45,696 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-2:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:26:45,748 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-2?flat 2025-10-16 04:26:45,784 - DEBUG - VM b63488fc-79b4-48c6-857f-07e2535a468a is still present in vrouter an-jenkins-deploy-platform-ansible-os-4339-2 uve 2025-10-16 04:26:45,784 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-16 04:26:45,784 - DEBUG - interface for vm b63488fc-79b4-48c6-857f-07e2535a468a deleted from vrouter uve ... 2025-10-16 04:26:45,784 - DEBUG - Deleted interface not in error interface list ... 2025-10-16 04:26:48,785 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-2:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:26:48,831 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-2?flat 2025-10-16 04:26:48,861 - DEBUG - VM b63488fc-79b4-48c6-857f-07e2535a468a is still present in vrouter an-jenkins-deploy-platform-ansible-os-4339-2 uve 2025-10-16 04:26:48,861 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-16 04:26:48,861 - DEBUG - interface for vm b63488fc-79b4-48c6-857f-07e2535a468a deleted from vrouter uve ... 2025-10-16 04:26:48,861 - DEBUG - Deleted interface not in error interface list ... 2025-10-16 04:26:51,862 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-2:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:26:51,927 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-2?flat 2025-10-16 04:26:51,976 - DEBUG - VM b63488fc-79b4-48c6-857f-07e2535a468a is still present in vrouter an-jenkins-deploy-platform-ansible-os-4339-2 uve 2025-10-16 04:26:51,977 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-16 04:26:51,977 - DEBUG - interface for vm b63488fc-79b4-48c6-857f-07e2535a468a deleted from vrouter uve ... 2025-10-16 04:26:51,977 - DEBUG - Deleted interface not in error interface list ... 2025-10-16 04:26:54,977 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-2:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:26:55,021 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-2?flat 2025-10-16 04:26:55,053 - DEBUG - VM b63488fc-79b4-48c6-857f-07e2535a468a is not present in vrouter an-jenkins-deploy-platform-ansible-os-4339-2 uve 2025-10-16 04:26:55,053 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-16 04:26:55,053 - DEBUG - interface for vm b63488fc-79b4-48c6-857f-07e2535a468a deleted from vrouter uve ... 2025-10-16 04:26:55,053 - DEBUG - Deleted interface not in error interface list ... 2025-10-16 04:26:55,053 - DEBUG - Validated that VM b63488fc-79b4-48c6-857f-07e2535a468a is removed in Vrouter UVE 2025-10-16 04:26:55,053 - DEBUG - Verifying through opserver in 10.0.0.19 2025-10-16 04:26:55,053 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678?flat 2025-10-16 04:26:55,077 - DEBUG - b63488fc-79b4-48c6-857f-07e2535a468a BM not in default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn2-58198678 uve 2025-10-16 04:26:55,077 - DEBUG - Validated that VM b63488fc-79b4-48c6-857f-07e2535a468a is not present in the VN UVE 2025-10-16 04:26:55,077 - DEBUG - Verifying the b63488fc-79b4-48c6-857f-07e2535a468a virtual network link through opserver 10.0.0.19 2025-10-16 04:26:55,077 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machines 2025-10-16 04:26:55,089 - INFO - b63488fc-79b4-48c6-857f-07e2535a468a vm uve deleted from opserver 2025-10-16 04:26:55,089 - DEBUG - Verifying the b63488fc-79b4-48c6-857f-07e2535a468a virtual network link through opserver 10.0.0.57 2025-10-16 04:26:55,089 - DEBUG - Requesting: http://10.0.0.57:8081/analytics/uves/virtual-machines 2025-10-16 04:26:55,105 - INFO - b63488fc-79b4-48c6-857f-07e2535a468a vm uve deleted from opserver 2025-10-16 04:26:55,105 - DEBUG - Verifying the b63488fc-79b4-48c6-857f-07e2535a468a virtual network link through opserver 10.0.0.23 2025-10-16 04:26:55,105 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-10-16 04:26:55,119 - INFO - b63488fc-79b4-48c6-857f-07e2535a468a vm uve deleted from opserver 2025-10-16 04:26:55,119 - INFO - Validated that VM b63488fc-79b4-48c6-857f-07e2535a468a is removed from Opserver 2025-10-16 04:26:55,119 - INFO - Deleting VM ctest-vn1_vm2-23587683 2025-10-16 04:26:55,189 - DEBUG - Verifying in api server 10.0.0.23 2025-10-16 04:26:55,189 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/012d19a3-853f-4c9e-b74b-a7e5e7c6a84f 2025-10-16 04:26:55,198 - DEBUG - VM ID 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f of VM ctest-vn1_vm2-23587683 is still found in API Server 2025-10-16 04:26:57,198 - DEBUG - Verifying in api server 10.0.0.23 2025-10-16 04:26:57,198 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/012d19a3-853f-4c9e-b74b-a7e5e7c6a84f 2025-10-16 04:26:57,205 - DEBUG - Response Code: 404 2025-10-16 04:26:57,205 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/012d19a3-853f-4c9e-b74b-a7e5e7c6a84f 2025-10-16 04:26:57,210 - DEBUG - Response Code: 404 2025-10-16 04:26:57,210 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/012d19a3-853f-4c9e-b74b-a7e5e7c6a84f 2025-10-16 04:26:57,257 - DEBUG - Response Code: 404 2025-10-16 04:26:57,257 - INFO - VM ctest-vn1_vm2-23587683 is fully removed in API-Server 2025-10-16 04:26:57,413 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-10-16 04:26:57,581 - DEBUG - VM route 192.168.1.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 of compute 10.0.0.38 2025-10-16 04:26:57,587 - DEBUG - Validated that vrouter 10.0.0.38 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 2025-10-16 04:26:57,587 - DEBUG - Validated that vrouter 10.0.0.129 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 2025-10-16 04:26:57,587 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 2025-10-16 04:26:57,587 - INFO - VM ctest-vn1_vm2-23587683 is removed in Compute, and routes are removed in all compute nodes 2025-10-16 04:26:57,802 - INFO - Routes for VM ctest-vn1_vm2-23587683 is removed in all control-nodes 2025-10-16 04:26:57,802 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-1:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:26:57,851 - DEBUG - Requesting: http://10.0.0.57:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-1?flat 2025-10-16 04:26:57,882 - DEBUG - VM 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f is still present in vrouter an-jenkins-deploy-platform-ansible-os-4339-1 uve 2025-10-16 04:26:57,882 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-16 04:26:57,882 - DEBUG - interface for vm 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f deleted from vrouter uve ... 2025-10-16 04:26:57,882 - DEBUG - Deleted interface not in error interface list ... 2025-10-16 04:27:00,883 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-1:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:27:00,936 - DEBUG - Requesting: http://10.0.0.57:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-1?flat 2025-10-16 04:27:00,966 - DEBUG - VM 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f is still present in vrouter an-jenkins-deploy-platform-ansible-os-4339-1 uve 2025-10-16 04:27:00,966 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-16 04:27:00,966 - DEBUG - interface for vm 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f deleted from vrouter uve ... 2025-10-16 04:27:00,966 - DEBUG - Deleted interface not in error interface list ... 2025-10-16 04:27:03,967 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-1:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:27:04,017 - DEBUG - Requesting: http://10.0.0.57:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-1?flat 2025-10-16 04:27:04,047 - DEBUG - VM 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f is still present in vrouter an-jenkins-deploy-platform-ansible-os-4339-1 uve 2025-10-16 04:27:04,047 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-16 04:27:04,047 - DEBUG - interface for vm 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f deleted from vrouter uve ... 2025-10-16 04:27:04,047 - DEBUG - Deleted interface not in error interface list ... 2025-10-16 04:27:07,048 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-1:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:27:07,099 - DEBUG - Requesting: http://10.0.0.57:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-1?flat 2025-10-16 04:27:07,128 - DEBUG - VM 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f is still present in vrouter an-jenkins-deploy-platform-ansible-os-4339-1 uve 2025-10-16 04:27:07,128 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-16 04:27:07,128 - DEBUG - interface for vm 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f deleted from vrouter uve ... 2025-10-16 04:27:07,128 - DEBUG - Deleted interface not in error interface list ... 2025-10-16 04:27:10,129 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-1:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:27:10,187 - DEBUG - Requesting: http://10.0.0.57:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-1?flat 2025-10-16 04:27:10,222 - DEBUG - VM 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f is still present in vrouter an-jenkins-deploy-platform-ansible-os-4339-1 uve 2025-10-16 04:27:10,222 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-16 04:27:10,222 - DEBUG - interface for vm 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f deleted from vrouter uve ... 2025-10-16 04:27:10,222 - DEBUG - Deleted interface not in error interface list ... 2025-10-16 04:27:13,223 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-1:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:27:13,267 - DEBUG - Requesting: http://10.0.0.57:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-1?flat 2025-10-16 04:27:13,296 - DEBUG - VM 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f is still present in vrouter an-jenkins-deploy-platform-ansible-os-4339-1 uve 2025-10-16 04:27:13,296 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-16 04:27:13,296 - DEBUG - interface for vm 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f deleted from vrouter uve ... 2025-10-16 04:27:13,296 - DEBUG - Deleted interface not in error interface list ... 2025-10-16 04:27:16,296 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4339-1:Compute:contrail-vrouter-agent:0?flat 2025-10-16 04:27:16,342 - DEBUG - Requesting: http://10.0.0.57:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4339-1?flat 2025-10-16 04:27:16,373 - DEBUG - VM 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f is not present in vrouter an-jenkins-deploy-platform-ansible-os-4339-1 uve 2025-10-16 04:27:16,373 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-16 04:27:16,373 - DEBUG - interface for vm 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f deleted from vrouter uve ... 2025-10-16 04:27:16,373 - DEBUG - Deleted interface not in error interface list ... 2025-10-16 04:27:16,373 - DEBUG - Validated that VM 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f is removed in Vrouter UVE 2025-10-16 04:27:16,373 - DEBUG - Verifying through opserver in 10.0.0.19 2025-10-16 04:27:16,373 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179?flat 2025-10-16 04:27:16,396 - DEBUG - 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f BM not in default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 uve 2025-10-16 04:27:16,396 - DEBUG - Validated that VM 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f is not present in the VN UVE 2025-10-16 04:27:16,396 - DEBUG - Verifying the 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f virtual network link through opserver 10.0.0.19 2025-10-16 04:27:16,396 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machines 2025-10-16 04:27:16,407 - INFO - 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f vm uve deleted from opserver 2025-10-16 04:27:16,408 - DEBUG - Verifying the 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f virtual network link through opserver 10.0.0.57 2025-10-16 04:27:16,408 - DEBUG - Requesting: http://10.0.0.57:8081/analytics/uves/virtual-machines 2025-10-16 04:27:16,419 - INFO - 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f vm uve deleted from opserver 2025-10-16 04:27:16,419 - DEBUG - Verifying the 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f virtual network link through opserver 10.0.0.23 2025-10-16 04:27:16,419 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-10-16 04:27:16,432 - INFO - 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f vm uve deleted from opserver 2025-10-16 04:27:16,432 - INFO - Validated that VM 012d19a3-853f-4c9e-b74b-a7e5e7c6a84f is removed from Opserver 2025-10-16 04:27:16,432 - INFO - Deleting VN ctest-vn3-24509381 2025-10-16 04:27:16,549 - INFO - Deleting VM ctest-vn2_vm1-98937197 2025-10-16 04:27:16,633 - INFO - Deleting VM ctest-vn1_vm1-67798974 2025-10-16 04:27:16,865 - INFO - Deleting VN ctest-vn2-58198678 2025-10-16 04:27:16,914 - DEBUG - RefsExistError Delete when resource still referred: ['http://10.0.0.23:8082/virtual-machine-interface/74a143f2-29c4-4841-973a-5980770490bb'] while deleting VN ctest-vn2-58198678..Will retry 2025-10-16 04:27:19,022 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/e64ce128-57bf-4ddf-b2c5-d5cfe5c76a7a 2025-10-16 04:27:19,028 - DEBUG - Response Code: 404 2025-10-16 04:27:19,029 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-10-16 04:27:19,035 - DEBUG - Requesting: http://10.0.0.19:8082/domain/446ea2e6-9d96-4973-a55a-3f633ea4d0f9 2025-10-16 04:27:19,048 - DEBUG - Requesting: http://10.0.0.19:8082/project/03e29f65-50a8-4f9b-937e-9a83abd9c7d6 2025-10-16 04:27:19,149 - INFO - Validated that VN ctest-vn2-58198678 is not found in API Server 2025-10-16 04:27:19,159 - DEBUG - VN ctest-vn2-58198678 is not present in Agent 10.0.0.38 2025-10-16 04:27:19,170 - DEBUG - VN ctest-vn2-58198678 is not present in Agent 10.0.0.129 2025-10-16 04:27:19,170 - INFO - Validated that VN ctest-vn2-58198678 is not in any agent 2025-10-16 04:27:19,199 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn2-58198678 info 2025-10-16 04:27:19,338 - INFO - Deleting VN ctest-vn1-59270179 2025-10-16 04:27:19,441 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/5abac593-02f5-480b-b1f1-dc2d65495173 2025-10-16 04:27:19,448 - DEBUG - Response Code: 404 2025-10-16 04:27:19,448 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-10-16 04:27:19,453 - DEBUG - Requesting: http://10.0.0.19:8082/domain/446ea2e6-9d96-4973-a55a-3f633ea4d0f9 2025-10-16 04:27:19,464 - DEBUG - Requesting: http://10.0.0.19:8082/project/03e29f65-50a8-4f9b-937e-9a83abd9c7d6 2025-10-16 04:27:19,550 - INFO - Validated that VN ctest-vn1-59270179 is not found in API Server 2025-10-16 04:27:19,563 - DEBUG - VN ctest-vn1-59270179 is not present in Agent 10.0.0.38 2025-10-16 04:27:19,575 - DEBUG - VN ctest-vn1-59270179 is not present in Agent 10.0.0.129 2025-10-16 04:27:19,575 - INFO - Validated that VN ctest-vn1-59270179 is not in any agent 2025-10-16 04:27:19,585 - DEBUG - VRF 2 is not seen in agent 10.0.0.38 2025-10-16 04:27:19,592 - DEBUG - Vrouter 10.0.0.38 does not have vrf 2 for VN ctest-vn1-59270179 2025-10-16 04:27:19,592 - INFO - Validated that all vrouters do not have the route table for VN default-domain:ctest-TestBasicVMVN0-91100301:ctest-vn1-59270179 2025-10-16 04:27:19,615 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-59270179 info 2025-10-16 04:27:19,650 - INFO - Deleted policy policy2 2025-10-16 04:27:19,684 - INFO - Deleted policy policy1 2025-10-16 04:27:20,980 - DEBUG - Skipping xmpp flap check 2025-10-16 04:27:20,980 - INFO - END TEST : test_process_restart_in_policy_between_vns : PASSED[0:06:40] 2025-10-16 04:27:20,980 - INFO - -------------------------------------------------------------------------------- 2025-10-16 04:27:20,983 - INFO - ================================================================================ 2025-10-16 04:27:20,983 - INFO - STARTING TEST : test_underlay_broadcast_traffic_handling 2025-10-16 04:27:20,983 - INFO - TEST DESCRIPTION : Test the underlay brocast traffic handling by vrouter. (Bug-1545229). 1. Send broadcast traffic from one compute node. 2. Other compute in same subnet should receive that traffic. 3. Receiving compute should treat this traffic as underlay. 4. Compute should not replicate the packet and send the copy back. Pass criteria: Step 3-4 should pass Maintainer : chhandak@juniper.net 2025-10-16 04:27:22,241 - DEBUG - Skipping xmpp flap check 2025-10-16 04:27:22,242 - INFO - Initial checks done. Running the testcase now 2025-10-16 04:27:22,242 - INFO - 2025-10-16 04:27:22,656 - INFO - Agent interface name: ens6 2025-10-16 04:27:22,738 - DEBUG - Executing command: sudo tcpdump -nni ens6 -U host 0.0.0.0 -w /tmp/ens6_ctest-random-26445910.pcap 2025-10-16 04:27:28,153 - INFO - Pinging broacast address 0.0.0.0 from compute 10.20.0.145 2025-10-16 04:27:42,466 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-10-16 04:27:44,603 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-26445910.pcap | grep -c length 2025-10-16 04:27:44,620 - DEBUG - STDOUT: 5 2025-10-16 04:27:44,621 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-26445910.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-10-16 04:27:44,621 - WARNING - 5 packets are found in tcpdump output file /tmp/ens6_ctest-random-26445910.pcap but expected 10 2025-10-16 04:27:46,622 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-26445910.pcap | grep -c length 2025-10-16 04:27:46,639 - DEBUG - STDOUT: 5 2025-10-16 04:27:46,639 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-26445910.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-10-16 04:27:46,639 - WARNING - 5 packets are found in tcpdump output file /tmp/ens6_ctest-random-26445910.pcap but expected 10 2025-10-16 04:27:48,640 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-26445910.pcap | grep -c length 2025-10-16 04:27:48,655 - DEBUG - STDOUT: 5 2025-10-16 04:27:48,655 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-26445910.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-10-16 04:27:48,655 - WARNING - 5 packets are found in tcpdump output file /tmp/ens6_ctest-random-26445910.pcap but expected 10 2025-10-16 04:27:50,656 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-26445910.pcap | grep -c length 2025-10-16 04:27:50,672 - DEBUG - STDOUT: 5 2025-10-16 04:27:50,672 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-26445910.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-10-16 04:27:50,672 - WARNING - 5 packets are found in tcpdump output file /tmp/ens6_ctest-random-26445910.pcap but expected 10 2025-10-16 04:27:52,672 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-26445910.pcap | grep -c length 2025-10-16 04:27:52,687 - DEBUG - STDOUT: 5 2025-10-16 04:27:52,687 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-26445910.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-10-16 04:27:52,687 - WARNING - 5 packets are found in tcpdump output file /tmp/ens6_ctest-random-26445910.pcap but expected 10 2025-10-16 04:27:54,688 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-26445910.pcap | grep -c length 2025-10-16 04:27:54,705 - DEBUG - STDOUT: 5 2025-10-16 04:27:54,705 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-26445910.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-10-16 04:27:54,705 - WARNING - 5 packets are found in tcpdump output file /tmp/ens6_ctest-random-26445910.pcap but expected 10 2025-10-16 04:27:56,705 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-26445910.pcap | grep -c length 2025-10-16 04:27:56,722 - DEBUG - STDOUT: 5 2025-10-16 04:27:56,722 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-26445910.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-10-16 04:27:56,722 - WARNING - 5 packets are found in tcpdump output file /tmp/ens6_ctest-random-26445910.pcap but expected 10 2025-10-16 04:27:58,006 - ERROR - AssertionError Python 3.9.21: /usr/bin/python3 Thu Oct 16 04:27:56 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={}) 78 log.info('Initial checks done. Running the testcase now') 79 log.info('') 80 result = function(self, *args, **kwargs) 81 if self.inputs.upgrade: 82 pid = os.getpid() result = None function = self = args = () kwargs = {} /contrail-test/serial_scripts/vm_regression/test_vm_serial.py in test_underlay_broadcast_traffic_handling(self=) 948 949 # Analyze pcap 950 assert verify_tcpdump_count(self, session, pcap, exp_count=packet_count), "There should only be %s\ 951 packet from source %s on compute %s" %(packet_count, broadcast_address, compute_ip) 952 self.logger.info( global verify_tcpdump_count = .deco_retry..f_retry> self = session = pcap = '/tmp/ens6_ctest-random-26445910.pcap' exp_count undefined packet_count = 10 broadcast_address = '0.0.0.0' compute_ip = '10.0.0.129' AssertionError: There should only be 10 packet from source 0.0.0.0 on compute 10.0.0.129 __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {} __dir__ = __doc__ = 'Assertion failed.' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = args = ('There should only be 10 ... packet from source 0.0.0.0 on compute 10.0.0.129',) with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper result = function(self, *args, **kwargs) File "/contrail-test/serial_scripts/vm_regression/test_vm_serial.py", line 950, in test_underlay_broadcast_traffic_handling assert verify_tcpdump_count(self, session, pcap, exp_count=packet_count), "There should only be %s\ AssertionError: There should only be 10 packet from source 0.0.0.0 on compute 10.0.0.129 2025-10-16 04:27:58,006 - DEBUG - Skipping xmpp flap check 2025-10-16 04:27:58,006 - INFO - 2025-10-16 04:27:58,006 - INFO - END TEST : test_underlay_broadcast_traffic_handling : FAILED[0:00:38] 2025-10-16 04:27:58,006 - INFO - -------------------------------------------------------------------------------- 2025-10-16 04:27:58,755 - INFO - Deleted project: ctest-TestBasicVMVN0-91100301, ID : 03e29f65-50a8-4f9b-937e-9a83abd9c7d6