2025-11-20 03:55:22,218 - INFO - Domain Default found not creating 2025-11-20 03:55:22,375 - INFO - Project ctest-TestBasicVMVN0-17624131 not found, creating it 2025-11-20 03:55:22,950 - INFO - Created Project:ctest-TestBasicVMVN0-17624131, ID : 3339bf5d-318a-4edf-9cb8-4a7fd32a3af6 2025-11-20 03:55:24,475 - INFO - ================================================================================ 2025-11-20 03:55:24,475 - INFO - STARTING TEST : test_control_node_switchover 2025-11-20 03:55:24,475 - INFO - TEST DESCRIPTION : Stop the control node and check peering with agent fallback to other control node. 1. Pick one VN from respource pool which has 2 VM's in it 2. Verify ping between VM's 3. Find active control node in cluster by agent inspect 4. Stop control service on active control node 5. Verify agents are connected to new active control-node using xmpp connections 6. Bring back control service on previous active node 7. Verify ping between VM's again after bringing up control serveice Pass criteria: Step 2,5 and 7 should pass 2025-11-20 03:55:25,728 - DEBUG - Skipping xmpp flap check 2025-11-20 03:55:25,729 - INFO - Initial checks done. Running the testcase now 2025-11-20 03:55:25,729 - INFO - 2025-11-20 03:55:26,418 - DEBUG - Response for create_network : {'network': {'id': 'c6a6d9b7-04be-4849-a44b-71d9d93fc5ff', 'name': 'ctest-vn1-56878927', 'tenant_id': '3339bf5d318a4edf9cb84a7fd32a3af6', 'project_id': '3339bf5d318a4edf9cb84a7fd32a3af6', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestBasicVMVN0-17624131', 'ctest-vn1-56878927'], 'port_security_enabled': True, 'description': ''}} 2025-11-20 03:55:26,703 - DEBUG - Response for create_subnet : {'subnet': {'id': '910ffad9-630b-4186-81d7-73851fc3f45a', 'name': '', 'tenant_id': '3339bf5d318a4edf9cb84a7fd32a3af6', 'network_id': 'c6a6d9b7-04be-4849-a44b-71d9d93fc5ff', 'ip_version': 4, 'cidr': '192.168.1.0/24', 'allocation_pools': [{'start': '192.168.1.2', 'end': '192.168.1.254'}], 'gateway_ip': '192.168.1.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '192.168.1.2', 'tags': [], 'project_id': '3339bf5d318a4edf9cb84a7fd32a3af6'}} 2025-11-20 03:55:26,727 - INFO - Created VN ctest-vn1-56878927 2025-11-20 03:55:26,741 - DEBUG - VN ctest-vn1-56878927 UUID is c6a6d9b7-04be-4849-a44b-71d9d93fc5ff 2025-11-20 03:55:26,814 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-11-20 03:55:27,129 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-11-20 03:55:27,163 - DEBUG - Requesting: http://10.0.0.23:8082/domain/785d7fdc-4b61-45a1-9add-76a6d523389b 2025-11-20 03:55:27,184 - DEBUG - Requesting: http://10.0.0.23:8082/project/3339bf5d-318a-4edf-9cb8-4a7fd32a3af6 2025-11-20 03:55:27,288 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/c6a6d9b7-04be-4849-a44b-71d9d93fc5ff 2025-11-20 03:55:27,299 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/c6a6d9b7-04be-4849-a44b-71d9d93fc5ff 2025-11-20 03:55:27,307 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/cce25481-1121-4642-a92e-da98cda2e575 2025-11-20 03:55:27,315 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/cce25481-1121-4642-a92e-da98cda2e575 2025-11-20 03:55:27,323 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/e6eadb49-e10a-487b-a369-290c58a96ccd 2025-11-20 03:55:27,330 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-11-20 03:55:27,330 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/c6a6d9b7-04be-4849-a44b-71d9d93fc5ff 2025-11-20 03:55:27,341 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/cce25481-1121-4642-a92e-da98cda2e575 2025-11-20 03:55:27,356 - INFO - Verified VN network id 15 for VN c6a6d9b7-04be-4849-a44b-71d9d93fc5ff 2025-11-20 03:55:27,356 - INFO - Verifications in API Server for VN ctest-vn1-56878927 passed 2025-11-20 03:55:27,357 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/c6a6d9b7-04be-4849-a44b-71d9d93fc5ff 2025-11-20 03:55:27,369 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/cce25481-1121-4642-a92e-da98cda2e575 2025-11-20 03:55:27,376 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/cce25481-1121-4642-a92e-da98cda2e575 2025-11-20 03:55:27,385 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/e6eadb49-e10a-487b-a369-290c58a96ccd 2025-11-20 03:55:27,401 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-17624131', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '14314367845725521993', 'uuid-lslong': '11838681225940157951'}, 'enable': 'true', 'created': '2025-11-20T03:55:26', 'last-modified': '2025-11-20T03:55:26', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.745279'} 2025-11-20 03:55:27,409 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-11-20 03:55:27,420 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-17624131', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '14314367845725521993', 'uuid-lslong': '11838681225940157951'}, 'enable': 'true', 'created': '2025-11-20T03:55:26', 'last-modified': '2025-11-20T03:55:26', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.776515'} 2025-11-20 03:55:27,427 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-11-20 03:55:27,432 - DEBUG - Control-node 10.0.0.22 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-17624131', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '14314367845725521993', 'uuid-lslong': '11838681225940157951'}, 'enable': 'true', 'created': '2025-11-20T03:55:26', 'last-modified': '2025-11-20T03:55:26', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.783047'} 2025-11-20 03:55:27,438 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-11-20 03:55:27,438 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-56878927 passed 2025-11-20 03:55:27,438 - DEBUG - ====Verifying policy data for ctest-vn1-56878927 in API_Server ====== 2025-11-20 03:55:27,438 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-11-20 03:55:27,446 - DEBUG - Requesting: http://10.0.0.23:8082/domain/785d7fdc-4b61-45a1-9add-76a6d523389b 2025-11-20 03:55:27,457 - DEBUG - Requesting: http://10.0.0.23:8082/project/3339bf5d-318a-4edf-9cb8-4a7fd32a3af6 2025-11-20 03:55:27,548 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/c6a6d9b7-04be-4849-a44b-71d9d93fc5ff 2025-11-20 03:55:27,556 - DEBUG - =>VN ctest-vn1-56878927 has no policy to be verified 2025-11-20 03:55:27,556 - DEBUG - Verifying the vn in opserver 2025-11-20 03:55:27,556 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 virtual network link through opserver 10.0.0.23 2025-11-20 03:55:27,556 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-11-20 03:55:27,855 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-11-20 03:55:27,925 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927?flat'} 2025-11-20 03:55:27,925 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 virtual network link through opserver 10.0.0.50 2025-11-20 03:55:27,925 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2025-11-20 03:55:28,229 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2025-11-20 03:55:28,311 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927?flat'} 2025-11-20 03:55:28,311 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 virtual network link through opserver 10.0.0.22 2025-11-20 03:55:28,312 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-networks 2025-11-20 03:55:28,619 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-networks 2025-11-20 03:55:28,685 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927?flat'} 2025-11-20 03:55:28,685 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 is found in opserver 2025-11-20 03:55:28,709 - DEBUG - Do not have enough data to verify VN in agent 2025-11-20 03:55:28,723 - DEBUG - VRF ids for VN ctest-vn1-56878927: {} 2025-11-20 03:55:28,776 - DEBUG - Services list from nova: [, , , ] 2025-11-20 03:55:29,814 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4768-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4768-2) 2025-11-20 03:55:31,156 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4768-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4768-1) 2025-11-20 03:55:31,156 - INFO - Waiting for VM ctest-vn1_vm1-82591532 to be up.. 2025-11-20 03:55:31,249 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-11-20 03:55:36,326 - DEBUG - VM is in ACTIVE state now 2025-11-20 03:55:36,326 - INFO - VM name : ctest-vn1_vm1-82591532 2025-11-20 03:55:36,410 - DEBUG - VM ctest-vn1_vm1-82591532 ID is 123b0729-3eb7-4874-a0b0-968ababe97e2 2025-11-20 03:55:36,436 - DEBUG - VM ctest-vn1_vm1-82591532 launched on Node an-jenkins-deploy-platform-ansible-os-4768-2 2025-11-20 03:55:36,513 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/123b0729-3eb7-4874-a0b0-968ababe97e2 2025-11-20 03:55:36,809 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/123b0729-3eb7-4874-a0b0-968ababe97e2 2025-11-20 03:55:36,846 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine-interface/9da5ba90-570d-4544-9d68-0cd08fcc0f84 2025-11-20 03:55:40,029 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-11-20 03:55:40,029 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-82591532 failed! 2025-11-20 03:55:40,045 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 is 192.168.1.1 and allocation pool is NOT set 2025-11-20 03:55:44,132 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-11-20 03:55:44,132 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-82591532 failed! 2025-11-20 03:55:44,146 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 is 192.168.1.1 and allocation pool is NOT set 2025-11-20 03:55:48,224 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-11-20 03:55:48,225 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-82591532 failed! 2025-11-20 03:55:48,240 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 is 192.168.1.1 and allocation pool is NOT set 2025-11-20 03:55:52,322 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-11-20 03:55:52,322 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-82591532 failed! 2025-11-20 03:55:52,338 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 is 192.168.1.1 and allocation pool is NOT set 2025-11-20 03:55:56,413 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-11-20 03:55:56,413 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-82591532 failed! 2025-11-20 03:55:56,427 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 is 192.168.1.1 and allocation pool is NOT set 2025-11-20 03:56:00,513 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2025-11-20 03:56:00,513 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-82591532 failed! 2025-11-20 03:56:00,527 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 is 192.168.1.1 and allocation pool is NOT set 2025-11-20 03:56:04,605 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-11-20 03:56:04,605 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-82591532 failed! 2025-11-20 03:56:04,621 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 is 192.168.1.1 and allocation pool is NOT set 2025-11-20 03:56:08,703 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-11-20 03:56:08,703 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-82591532 failed! 2025-11-20 03:56:08,719 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 is 192.168.1.1 and allocation pool is NOT set 2025-11-20 03:56:10,790 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=5.13 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=6.28 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 5.127/5.702/6.277/0.575 ms') 2025-11-20 03:56:10,790 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-82591532 passed 2025-11-20 03:56:10,862 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 03:56:10,862 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-82591532, IP 192.168.1.3, Port 22 2025-11-20 03:56:10,930 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 03:56:11,041 - DEBUG - VM ctest-vn1_vm1-82591532 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 03:56:16,041 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 03:56:16,041 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-82591532, IP 192.168.1.3, Port 22 2025-11-20 03:56:16,110 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 03:56:16,197 - DEBUG - VM ctest-vn1_vm1-82591532 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 03:56:21,197 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 03:56:21,197 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-82591532, IP 192.168.1.3, Port 22 2025-11-20 03:56:21,266 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 03:56:21,347 - DEBUG - VM ctest-vn1_vm1-82591532 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 03:56:26,348 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 03:56:26,348 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-82591532, IP 192.168.1.3, Port 22 2025-11-20 03:56:26,414 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 03:56:26,495 - DEBUG - VM ctest-vn1_vm1-82591532 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 03:56:31,496 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 03:56:31,496 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-82591532, IP 192.168.1.3, Port 22 2025-11-20 03:56:31,662 - DEBUG - VM ctest-vn1_vm1-82591532 is ready for SSH connections 2025-11-20 03:56:31,662 - INFO - Waiting for VM ctest-vn1_vm2-58292070 to be up.. 2025-11-20 03:56:31,745 - DEBUG - VM is in ACTIVE state now 2025-11-20 03:56:31,745 - INFO - VM name : ctest-vn1_vm2-58292070 2025-11-20 03:56:31,825 - DEBUG - VM ctest-vn1_vm2-58292070 ID is e6a8c7e5-fc87-473b-b952-fc40f6321816 2025-11-20 03:56:31,825 - DEBUG - VM ctest-vn1_vm2-58292070 launched on Node an-jenkins-deploy-platform-ansible-os-4768-1 2025-11-20 03:56:31,905 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/e6a8c7e5-fc87-473b-b952-fc40f6321816 2025-11-20 03:56:31,915 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine-interface/e3ec460b-2fa8-42e8-8e74-fc87d3394e32 2025-11-20 03:56:33,081 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=5.84 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=1.60 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 1.603/3.720/5.837/2.117 ms') 2025-11-20 03:56:33,081 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm2-58292070 passed 2025-11-20 03:56:33,150 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 03:56:33,150 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-58292070, IP 192.168.1.4, Port 22 2025-11-20 03:56:33,318 - DEBUG - VM ctest-vn1_vm2-58292070 is ready for SSH connections 2025-11-20 03:56:33,318 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 03:56:33,318 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.48, gateway password: c0ntrail123 2025-11-20 03:56:33,318 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4 2025-11-20 03:56:54,983 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data. 64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=13.9 ms 64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.10 ms 64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.27 ms --- 192.168.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.101/5.436/13.938/6.012 ms 2025-11-20 03:56:54,983 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-82591532 passed 2025-11-20 03:56:54,983 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 03:56:54,983 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.131, gateway password: c0ntrail123 2025-11-20 03:56:54,984 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3 2025-11-20 03:57:37,907 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. 64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=6.74 ms 64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.15 ms 64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.37 ms --- 192.168.1.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.153/3.092/6.748/2.586 ms 2025-11-20 03:57:37,907 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-58292070 passed 2025-11-20 03:57:37,915 - INFO - Active control node from the Agent 10.0.0.48 is 10.0.0.23 2025-11-20 03:57:37,915 - INFO - Stoping the Control service in 10.0.0.23 2025-11-20 03:57:38,404 - INFO - 10.0.0.23 2025-11-20 03:57:38,405 - INFO - control:inactive 2025-11-20 03:57:38,405 - INFO - Contrail services control are down on nodes 10.0.0.23 2025-11-20 03:57:43,412 - INFO - Active control node from the Agent 10.0.0.48 is 10.0.0.50 2025-11-20 03:57:43,412 - INFO - Starting the Control service in 10.0.0.23 2025-11-20 03:57:43,804 - INFO - 10.0.0.23 2025-11-20 03:57:43,807 - INFO - control:initializing 2025-11-20 03:57:43,807 - DEBUG - defaultdict(, {'10.0.0.23': {'control': {'status': 'initializing', 'description': None}}}) 2025-11-20 03:57:43,807 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-11-20 03:57:48,905 - INFO - 10.0.0.23 2025-11-20 03:57:48,910 - INFO - control:active 2025-11-20 03:57:48,910 - INFO - Contrail services ['control'] are up on nodes 10.0.0.23 2025-11-20 03:57:54,010 - DEBUG - VM is in ACTIVE state now 2025-11-20 03:57:54,010 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 03:57:54,010 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/123b0729-3eb7-4874-a0b0-968ababe97e2 2025-11-20 03:57:54,019 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine-interface/9da5ba90-570d-4544-9d68-0cd08fcc0f84 2025-11-20 03:57:54,033 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/123b0729-3eb7-4874-a0b0-968ababe97e2 2025-11-20 03:57:54,042 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine-interface/9da5ba90-570d-4544-9d68-0cd08fcc0f84 2025-11-20 03:57:54,053 - DEBUG - Requesting: http://10.0.0.22:8082/instance-ip/52910a4d-8e7b-40e0-8d96-a6598f6d382f 2025-11-20 03:57:54,063 - DEBUG - Verifying in api server 10.0.0.22 2025-11-20 03:57:54,063 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/123b0729-3eb7-4874-a0b0-968ababe97e2 2025-11-20 03:57:54,071 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine-interface/9da5ba90-570d-4544-9d68-0cd08fcc0f84 2025-11-20 03:57:54,082 - DEBUG - Requesting: http://10.0.0.22:8082/instance-ip/52910a4d-8e7b-40e0-8d96-a6598f6d382f 2025-11-20 03:57:54,091 - INFO - VM ctest-vn1_vm1-82591532 verfication in all API Servers passed 2025-11-20 03:57:54,114 - DEBUG - VM ctest-vn1_vm1-82591532 Tap interface: {'index': '3', 'name': 'tap9da5ba90-57', 'uuid': '9da5ba90-570d-4544-9d68-0cd08fcc0f84', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927:ctest-vn1-56878927', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927', 'vm_uuid': '123b0729-3eb7-4874-a0b0-968ababe97e2', 'vm_name': 'ctest-vn1_vm1-82591532', 'ip_addr': '192.168.1.3', 'mac_addr': '02:9d:a5:ba:90:57', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '163', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-17624131:9da5ba90-570d-4544-9d68-0cd08fcc0f84', 'sg_uuid_list': ['db81be13-9b62-47a3-a92b-985308dc1af8'], 'static_route_list': None, 'vm_project_uuid': '3339bf5d-318a-4edf-9cb8-4a7fd32a3af6', 'admin_state': 'Enabled', 'flow_key_idx': '28', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-11-20 03:57:54,114 - DEBUG - Agent 10.0.0.48 vrf name: default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927:ctest-vn1-56878927 2025-11-20 03:57:54,120 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927:ctest-vn1-56878927', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927', 'table_label': '-1', 'vxlan_id': '15', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.25:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-11-20 03:57:54,207 - DEBUG - VM ctest-vn1_vm1-82591532 : Tap interface tap9da5ba90-57 is set to Active 2025-11-20 03:57:54,208 - DEBUG - Tap interface tap9da5ba90-57 detail : {'index': '3', 'name': 'tap9da5ba90-57', 'uuid': '9da5ba90-570d-4544-9d68-0cd08fcc0f84', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927:ctest-vn1-56878927', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927', 'vm_uuid': '123b0729-3eb7-4874-a0b0-968ababe97e2', 'vm_name': 'ctest-vn1_vm1-82591532', 'ip_addr': '192.168.1.3', 'mac_addr': '02:9d:a5:ba:90:57', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '163', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-17624131:9da5ba90-570d-4544-9d68-0cd08fcc0f84', 'sg_uuid_list': ['db81be13-9b62-47a3-a92b-985308dc1af8'], 'static_route_list': None, 'vm_project_uuid': '3339bf5d-318a-4edf-9cb8-4a7fd32a3af6', 'admin_state': 'Enabled', 'flow_key_idx': '28', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-11-20 03:57:54,326 - DEBUG - Starting Layer 2 verification in Agent 2025-11-20 03:57:54,338 - DEBUG - Layer 2 path is seen for VM MAC 02:9d:a5:ba:90:57 in agent 10.0.0.48 2025-11-20 03:57:54,338 - DEBUG - Active layer 2 route in agent is present for VMI tap9da5ba90-57 2025-11-20 03:57:54,338 - DEBUG - L2 label(29) matches bw route table and interface table 2025-11-20 03:57:54,344 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 is found in Agent of node 10.0.0.131 2025-11-20 03:57:54,468 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 is consistent in agent 10.0.0.131 2025-11-20 03:57:54,468 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.131 2025-11-20 03:57:54,468 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 verification for VM ctest-vn1_vm1-82591532 in Agent 10.0.0.131 passed 2025-11-20 03:57:54,582 - DEBUG - Starting all layer 2 verification in agent 10.0.0.131 2025-11-20 03:57:54,593 - DEBUG - Route for VM MAC 02:9d:a5:ba:90:57 is consistent in agent 10.0.0.131 2025-11-20 03:57:54,600 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 is found in Agent of node 10.0.0.48 2025-11-20 03:57:54,728 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 is consistent in agent 10.0.0.48 2025-11-20 03:57:54,728 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.48 2025-11-20 03:57:54,728 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 verification for VM ctest-vn1_vm1-82591532 in Agent 10.0.0.48 passed 2025-11-20 03:57:54,848 - DEBUG - Starting all layer 2 verification in agent 10.0.0.48 2025-11-20 03:57:54,860 - DEBUG - Route for VM MAC 02:9d:a5:ba:90:57 is consistent in agent 10.0.0.48 2025-11-20 03:57:54,861 - INFO - VM ctest-vn1_vm1-82591532 verifications in Compute nodes passed 2025-11-20 03:57:55,097 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.131 2025-11-20 03:57:55,101 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.25, 24; Got 10.20.0.25,24 2025-11-20 03:57:55,102 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.131 2025-11-20 03:57:55,221 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.48 2025-11-20 03:57:55,227 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-11-20 03:57:55,227 - INFO - Validated routes of VM ctest-vn1_vm1-82591532 in all vrouters 2025-11-20 03:57:55,751 - DEBUG - Starting all layer2 verification in 10.0.0.50 Control Node 2025-11-20 03:57:55,882 - DEBUG - Layer2 route found for VM MAC 02:9d:a5:ba:90:57 in Control-node 10.0.0.50 2025-11-20 03:57:55,883 - DEBUG - L2 Label for VM ctest-vn1_vm1-82591532 same between Control-node 10.0.0.50 and Agent, Expected: 29, Seen: 29 2025-11-20 03:57:55,883 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2025-11-20 03:57:56,016 - DEBUG - Layer2 route found for VM MAC 02:9d:a5:ba:90:57 in Control-node 10.0.0.23 2025-11-20 03:57:56,016 - DEBUG - L2 Label for VM ctest-vn1_vm1-82591532 same between Control-node 10.0.0.23 and Agent, Expected: 29, Seen: 29 2025-11-20 03:57:56,016 - DEBUG - Starting all layer2 verification in 10.0.0.22 Control Node 2025-11-20 03:57:56,142 - DEBUG - Layer2 route found for VM MAC 02:9d:a5:ba:90:57 in Control-node 10.0.0.22 2025-11-20 03:57:56,142 - DEBUG - L2 Label for VM ctest-vn1_vm1-82591532 same between Control-node 10.0.0.22 and Agent, Expected: 29, Seen: 29 2025-11-20 03:57:56,142 - INFO - Verification in Control-nodes for VM ctest-vn1_vm1-82591532 passed 2025-11-20 03:57:56,142 - DEBUG - Verifying the vm in opserver 2025-11-20 03:57:56,142 - DEBUG - Verifying in collector 10.0.0.22 ... 2025-11-20 03:57:56,142 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machine/123b0729-3eb7-4874-a0b0-968ababe97e2?flat 2025-11-20 03:57:56,160 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-17624131:9da5ba90-570d-4544-9d68-0cd08fcc0f84?flat 2025-11-20 03:57:56,179 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-17624131:9da5ba90-570d-4544-9d68-0cd08fcc0f84?flat 2025-11-20 03:57:56,258 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machine/123b0729-3eb7-4874-a0b0-968ababe97e2?flat 2025-11-20 03:57:56,327 - DEBUG - Verifying vm in vn uve 2025-11-20 03:57:56,327 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-17624131:9da5ba90-570d-4544-9d68-0cd08fcc0f84?flat 2025-11-20 03:57:56,391 - DEBUG - VM uve shows interface as {'l2_active': True, 'ip6_active': False, 'mac_address': '02:9d:a5:ba:90:57', 'vm_name': 'ctest-vn1_vm1-82591532', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927', 'vhostuser_mode': 0, 'label': 24, 'rx_vlan': 65535, 'uuid': '9da5ba90-570d-4544-9d68-0cd08fcc0f84', 'ip6_address': '::', 'is_health_check_active': True, 'ip_address': '192.168.1.3', 'admin_state': True, 'port_mirror_enabled': False, 'tx_vlan': 65535, '__T': 1763611062169177, 'active': True, 'vm_uuid': '123b0729-3eb7-4874-a0b0-968ababe97e2', 'ip4_active': True, 'vn_uuid': 'c6a6d9b7-04be-4849-a44b-71d9d93fc5ff', 'gateway': '192.168.1.1', 'fixed_ip4_list': ['192.168.1.3']} 2025-11-20 03:57:56,391 - DEBUG - VM uve shows ip address as ['192.168.1.3', '::'] 2025-11-20 03:57:56,391 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 2025-11-20 03:57:56,391 - INFO - Verifying through opserver in 10.0.0.23 2025-11-20 03:57:56,391 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927?flat 2025-11-20 03:57:56,417 - DEBUG - expected vm list 123b0729-3eb7-4874-a0b0-968ababe97e2 2025-11-20 03:57:56,417 - DEBUG - Extracted vm list ['123b0729-3eb7-4874-a0b0-968ababe97e2'] 2025-11-20 03:57:56,417 - INFO - VM 123b0729-3eb7-4874-a0b0-968ababe97e2 is present in default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 2025-11-20 03:57:56,417 - DEBUG - Verifying vm in vrouter uve 2025-11-20 03:57:56,417 - DEBUG - Getting info from collector 10.0.0.22.. 2025-11-20 03:57:56,417 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machine/123b0729-3eb7-4874-a0b0-968ababe97e2?flat 2025-11-20 03:57:56,484 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4768-2'] 2025-11-20 03:57:56,484 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-2:Compute:contrail-vrouter-agent:0?flat 2025-11-20 03:57:56,548 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-2?flat 2025-11-20 03:57:56,583 - DEBUG - VM 123b0729-3eb7-4874-a0b0-968ababe97e2 is present in vrouter an-jenkins-deploy-platform-ansible-os-4768-2 uve 2025-11-20 03:57:56,584 - DEBUG - Validated that VM ctest-vn1_vm1-82591532 is in Vrouter an-jenkins-deploy-platform-ansible-os-4768-2 UVE 2025-11-20 03:57:56,584 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-11-20 03:57:56,593 - DEBUG - Expected tap interface of VM uuid 123b0729-3eb7-4874-a0b0-968ababe97e2 is default-domain:ctest-TestBasicVMVN0-17624131:9da5ba90-570d-4544-9d68-0cd08fcc0f84 2025-11-20 03:57:56,593 - DEBUG - Expected VN of VM uuid 123b0729-3eb7-4874-a0b0-968ababe97e2 is default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 2025-11-20 03:57:56,593 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-2:Compute:contrail-vrouter-agent:0?flat 2025-11-20 03:57:56,654 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-2?flat 2025-11-20 03:57:56,692 - DEBUG - VM 123b0729-3eb7-4874-a0b0-968ababe97e2 is present in vrouter an-jenkins-deploy-platform-ansible-os-4768-2 uve 2025-11-20 03:57:56,692 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-17624131:9da5ba90-570d-4544-9d68-0cd08fcc0f84 of vm 123b0729-3eb7-4874-a0b0-968ababe97e2 is present in vrouter an-jenkins-deploy-platform-ansible-os-4768-2 uve 2025-11-20 03:57:56,692 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 of vm 123b0729-3eb7-4874-a0b0-968ababe97e2 is present in vrouter an-jenkins-deploy-platform-ansible-os-4768-2 uve 2025-11-20 03:57:56,693 - DEBUG - Verifying the 123b0729-3eb7-4874-a0b0-968ababe97e2 virtual network link through opserver 10.0.0.23 2025-11-20 03:57:56,693 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-11-20 03:57:56,706 - DEBUG - VM link and name as {'name': '123b0729-3eb7-4874-a0b0-968ababe97e2', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/123b0729-3eb7-4874-a0b0-968ababe97e2?flat'} 2025-11-20 03:57:56,706 - DEBUG - Verifying the 123b0729-3eb7-4874-a0b0-968ababe97e2 virtual network link through opserver 10.0.0.50 2025-11-20 03:57:56,706 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2025-11-20 03:57:56,718 - DEBUG - VM link and name as {'name': '123b0729-3eb7-4874-a0b0-968ababe97e2', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-machine/123b0729-3eb7-4874-a0b0-968ababe97e2?flat'} 2025-11-20 03:57:56,718 - DEBUG - Verifying the 123b0729-3eb7-4874-a0b0-968ababe97e2 virtual network link through opserver 10.0.0.22 2025-11-20 03:57:56,718 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines 2025-11-20 03:57:56,730 - DEBUG - VM link and name as {'name': '123b0729-3eb7-4874-a0b0-968ababe97e2', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-machine/123b0729-3eb7-4874-a0b0-968ababe97e2?flat'} 2025-11-20 03:57:56,730 - INFO - VM ctest-vn1_vm1-82591532 validations in Opserver passed 2025-11-20 03:57:56,812 - DEBUG - VM is in ACTIVE state now 2025-11-20 03:57:56,812 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 03:57:56,812 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/e6a8c7e5-fc87-473b-b952-fc40f6321816 2025-11-20 03:57:56,821 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine-interface/e3ec460b-2fa8-42e8-8e74-fc87d3394e32 2025-11-20 03:57:56,832 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/e6a8c7e5-fc87-473b-b952-fc40f6321816 2025-11-20 03:57:56,840 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine-interface/e3ec460b-2fa8-42e8-8e74-fc87d3394e32 2025-11-20 03:57:56,852 - DEBUG - Requesting: http://10.0.0.22:8082/instance-ip/24dc5598-abe5-4d03-bbeb-0c1d44c60466 2025-11-20 03:57:56,862 - DEBUG - Verifying in api server 10.0.0.22 2025-11-20 03:57:56,862 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/e6a8c7e5-fc87-473b-b952-fc40f6321816 2025-11-20 03:57:56,870 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine-interface/e3ec460b-2fa8-42e8-8e74-fc87d3394e32 2025-11-20 03:57:56,880 - DEBUG - Requesting: http://10.0.0.22:8082/instance-ip/24dc5598-abe5-4d03-bbeb-0c1d44c60466 2025-11-20 03:57:56,888 - INFO - VM ctest-vn1_vm2-58292070 verfication in all API Servers passed 2025-11-20 03:57:56,907 - DEBUG - VM ctest-vn1_vm2-58292070 Tap interface: {'index': '3', 'name': 'tape3ec460b-2f', 'uuid': 'e3ec460b-2fa8-42e8-8e74-fc87d3394e32', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927:ctest-vn1-56878927', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927', 'vm_uuid': 'e6a8c7e5-fc87-473b-b952-fc40f6321816', 'vm_name': 'ctest-vn1_vm2-58292070', 'ip_addr': '192.168.1.4', 'mac_addr': '02:e3:ec:46:0b:2f', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '125', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-17624131:e3ec460b-2fa8-42e8-8e74-fc87d3394e32', 'sg_uuid_list': ['db81be13-9b62-47a3-a92b-985308dc1af8'], 'static_route_list': None, 'vm_project_uuid': '3339bf5d-318a-4edf-9cb8-4a7fd32a3af6', 'admin_state': 'Enabled', 'flow_key_idx': '26', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-11-20 03:57:56,907 - DEBUG - Agent 10.0.0.131 vrf name: default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927:ctest-vn1-56878927 2025-11-20 03:57:56,915 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927:ctest-vn1-56878927', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927', 'table_label': '-1', 'vxlan_id': '15', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.242:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-11-20 03:57:56,998 - DEBUG - VM ctest-vn1_vm2-58292070 : Tap interface tape3ec460b-2f is set to Active 2025-11-20 03:57:56,999 - DEBUG - Tap interface tape3ec460b-2f detail : {'index': '3', 'name': 'tape3ec460b-2f', 'uuid': 'e3ec460b-2fa8-42e8-8e74-fc87d3394e32', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927:ctest-vn1-56878927', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927', 'vm_uuid': 'e6a8c7e5-fc87-473b-b952-fc40f6321816', 'vm_name': 'ctest-vn1_vm2-58292070', 'ip_addr': '192.168.1.4', 'mac_addr': '02:e3:ec:46:0b:2f', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '125', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-17624131:e3ec460b-2fa8-42e8-8e74-fc87d3394e32', 'sg_uuid_list': ['db81be13-9b62-47a3-a92b-985308dc1af8'], 'static_route_list': None, 'vm_project_uuid': '3339bf5d-318a-4edf-9cb8-4a7fd32a3af6', 'admin_state': 'Enabled', 'flow_key_idx': '26', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-11-20 03:57:57,118 - DEBUG - Starting Layer 2 verification in Agent 2025-11-20 03:57:57,128 - DEBUG - Layer 2 path is seen for VM MAC 02:e3:ec:46:0b:2f in agent 10.0.0.131 2025-11-20 03:57:57,129 - DEBUG - Active layer 2 route in agent is present for VMI tape3ec460b-2f 2025-11-20 03:57:57,129 - DEBUG - L2 label(29) matches bw route table and interface table 2025-11-20 03:57:57,135 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 is found in Agent of node 10.0.0.131 2025-11-20 03:57:57,264 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 is consistent in agent 10.0.0.131 2025-11-20 03:57:57,264 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.131 2025-11-20 03:57:57,264 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 verification for VM ctest-vn1_vm2-58292070 in Agent 10.0.0.131 passed 2025-11-20 03:57:57,373 - DEBUG - Starting all layer 2 verification in agent 10.0.0.131 2025-11-20 03:57:57,384 - DEBUG - Route for VM MAC 02:e3:ec:46:0b:2f is consistent in agent 10.0.0.131 2025-11-20 03:57:57,391 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 is found in Agent of node 10.0.0.48 2025-11-20 03:57:57,525 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 is consistent in agent 10.0.0.48 2025-11-20 03:57:57,526 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.48 2025-11-20 03:57:57,526 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 verification for VM ctest-vn1_vm2-58292070 in Agent 10.0.0.48 passed 2025-11-20 03:57:57,642 - DEBUG - Starting all layer 2 verification in agent 10.0.0.48 2025-11-20 03:57:57,653 - DEBUG - Route for VM MAC 02:e3:ec:46:0b:2f is consistent in agent 10.0.0.48 2025-11-20 03:57:57,653 - INFO - VM ctest-vn1_vm2-58292070 verifications in Compute nodes passed 2025-11-20 03:57:57,883 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.131 2025-11-20 03:57:57,887 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-11-20 03:57:57,988 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.48 2025-11-20 03:57:57,992 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.242, 25; Got 10.20.0.242,25 2025-11-20 03:57:57,992 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.48 2025-11-20 03:57:57,992 - INFO - Validated routes of VM ctest-vn1_vm2-58292070 in all vrouters 2025-11-20 03:57:58,490 - DEBUG - Starting all layer2 verification in 10.0.0.50 Control Node 2025-11-20 03:57:58,616 - DEBUG - Layer2 route found for VM MAC 02:e3:ec:46:0b:2f in Control-node 10.0.0.50 2025-11-20 03:57:58,616 - DEBUG - L2 Label for VM ctest-vn1_vm2-58292070 same between Control-node 10.0.0.50 and Agent, Expected: 29, Seen: 29 2025-11-20 03:57:58,616 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2025-11-20 03:57:58,744 - DEBUG - Layer2 route found for VM MAC 02:e3:ec:46:0b:2f in Control-node 10.0.0.23 2025-11-20 03:57:58,744 - DEBUG - L2 Label for VM ctest-vn1_vm2-58292070 same between Control-node 10.0.0.23 and Agent, Expected: 29, Seen: 29 2025-11-20 03:57:58,744 - DEBUG - Starting all layer2 verification in 10.0.0.22 Control Node 2025-11-20 03:57:58,876 - DEBUG - Layer2 route found for VM MAC 02:e3:ec:46:0b:2f in Control-node 10.0.0.22 2025-11-20 03:57:58,876 - DEBUG - L2 Label for VM ctest-vn1_vm2-58292070 same between Control-node 10.0.0.22 and Agent, Expected: 29, Seen: 29 2025-11-20 03:57:58,876 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-58292070 passed 2025-11-20 03:57:58,876 - DEBUG - Verifying the vm in opserver 2025-11-20 03:57:58,876 - DEBUG - Verifying in collector 10.0.0.22 ... 2025-11-20 03:57:58,876 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machine/e6a8c7e5-fc87-473b-b952-fc40f6321816?flat 2025-11-20 03:57:58,894 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-17624131:e3ec460b-2fa8-42e8-8e74-fc87d3394e32?flat 2025-11-20 03:57:58,913 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-17624131:e3ec460b-2fa8-42e8-8e74-fc87d3394e32?flat 2025-11-20 03:57:58,934 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machine/e6a8c7e5-fc87-473b-b952-fc40f6321816?flat 2025-11-20 03:57:58,951 - DEBUG - Verifying vm in vn uve 2025-11-20 03:57:58,951 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-17624131:e3ec460b-2fa8-42e8-8e74-fc87d3394e32?flat 2025-11-20 03:57:58,979 - DEBUG - VM uve shows interface as {'l2_active': True, 'ip6_active': False, 'mac_address': '02:e3:ec:46:0b:2f', 'vm_name': 'ctest-vn1_vm2-58292070', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927', 'vhostuser_mode': 0, 'label': 25, 'rx_vlan': 65535, 'uuid': 'e3ec460b-2fa8-42e8-8e74-fc87d3394e32', 'ip6_address': '::', 'is_health_check_active': True, 'ip_address': '192.168.1.4', 'admin_state': True, 'port_mirror_enabled': False, 'tx_vlan': 65535, '__T': 1763610933693839, 'active': True, 'vm_uuid': 'e6a8c7e5-fc87-473b-b952-fc40f6321816', 'ip4_active': True, 'vn_uuid': 'c6a6d9b7-04be-4849-a44b-71d9d93fc5ff', 'gateway': '192.168.1.1', 'fixed_ip4_list': ['192.168.1.4']} 2025-11-20 03:57:58,979 - DEBUG - VM uve shows ip address as ['192.168.1.4', '::'] 2025-11-20 03:57:58,979 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 2025-11-20 03:57:58,979 - INFO - Verifying through opserver in 10.0.0.23 2025-11-20 03:57:58,979 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927?flat 2025-11-20 03:57:59,007 - DEBUG - expected vm list e6a8c7e5-fc87-473b-b952-fc40f6321816 2025-11-20 03:57:59,007 - DEBUG - Extracted vm list ['e6a8c7e5-fc87-473b-b952-fc40f6321816'] 2025-11-20 03:57:59,007 - INFO - VM e6a8c7e5-fc87-473b-b952-fc40f6321816 is present in default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 2025-11-20 03:57:59,007 - DEBUG - Verifying vm in vrouter uve 2025-11-20 03:57:59,007 - DEBUG - Getting info from collector 10.0.0.22.. 2025-11-20 03:57:59,007 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machine/e6a8c7e5-fc87-473b-b952-fc40f6321816?flat 2025-11-20 03:57:59,026 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4768-1'] 2025-11-20 03:57:59,026 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-1:Compute:contrail-vrouter-agent:0?flat 2025-11-20 03:57:59,086 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-1?flat 2025-11-20 03:57:59,120 - DEBUG - VM e6a8c7e5-fc87-473b-b952-fc40f6321816 is present in vrouter an-jenkins-deploy-platform-ansible-os-4768-1 uve 2025-11-20 03:57:59,121 - DEBUG - Validated that VM ctest-vn1_vm2-58292070 is in Vrouter an-jenkins-deploy-platform-ansible-os-4768-1 UVE 2025-11-20 03:57:59,121 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-11-20 03:57:59,134 - DEBUG - Expected tap interface of VM uuid e6a8c7e5-fc87-473b-b952-fc40f6321816 is default-domain:ctest-TestBasicVMVN0-17624131:e3ec460b-2fa8-42e8-8e74-fc87d3394e32 2025-11-20 03:57:59,134 - DEBUG - Expected VN of VM uuid e6a8c7e5-fc87-473b-b952-fc40f6321816 is default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 2025-11-20 03:57:59,134 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-1:Compute:contrail-vrouter-agent:0?flat 2025-11-20 03:57:59,190 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-1?flat 2025-11-20 03:57:59,222 - DEBUG - VM e6a8c7e5-fc87-473b-b952-fc40f6321816 is present in vrouter an-jenkins-deploy-platform-ansible-os-4768-1 uve 2025-11-20 03:57:59,222 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-17624131:e3ec460b-2fa8-42e8-8e74-fc87d3394e32 of vm e6a8c7e5-fc87-473b-b952-fc40f6321816 is present in vrouter an-jenkins-deploy-platform-ansible-os-4768-1 uve 2025-11-20 03:57:59,222 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 of vm e6a8c7e5-fc87-473b-b952-fc40f6321816 is present in vrouter an-jenkins-deploy-platform-ansible-os-4768-1 uve 2025-11-20 03:57:59,222 - DEBUG - Verifying the e6a8c7e5-fc87-473b-b952-fc40f6321816 virtual network link through opserver 10.0.0.23 2025-11-20 03:57:59,223 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-11-20 03:57:59,239 - DEBUG - VM link and name as {'name': 'e6a8c7e5-fc87-473b-b952-fc40f6321816', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/e6a8c7e5-fc87-473b-b952-fc40f6321816?flat'} 2025-11-20 03:57:59,239 - DEBUG - Verifying the e6a8c7e5-fc87-473b-b952-fc40f6321816 virtual network link through opserver 10.0.0.50 2025-11-20 03:57:59,240 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2025-11-20 03:57:59,304 - DEBUG - VM link and name as {'name': 'e6a8c7e5-fc87-473b-b952-fc40f6321816', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-machine/e6a8c7e5-fc87-473b-b952-fc40f6321816?flat'} 2025-11-20 03:57:59,304 - DEBUG - Verifying the e6a8c7e5-fc87-473b-b952-fc40f6321816 virtual network link through opserver 10.0.0.22 2025-11-20 03:57:59,304 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines 2025-11-20 03:57:59,317 - DEBUG - VM link and name as {'name': 'e6a8c7e5-fc87-473b-b952-fc40f6321816', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-machine/e6a8c7e5-fc87-473b-b952-fc40f6321816?flat'} 2025-11-20 03:57:59,317 - INFO - VM ctest-vn1_vm2-58292070 validations in Opserver passed 2025-11-20 03:57:59,317 - INFO - Checking the ping between the VM again 2025-11-20 03:57:59,317 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 03:57:59,317 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.48, gateway password: c0ntrail123 2025-11-20 03:57:59,317 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4 2025-11-20 03:58:01,815 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data. 64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=4.95 ms 64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.14 ms 64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.51 ms --- 192.168.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.145/2.538/4.954/1.715 ms 2025-11-20 03:58:01,815 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-82591532 passed 2025-11-20 03:58:01,816 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 03:58:01,816 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.131, gateway password: c0ntrail123 2025-11-20 03:58:01,816 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3 2025-11-20 03:58:04,363 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. 64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=2.87 ms 64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.10 ms 64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.11 ms --- 192.168.1.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.103/1.698/2.873/0.830 ms 2025-11-20 03:58:04,364 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-58292070 passed 2025-11-20 03:58:04,364 - INFO - Deleting VM ctest-vn1_vm2-58292070 2025-11-20 03:58:04,436 - DEBUG - Verifying in api server 10.0.0.22 2025-11-20 03:58:04,436 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/e6a8c7e5-fc87-473b-b952-fc40f6321816 2025-11-20 03:58:04,451 - DEBUG - VM ID e6a8c7e5-fc87-473b-b952-fc40f6321816 of VM ctest-vn1_vm2-58292070 is still found in API Server 2025-11-20 03:58:06,451 - DEBUG - Verifying in api server 10.0.0.22 2025-11-20 03:58:06,452 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/e6a8c7e5-fc87-473b-b952-fc40f6321816 2025-11-20 03:58:06,458 - DEBUG - Response Code: 404 2025-11-20 03:58:06,458 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/e6a8c7e5-fc87-473b-b952-fc40f6321816 2025-11-20 03:58:06,465 - DEBUG - Response Code: 404 2025-11-20 03:58:06,465 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/e6a8c7e5-fc87-473b-b952-fc40f6321816 2025-11-20 03:58:06,470 - DEBUG - Response Code: 404 2025-11-20 03:58:06,470 - INFO - VM ctest-vn1_vm2-58292070 is fully removed in API-Server 2025-11-20 03:58:06,624 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-11-20 03:58:06,658 - DEBUG - Validated that vrouter 10.0.0.131 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 2025-11-20 03:58:06,771 - DEBUG - VM route 192.168.1.4 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 of compute 10.0.0.48 2025-11-20 03:58:06,776 - DEBUG - Validated that vrouter 10.0.0.48 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 2025-11-20 03:58:06,776 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 2025-11-20 03:58:06,776 - INFO - VM ctest-vn1_vm2-58292070 is removed in Compute, and routes are removed in all compute nodes 2025-11-20 03:58:07,020 - INFO - Routes for VM ctest-vn1_vm2-58292070 is removed in all control-nodes 2025-11-20 03:58:07,020 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-1:Compute:contrail-vrouter-agent:0?flat 2025-11-20 03:58:07,066 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-1?flat 2025-11-20 03:58:07,096 - DEBUG - VM e6a8c7e5-fc87-473b-b952-fc40f6321816 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4768-1 uve 2025-11-20 03:58:07,096 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-11-20 03:58:07,096 - DEBUG - interface for vm e6a8c7e5-fc87-473b-b952-fc40f6321816 deleted from vrouter uve ... 2025-11-20 03:58:07,096 - DEBUG - Deleted interface not in error interface list ... 2025-11-20 03:58:10,097 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-1:Compute:contrail-vrouter-agent:0?flat 2025-11-20 03:58:10,150 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-1?flat 2025-11-20 03:58:10,178 - DEBUG - VM e6a8c7e5-fc87-473b-b952-fc40f6321816 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4768-1 uve 2025-11-20 03:58:10,178 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-11-20 03:58:10,178 - DEBUG - interface for vm e6a8c7e5-fc87-473b-b952-fc40f6321816 deleted from vrouter uve ... 2025-11-20 03:58:10,178 - DEBUG - Deleted interface not in error interface list ... 2025-11-20 03:58:13,179 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-1:Compute:contrail-vrouter-agent:0?flat 2025-11-20 03:58:13,231 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-1?flat 2025-11-20 03:58:13,261 - DEBUG - VM e6a8c7e5-fc87-473b-b952-fc40f6321816 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4768-1 uve 2025-11-20 03:58:13,261 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-11-20 03:58:13,261 - DEBUG - interface for vm e6a8c7e5-fc87-473b-b952-fc40f6321816 deleted from vrouter uve ... 2025-11-20 03:58:13,261 - DEBUG - Deleted interface not in error interface list ... 2025-11-20 03:58:16,262 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-1:Compute:contrail-vrouter-agent:0?flat 2025-11-20 03:58:16,308 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-1?flat 2025-11-20 03:58:16,342 - DEBUG - VM e6a8c7e5-fc87-473b-b952-fc40f6321816 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4768-1 uve 2025-11-20 03:58:16,342 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-11-20 03:58:16,342 - DEBUG - interface for vm e6a8c7e5-fc87-473b-b952-fc40f6321816 deleted from vrouter uve ... 2025-11-20 03:58:16,342 - DEBUG - Deleted interface not in error interface list ... 2025-11-20 03:58:19,343 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-1:Compute:contrail-vrouter-agent:0?flat 2025-11-20 03:58:19,389 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-1?flat 2025-11-20 03:58:19,417 - DEBUG - VM e6a8c7e5-fc87-473b-b952-fc40f6321816 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4768-1 uve 2025-11-20 03:58:19,418 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-11-20 03:58:19,418 - DEBUG - interface for vm e6a8c7e5-fc87-473b-b952-fc40f6321816 deleted from vrouter uve ... 2025-11-20 03:58:19,418 - DEBUG - Deleted interface not in error interface list ... 2025-11-20 03:58:22,419 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-1:Compute:contrail-vrouter-agent:0?flat 2025-11-20 03:58:22,463 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-1?flat 2025-11-20 03:58:22,493 - DEBUG - VM e6a8c7e5-fc87-473b-b952-fc40f6321816 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4768-1 uve 2025-11-20 03:58:22,493 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-11-20 03:58:22,493 - DEBUG - interface for vm e6a8c7e5-fc87-473b-b952-fc40f6321816 deleted from vrouter uve ... 2025-11-20 03:58:22,493 - DEBUG - Deleted interface not in error interface list ... 2025-11-20 03:58:25,494 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-1:Compute:contrail-vrouter-agent:0?flat 2025-11-20 03:58:25,542 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-1?flat 2025-11-20 03:58:25,570 - DEBUG - VM e6a8c7e5-fc87-473b-b952-fc40f6321816 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4768-1 uve 2025-11-20 03:58:25,570 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-11-20 03:58:25,570 - DEBUG - interface for vm e6a8c7e5-fc87-473b-b952-fc40f6321816 deleted from vrouter uve ... 2025-11-20 03:58:25,570 - DEBUG - Deleted interface not in error interface list ... 2025-11-20 03:58:28,571 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-1:Compute:contrail-vrouter-agent:0?flat 2025-11-20 03:58:28,616 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-1?flat 2025-11-20 03:58:28,645 - DEBUG - VM e6a8c7e5-fc87-473b-b952-fc40f6321816 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4768-1 uve 2025-11-20 03:58:28,645 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-11-20 03:58:28,645 - DEBUG - interface for vm e6a8c7e5-fc87-473b-b952-fc40f6321816 deleted from vrouter uve ... 2025-11-20 03:58:28,645 - DEBUG - Deleted interface not in error interface list ... 2025-11-20 03:58:31,646 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-1:Compute:contrail-vrouter-agent:0?flat 2025-11-20 03:58:31,689 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-1?flat 2025-11-20 03:58:31,717 - DEBUG - VM e6a8c7e5-fc87-473b-b952-fc40f6321816 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4768-1 uve 2025-11-20 03:58:31,717 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-11-20 03:58:31,717 - DEBUG - interface for vm e6a8c7e5-fc87-473b-b952-fc40f6321816 deleted from vrouter uve ... 2025-11-20 03:58:31,717 - DEBUG - Deleted interface not in error interface list ... 2025-11-20 03:58:34,718 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-1:Compute:contrail-vrouter-agent:0?flat 2025-11-20 03:58:34,761 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-1?flat 2025-11-20 03:58:34,800 - DEBUG - VM e6a8c7e5-fc87-473b-b952-fc40f6321816 is not present in vrouter an-jenkins-deploy-platform-ansible-os-4768-1 uve 2025-11-20 03:58:34,800 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-11-20 03:58:34,800 - DEBUG - interface for vm e6a8c7e5-fc87-473b-b952-fc40f6321816 deleted from vrouter uve ... 2025-11-20 03:58:34,800 - DEBUG - Deleted interface not in error interface list ... 2025-11-20 03:58:34,800 - DEBUG - Validated that VM e6a8c7e5-fc87-473b-b952-fc40f6321816 is removed in Vrouter UVE 2025-11-20 03:58:34,800 - DEBUG - Verifying through opserver in 10.0.0.23 2025-11-20 03:58:34,800 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927?flat 2025-11-20 03:58:34,825 - DEBUG - e6a8c7e5-fc87-473b-b952-fc40f6321816 BM not in default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 uve 2025-11-20 03:58:34,825 - DEBUG - Validated that VM e6a8c7e5-fc87-473b-b952-fc40f6321816 is not present in the VN UVE 2025-11-20 03:58:34,825 - DEBUG - Verifying the e6a8c7e5-fc87-473b-b952-fc40f6321816 virtual network link through opserver 10.0.0.23 2025-11-20 03:58:34,825 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-11-20 03:58:34,845 - INFO - e6a8c7e5-fc87-473b-b952-fc40f6321816 vm uve deleted from opserver 2025-11-20 03:58:34,845 - DEBUG - Verifying the e6a8c7e5-fc87-473b-b952-fc40f6321816 virtual network link through opserver 10.0.0.50 2025-11-20 03:58:34,845 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2025-11-20 03:58:34,923 - INFO - e6a8c7e5-fc87-473b-b952-fc40f6321816 vm uve deleted from opserver 2025-11-20 03:58:34,923 - DEBUG - Verifying the e6a8c7e5-fc87-473b-b952-fc40f6321816 virtual network link through opserver 10.0.0.22 2025-11-20 03:58:34,923 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines 2025-11-20 03:58:34,936 - INFO - e6a8c7e5-fc87-473b-b952-fc40f6321816 vm uve deleted from opserver 2025-11-20 03:58:34,936 - INFO - Validated that VM e6a8c7e5-fc87-473b-b952-fc40f6321816 is removed from Opserver 2025-11-20 03:58:34,936 - INFO - Deleting VM ctest-vn1_vm1-82591532 2025-11-20 03:58:35,013 - DEBUG - Verifying in api server 10.0.0.22 2025-11-20 03:58:35,013 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/123b0729-3eb7-4874-a0b0-968ababe97e2 2025-11-20 03:58:35,021 - DEBUG - VM ID 123b0729-3eb7-4874-a0b0-968ababe97e2 of VM ctest-vn1_vm1-82591532 is still found in API Server 2025-11-20 03:58:37,022 - DEBUG - Verifying in api server 10.0.0.22 2025-11-20 03:58:37,022 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/123b0729-3eb7-4874-a0b0-968ababe97e2 2025-11-20 03:58:37,029 - DEBUG - Response Code: 404 2025-11-20 03:58:37,029 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/123b0729-3eb7-4874-a0b0-968ababe97e2 2025-11-20 03:58:37,035 - DEBUG - Response Code: 404 2025-11-20 03:58:37,035 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/123b0729-3eb7-4874-a0b0-968ababe97e2 2025-11-20 03:58:37,041 - DEBUG - Response Code: 404 2025-11-20 03:58:37,041 - INFO - VM ctest-vn1_vm1-82591532 is fully removed in API-Server 2025-11-20 03:58:37,196 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-11-20 03:58:37,240 - DEBUG - Validated that vrouter 10.0.0.131 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 2025-11-20 03:58:37,251 - DEBUG - Validated that vrouter 10.0.0.48 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 2025-11-20 03:58:37,251 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 2025-11-20 03:58:37,251 - INFO - VM ctest-vn1_vm1-82591532 is removed in Compute, and routes are removed in all compute nodes 2025-11-20 03:58:37,485 - INFO - Routes for VM ctest-vn1_vm1-82591532 is removed in all control-nodes 2025-11-20 03:58:37,485 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-2:Compute:contrail-vrouter-agent:0?flat 2025-11-20 03:58:37,538 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-2?flat 2025-11-20 03:58:37,567 - DEBUG - VM 123b0729-3eb7-4874-a0b0-968ababe97e2 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4768-2 uve 2025-11-20 03:58:37,567 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-11-20 03:58:37,567 - DEBUG - interface for vm 123b0729-3eb7-4874-a0b0-968ababe97e2 deleted from vrouter uve ... 2025-11-20 03:58:37,567 - DEBUG - Deleted interface not in error interface list ... 2025-11-20 03:58:40,567 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-2:Compute:contrail-vrouter-agent:0?flat 2025-11-20 03:58:40,614 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-2?flat 2025-11-20 03:58:40,640 - DEBUG - VM 123b0729-3eb7-4874-a0b0-968ababe97e2 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4768-2 uve 2025-11-20 03:58:40,640 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-11-20 03:58:40,640 - DEBUG - interface for vm 123b0729-3eb7-4874-a0b0-968ababe97e2 deleted from vrouter uve ... 2025-11-20 03:58:40,640 - DEBUG - Deleted interface not in error interface list ... 2025-11-20 03:58:43,641 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-2:Compute:contrail-vrouter-agent:0?flat 2025-11-20 03:58:43,682 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-2?flat 2025-11-20 03:58:43,710 - DEBUG - VM 123b0729-3eb7-4874-a0b0-968ababe97e2 is not present in vrouter an-jenkins-deploy-platform-ansible-os-4768-2 uve 2025-11-20 03:58:43,710 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-11-20 03:58:43,710 - DEBUG - interface for vm 123b0729-3eb7-4874-a0b0-968ababe97e2 deleted from vrouter uve ... 2025-11-20 03:58:43,710 - DEBUG - Deleted interface not in error interface list ... 2025-11-20 03:58:43,710 - DEBUG - Validated that VM 123b0729-3eb7-4874-a0b0-968ababe97e2 is removed in Vrouter UVE 2025-11-20 03:58:43,710 - DEBUG - Verifying through opserver in 10.0.0.23 2025-11-20 03:58:43,710 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927?flat 2025-11-20 03:58:43,729 - DEBUG - 123b0729-3eb7-4874-a0b0-968ababe97e2 BM not in default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-56878927 uve 2025-11-20 03:58:43,729 - DEBUG - Validated that VM 123b0729-3eb7-4874-a0b0-968ababe97e2 is not present in the VN UVE 2025-11-20 03:58:43,729 - DEBUG - Verifying the 123b0729-3eb7-4874-a0b0-968ababe97e2 virtual network link through opserver 10.0.0.23 2025-11-20 03:58:43,729 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-11-20 03:58:43,741 - INFO - 123b0729-3eb7-4874-a0b0-968ababe97e2 vm uve deleted from opserver 2025-11-20 03:58:43,741 - DEBUG - Verifying the 123b0729-3eb7-4874-a0b0-968ababe97e2 virtual network link through opserver 10.0.0.50 2025-11-20 03:58:43,741 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2025-11-20 03:58:43,753 - INFO - 123b0729-3eb7-4874-a0b0-968ababe97e2 vm uve deleted from opserver 2025-11-20 03:58:43,753 - DEBUG - Verifying the 123b0729-3eb7-4874-a0b0-968ababe97e2 virtual network link through opserver 10.0.0.22 2025-11-20 03:58:43,753 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines 2025-11-20 03:58:43,765 - INFO - 123b0729-3eb7-4874-a0b0-968ababe97e2 vm uve deleted from opserver 2025-11-20 03:58:43,765 - INFO - Validated that VM 123b0729-3eb7-4874-a0b0-968ababe97e2 is removed from Opserver 2025-11-20 03:58:43,765 - INFO - Deleting VN ctest-vn1-56878927 2025-11-20 03:58:43,942 - DEBUG - Response for deleting network () 2025-11-20 03:58:43,943 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/cce25481-1121-4642-a92e-da98cda2e575 2025-11-20 03:58:43,950 - DEBUG - Response Code: 404 2025-11-20 03:58:43,951 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-11-20 03:58:43,964 - DEBUG - Requesting: http://10.0.0.23:8082/domain/785d7fdc-4b61-45a1-9add-76a6d523389b 2025-11-20 03:58:44,024 - DEBUG - Requesting: http://10.0.0.23:8082/project/3339bf5d-318a-4edf-9cb8-4a7fd32a3af6 2025-11-20 03:58:44,119 - INFO - Validated that VN ctest-vn1-56878927 is not found in API Server 2025-11-20 03:58:44,130 - DEBUG - VN ctest-vn1-56878927 is not present in Agent 10.0.0.131 2025-11-20 03:58:44,140 - DEBUG - VN ctest-vn1-56878927 is not present in Agent 10.0.0.48 2025-11-20 03:58:44,140 - INFO - Validated that VN ctest-vn1-56878927 is not in any agent 2025-11-20 03:58:44,170 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-56878927 info 2025-11-20 03:58:45,424 - DEBUG - Skipping xmpp flap check 2025-11-20 03:58:45,424 - INFO - END TEST : test_control_node_switchover : PASSED[0:03:21] 2025-11-20 03:58:45,424 - INFO - -------------------------------------------------------------------------------- 2025-11-20 03:58:45,428 - INFO - ================================================================================ 2025-11-20 03:58:45,429 - INFO - STARTING TEST : test_process_restart_in_policy_between_vns 2025-11-20 03:58:45,429 - INFO - TEST DESCRIPTION : Test to validate that with policy having rule to check icmp fwding between VMs on different VNs , ping between VMs should pass with process restarts 1. Pick 2 VN's from resource pool which has one VM each 2. Create policy with icmp allow rule between those VN's and bind it networks 3. Ping from one VM to another VM 4. Restart process 'vrouter' and 'control' on setup 5. Ping again between VM's after process restart Pass criteria: Step 2,3,4 and 5 should pass 2025-11-20 03:58:46,676 - DEBUG - Skipping xmpp flap check 2025-11-20 03:58:46,676 - INFO - Initial checks done. Running the testcase now 2025-11-20 03:58:46,676 - INFO - 2025-11-20 03:58:46,686 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-11-20 03:58:46,989 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-11-20 03:58:47,580 - INFO - Created VN ctest-vn1-60419904, UUID :de5b015d-d5f7-42fe-a691-8c1cdd7e7a17 2025-11-20 03:58:47,892 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-11-20 03:58:47,898 - DEBUG - Requesting: http://10.0.0.23:8082/domain/785d7fdc-4b61-45a1-9add-76a6d523389b 2025-11-20 03:58:47,909 - DEBUG - Requesting: http://10.0.0.23:8082/project/3339bf5d-318a-4edf-9cb8-4a7fd32a3af6 2025-11-20 03:58:48,010 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/de5b015d-d5f7-42fe-a691-8c1cdd7e7a17 2025-11-20 03:58:48,025 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/de5b015d-d5f7-42fe-a691-8c1cdd7e7a17 2025-11-20 03:58:48,037 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/46baa226-2fa2-4a9d-98af-eceff3d49ff5 2025-11-20 03:58:48,050 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/46baa226-2fa2-4a9d-98af-eceff3d49ff5 2025-11-20 03:58:48,059 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/7f938436-71dd-48a6-aa91-d10dac176928 2025-11-20 03:58:48,066 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-11-20 03:58:48,066 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/de5b015d-d5f7-42fe-a691-8c1cdd7e7a17 2025-11-20 03:58:48,075 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/46baa226-2fa2-4a9d-98af-eceff3d49ff5 2025-11-20 03:58:48,092 - INFO - Verified VN network id 15 for VN de5b015d-d5f7-42fe-a691-8c1cdd7e7a17 2025-11-20 03:58:48,092 - INFO - Verifications in API Server for VN ctest-vn1-60419904 passed 2025-11-20 03:58:48,093 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/de5b015d-d5f7-42fe-a691-8c1cdd7e7a17 2025-11-20 03:58:48,101 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/46baa226-2fa2-4a9d-98af-eceff3d49ff5 2025-11-20 03:58:48,109 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/46baa226-2fa2-4a9d-98af-eceff3d49ff5 2025-11-20 03:58:48,116 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/7f938436-71dd-48a6-aa91-d10dac176928 2025-11-20 03:58:48,128 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-17624131', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '16022401601834009342', 'uuid-lslong': '12002528537522108951'}, 'enable': 'true', 'created': '2025-11-20T03:58:47', 'last-modified': '2025-11-20T03:58:47', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.406224'} 2025-11-20 03:58:48,133 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-11-20 03:58:48,138 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-17624131', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '16022401601834009342', 'uuid-lslong': '12002528537522108951'}, 'enable': 'true', 'created': '2025-11-20T03:58:47', 'last-modified': '2025-11-20T03:58:47', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.417608'} 2025-11-20 03:58:48,144 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-11-20 03:58:48,150 - DEBUG - Control-node 10.0.0.22 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-17624131', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '16022401601834009342', 'uuid-lslong': '12002528537522108951'}, 'enable': 'true', 'created': '2025-11-20T03:58:47', 'last-modified': '2025-11-20T03:58:47', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.425577'} 2025-11-20 03:58:48,156 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-11-20 03:58:48,156 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-60419904 passed 2025-11-20 03:58:48,156 - DEBUG - ====Verifying policy data for ctest-vn1-60419904 in API_Server ====== 2025-11-20 03:58:48,156 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-11-20 03:58:48,161 - DEBUG - Requesting: http://10.0.0.23:8082/domain/785d7fdc-4b61-45a1-9add-76a6d523389b 2025-11-20 03:58:48,171 - DEBUG - Requesting: http://10.0.0.23:8082/project/3339bf5d-318a-4edf-9cb8-4a7fd32a3af6 2025-11-20 03:58:48,257 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/de5b015d-d5f7-42fe-a691-8c1cdd7e7a17 2025-11-20 03:58:48,268 - DEBUG - =>VN ctest-vn1-60419904 has no policy to be verified 2025-11-20 03:58:48,268 - DEBUG - Verifying the vn in opserver 2025-11-20 03:58:48,268 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 virtual network link through opserver 10.0.0.23 2025-11-20 03:58:48,268 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-11-20 03:58:48,280 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904?flat'} 2025-11-20 03:58:48,281 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 virtual network link through opserver 10.0.0.50 2025-11-20 03:58:48,281 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2025-11-20 03:58:48,295 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904?flat'} 2025-11-20 03:58:48,295 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 virtual network link through opserver 10.0.0.22 2025-11-20 03:58:48,295 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-networks 2025-11-20 03:58:48,310 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904?flat'} 2025-11-20 03:58:48,310 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 is found in opserver 2025-11-20 03:58:48,330 - DEBUG - Do not have enough data to verify VN in agent 2025-11-20 03:58:48,340 - DEBUG - VRF ids for VN ctest-vn1-60419904: {} 2025-11-20 03:58:48,984 - INFO - Created VN ctest-vn2-02050304, UUID :48185bc6-b2d0-44ad-9dad-8abd34951e77 2025-11-20 03:58:49,315 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-11-20 03:58:49,322 - DEBUG - Requesting: http://10.0.0.23:8082/domain/785d7fdc-4b61-45a1-9add-76a6d523389b 2025-11-20 03:58:49,335 - DEBUG - Requesting: http://10.0.0.23:8082/project/3339bf5d-318a-4edf-9cb8-4a7fd32a3af6 2025-11-20 03:58:49,435 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/48185bc6-b2d0-44ad-9dad-8abd34951e77 2025-11-20 03:58:49,447 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/48185bc6-b2d0-44ad-9dad-8abd34951e77 2025-11-20 03:58:49,465 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/7c46d3be-8510-44b3-b6e5-5c50faeefb6f 2025-11-20 03:58:49,472 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/7c46d3be-8510-44b3-b6e5-5c50faeefb6f 2025-11-20 03:58:49,480 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/69728de9-9068-4671-a46f-7f9729efbd68 2025-11-20 03:58:49,487 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-11-20 03:58:49,487 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/48185bc6-b2d0-44ad-9dad-8abd34951e77 2025-11-20 03:58:49,499 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/7c46d3be-8510-44b3-b6e5-5c50faeefb6f 2025-11-20 03:58:49,520 - INFO - Verified VN network id 16 for VN 48185bc6-b2d0-44ad-9dad-8abd34951e77 2025-11-20 03:58:49,520 - INFO - Verifications in API Server for VN ctest-vn2-02050304 passed 2025-11-20 03:58:49,520 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/48185bc6-b2d0-44ad-9dad-8abd34951e77 2025-11-20 03:58:49,533 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/7c46d3be-8510-44b3-b6e5-5c50faeefb6f 2025-11-20 03:58:49,541 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/7c46d3be-8510-44b3-b6e5-5c50faeefb6f 2025-11-20 03:58:49,551 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/69728de9-9068-4671-a46f-7f9729efbd68 2025-11-20 03:58:49,565 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-17624131', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '5195003079133512877', 'uuid-lslong': '11361889980161269367'}, 'enable': 'true', 'created': '2025-11-20T03:58:48', 'last-modified': '2025-11-20T03:58:49', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.428541'} 2025-11-20 03:58:49,572 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-11-20 03:58:49,578 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-17624131', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '5195003079133512877', 'uuid-lslong': '11361889980161269367'}, 'enable': 'true', 'created': '2025-11-20T03:58:48', 'last-modified': '2025-11-20T03:58:49', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.449476'} 2025-11-20 03:58:49,586 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-11-20 03:58:49,593 - DEBUG - Control-node 10.0.0.22 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-17624131', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '5195003079133512877', 'uuid-lslong': '11361889980161269367'}, 'enable': 'true', 'created': '2025-11-20T03:58:48', 'last-modified': '2025-11-20T03:58:49', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.464333'} 2025-11-20 03:58:49,599 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-11-20 03:58:49,599 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn2-02050304 passed 2025-11-20 03:58:49,599 - DEBUG - ====Verifying policy data for ctest-vn2-02050304 in API_Server ====== 2025-11-20 03:58:49,600 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-11-20 03:58:49,606 - DEBUG - Requesting: http://10.0.0.23:8082/domain/785d7fdc-4b61-45a1-9add-76a6d523389b 2025-11-20 03:58:49,618 - DEBUG - Requesting: http://10.0.0.23:8082/project/3339bf5d-318a-4edf-9cb8-4a7fd32a3af6 2025-11-20 03:58:49,718 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/48185bc6-b2d0-44ad-9dad-8abd34951e77 2025-11-20 03:58:49,732 - DEBUG - =>VN ctest-vn2-02050304 has no policy to be verified 2025-11-20 03:58:49,732 - DEBUG - Verifying the vn in opserver 2025-11-20 03:58:49,732 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 virtual network link through opserver 10.0.0.23 2025-11-20 03:58:49,732 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-11-20 03:58:49,748 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304?flat'} 2025-11-20 03:58:49,748 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 virtual network link through opserver 10.0.0.50 2025-11-20 03:58:49,748 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2025-11-20 03:58:49,763 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304?flat'} 2025-11-20 03:58:49,763 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 virtual network link through opserver 10.0.0.22 2025-11-20 03:58:49,763 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-networks 2025-11-20 03:58:49,775 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304?flat'} 2025-11-20 03:58:49,775 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 is found in opserver 2025-11-20 03:58:49,796 - DEBUG - Do not have enough data to verify VN in agent 2025-11-20 03:58:49,810 - DEBUG - VRF ids for VN ctest-vn2-02050304: {} 2025-11-20 03:58:51,378 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4768-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4768-2) 2025-11-20 03:58:52,753 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4768-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4768-1) 2025-11-20 03:58:52,754 - INFO - Waiting for VM ctest-vn1_vm1-06994196 to be up.. 2025-11-20 03:58:52,852 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-11-20 03:58:57,935 - DEBUG - VM is in ACTIVE state now 2025-11-20 03:58:57,935 - INFO - VM name : ctest-vn1_vm1-06994196 2025-11-20 03:58:58,013 - DEBUG - VM ctest-vn1_vm1-06994196 ID is 17c6dd2e-c649-4d03-a808-5e1fb84b0f7d 2025-11-20 03:58:58,013 - DEBUG - VM ctest-vn1_vm1-06994196 launched on Node an-jenkins-deploy-platform-ansible-os-4768-2 2025-11-20 03:58:58,088 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/17c6dd2e-c649-4d03-a808-5e1fb84b0f7d 2025-11-20 03:58:58,098 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine-interface/51607d0e-83dc-469f-9445-305fd2a9bb78 2025-11-20 03:59:01,282 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1021ms') 2025-11-20 03:59:01,282 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-06994196 failed! 2025-11-20 03:59:01,299 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 is 192.168.1.254 and allocation pool is NOT set 2025-11-20 03:59:05,377 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1016ms') 2025-11-20 03:59:05,377 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-06994196 failed! 2025-11-20 03:59:05,391 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 is 192.168.1.254 and allocation pool is NOT set 2025-11-20 03:59:09,475 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-11-20 03:59:09,475 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-06994196 failed! 2025-11-20 03:59:09,494 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 is 192.168.1.254 and allocation pool is NOT set 2025-11-20 03:59:13,566 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2025-11-20 03:59:13,566 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-06994196 failed! 2025-11-20 03:59:13,581 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 is 192.168.1.254 and allocation pool is NOT set 2025-11-20 03:59:17,663 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-11-20 03:59:17,663 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-06994196 failed! 2025-11-20 03:59:17,676 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 is 192.168.1.254 and allocation pool is NOT set 2025-11-20 03:59:21,757 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-11-20 03:59:21,758 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-06994196 failed! 2025-11-20 03:59:21,771 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 is 192.168.1.254 and allocation pool is NOT set 2025-11-20 03:59:25,855 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1027ms') 2025-11-20 03:59:25,856 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-06994196 failed! 2025-11-20 03:59:25,871 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 is 192.168.1.254 and allocation pool is NOT set 2025-11-20 03:59:29,949 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-11-20 03:59:29,949 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-06994196 failed! 2025-11-20 03:59:29,964 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 is 192.168.1.254 and allocation pool is NOT set 2025-11-20 03:59:32,035 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=87.8 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=6.11 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 6.114/46.944/87.775/40.830 ms') 2025-11-20 03:59:32,035 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-06994196 passed 2025-11-20 03:59:32,110 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 03:59:32,110 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-06994196, IP 192.168.1.252, Port 22 2025-11-20 03:59:32,178 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 03:59:32,258 - DEBUG - VM ctest-vn1_vm1-06994196 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 03:59:37,259 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 03:59:37,259 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-06994196, IP 192.168.1.252, Port 22 2025-11-20 03:59:37,327 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 03:59:37,413 - DEBUG - VM ctest-vn1_vm1-06994196 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 03:59:42,413 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 03:59:42,413 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-06994196, IP 192.168.1.252, Port 22 2025-11-20 03:59:42,486 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 03:59:42,568 - DEBUG - VM ctest-vn1_vm1-06994196 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 03:59:47,568 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 03:59:47,569 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-06994196, IP 192.168.1.252, Port 22 2025-11-20 03:59:47,635 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 03:59:47,718 - DEBUG - VM ctest-vn1_vm1-06994196 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 03:59:52,719 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 03:59:52,719 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-06994196, IP 192.168.1.252, Port 22 2025-11-20 03:59:52,886 - DEBUG - VM ctest-vn1_vm1-06994196 is ready for SSH connections 2025-11-20 03:59:52,886 - INFO - Waiting for VM ctest-vn2_vm1-40788747 to be up.. 2025-11-20 03:59:52,969 - DEBUG - VM is in ACTIVE state now 2025-11-20 03:59:52,969 - INFO - VM name : ctest-vn2_vm1-40788747 2025-11-20 03:59:53,055 - DEBUG - VM ctest-vn2_vm1-40788747 ID is 8c997a08-f5ca-4d39-9047-d7c5eb494bd0 2025-11-20 03:59:53,055 - DEBUG - VM ctest-vn2_vm1-40788747 launched on Node an-jenkins-deploy-platform-ansible-os-4768-1 2025-11-20 03:59:53,142 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/8c997a08-f5ca-4d39-9047-d7c5eb494bd0 2025-11-20 03:59:53,153 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine-interface/987eb33b-d1d5-4a23-8a9d-6ebad070c276 2025-11-20 03:59:54,334 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=55.9 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=16.9 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1000ms\r\nrtt min/avg/max/mdev = 16.920/36.419/55.919/19.499 ms') 2025-11-20 03:59:54,334 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn2_vm1-40788747 passed 2025-11-20 03:59:54,406 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 03:59:54,406 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-40788747, IP 192.168.2.252, Port 22 2025-11-20 03:59:54,494 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 03:59:54,586 - DEBUG - VM ctest-vn2_vm1-40788747 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 03:59:59,587 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 03:59:59,587 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-40788747, IP 192.168.2.252, Port 22 2025-11-20 03:59:59,666 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 03:59:59,750 - DEBUG - VM ctest-vn2_vm1-40788747 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 04:00:04,751 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:00:04,751 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-40788747, IP 192.168.2.252, Port 22 2025-11-20 04:00:04,839 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 04:00:04,923 - DEBUG - VM ctest-vn2_vm1-40788747 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 04:00:09,925 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:00:09,925 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-40788747, IP 192.168.2.252, Port 22 2025-11-20 04:00:10,016 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 04:00:10,115 - DEBUG - VM ctest-vn2_vm1-40788747 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 04:00:15,116 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:00:15,116 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-40788747, IP 192.168.2.252, Port 22 2025-11-20 04:00:15,243 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 04:00:15,334 - DEBUG - VM ctest-vn2_vm1-40788747 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 04:00:20,335 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:00:20,335 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-40788747, IP 192.168.2.252, Port 22 2025-11-20 04:00:20,413 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 04:00:20,504 - DEBUG - VM ctest-vn2_vm1-40788747 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 04:00:25,505 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:00:25,505 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-40788747, IP 192.168.2.252, Port 22 2025-11-20 04:00:25,586 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 04:00:25,702 - DEBUG - VM ctest-vn2_vm1-40788747 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 04:00:30,703 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:00:30,703 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-40788747, IP 192.168.2.252, Port 22 2025-11-20 04:00:30,802 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 04:00:30,961 - DEBUG - VM ctest-vn2_vm1-40788747 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 04:00:35,962 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:00:35,962 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-40788747, IP 192.168.2.252, Port 22 2025-11-20 04:00:36,030 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 04:00:36,153 - DEBUG - VM ctest-vn2_vm1-40788747 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 04:00:41,154 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:00:41,154 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-40788747, IP 192.168.2.252, Port 22 2025-11-20 04:00:41,233 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 04:00:41,333 - DEBUG - VM ctest-vn2_vm1-40788747 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 04:00:46,334 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:00:46,335 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-40788747, IP 192.168.2.252, Port 22 2025-11-20 04:00:46,412 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 04:00:46,506 - DEBUG - VM ctest-vn2_vm1-40788747 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 04:00:51,506 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:00:51,507 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-40788747, IP 192.168.2.252, Port 22 2025-11-20 04:00:51,574 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 04:00:51,656 - DEBUG - VM ctest-vn2_vm1-40788747 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 04:00:56,657 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:00:56,657 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-40788747, IP 192.168.2.252, Port 22 2025-11-20 04:00:56,733 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 04:00:56,831 - DEBUG - VM ctest-vn2_vm1-40788747 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 04:01:01,832 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:01:01,832 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-40788747, IP 192.168.2.252, Port 22 2025-11-20 04:01:01,908 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 04:01:01,992 - DEBUG - VM ctest-vn2_vm1-40788747 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 04:01:06,993 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:01:06,993 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-40788747, IP 192.168.2.252, Port 22 2025-11-20 04:01:07,083 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 04:01:07,165 - DEBUG - VM ctest-vn2_vm1-40788747 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 04:01:12,165 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:01:12,165 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-40788747, IP 192.168.2.252, Port 22 2025-11-20 04:01:12,234 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 04:01:12,339 - DEBUG - VM ctest-vn2_vm1-40788747 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 04:01:17,340 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:01:17,340 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-40788747, IP 192.168.2.252, Port 22 2025-11-20 04:01:17,406 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 04:01:17,492 - DEBUG - VM ctest-vn2_vm1-40788747 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 04:01:22,492 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:01:22,492 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-40788747, IP 192.168.2.252, Port 22 2025-11-20 04:01:22,662 - DEBUG - VM ctest-vn2_vm1-40788747 is ready for SSH connections 2025-11-20 04:01:22,662 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:01:22,662 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.252, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.48, gateway password: c0ntrail123 2025-11-20 04:01:22,662 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.252 2025-11-20 04:02:06,486 - DEBUG - PING 192.168.2.252 (192.168.2.252) 56(84) bytes of data. 64 bytes from 192.168.2.252: icmp_req=1 ttl=63 time=10.3 ms 64 bytes from 192.168.2.252: icmp_req=2 ttl=63 time=1.05 ms 64 bytes from 192.168.2.252: icmp_req=3 ttl=63 time=1.45 ms --- 192.168.2.252 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.055/4.283/10.336/4.283 ms 2025-11-20 04:02:06,486 - INFO - Ping to IP 192.168.2.252 from VM ctest-vn1_vm1-06994196 passed 2025-11-20 04:02:09,890 - INFO - 10.0.0.131 2025-11-20 04:02:09,894 - INFO - agent:initializing 2025-11-20 04:02:09,894 - DEBUG - defaultdict(, {'10.0.0.131': {'agent': {'status': 'initializing', 'description': None}}}) 2025-11-20 04:02:09,894 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-11-20 04:02:14,983 - INFO - 10.0.0.131 2025-11-20 04:02:14,989 - INFO - agent:active 2025-11-20 04:02:14,989 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.131 2025-11-20 04:02:18,415 - INFO - 10.0.0.48 2025-11-20 04:02:18,419 - INFO - agent:initializing 2025-11-20 04:02:18,419 - DEBUG - defaultdict(, {'10.0.0.48': {'agent': {'status': 'initializing', 'description': None}}}) 2025-11-20 04:02:18,419 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-11-20 04:02:23,508 - INFO - 10.0.0.48 2025-11-20 04:02:23,513 - INFO - agent:active 2025-11-20 04:02:23,513 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.48 2025-11-20 04:02:24,007 - INFO - 10.0.0.23 2025-11-20 04:02:24,019 - INFO - control:initializing 2025-11-20 04:02:24,019 - DEBUG - defaultdict(, {'10.0.0.23': {'control': {'status': 'initializing', 'description': None}}}) 2025-11-20 04:02:24,019 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-11-20 04:02:29,139 - INFO - 10.0.0.23 2025-11-20 04:02:29,146 - INFO - control:active 2025-11-20 04:02:29,146 - INFO - Contrail services ['control'] are up on nodes 10.0.0.23 2025-11-20 04:02:29,555 - INFO - 10.0.0.50 2025-11-20 04:02:29,560 - INFO - control:initializing 2025-11-20 04:02:29,560 - DEBUG - defaultdict(, {'10.0.0.50': {'control': {'status': 'initializing', 'description': None}}}) 2025-11-20 04:02:29,560 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-11-20 04:02:34,659 - INFO - 10.0.0.50 2025-11-20 04:02:34,668 - INFO - control:active 2025-11-20 04:02:34,668 - INFO - Contrail services ['control'] are up on nodes 10.0.0.50 2025-11-20 04:02:35,101 - INFO - 10.0.0.22 2025-11-20 04:02:35,104 - INFO - control:initializing 2025-11-20 04:02:35,105 - DEBUG - defaultdict(, {'10.0.0.22': {'control': {'status': 'initializing', 'description': None}}}) 2025-11-20 04:02:35,105 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-11-20 04:02:40,205 - INFO - 10.0.0.22 2025-11-20 04:02:40,211 - INFO - control:active 2025-11-20 04:02:40,212 - INFO - Contrail services ['control'] are up on nodes 10.0.0.22 2025-11-20 04:02:40,976 - INFO - 10.0.0.23 2025-11-20 04:02:40,980 - INFO - api-server:initializing 2025-11-20 04:02:40,980 - DEBUG - defaultdict(, {'10.0.0.23': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-11-20 04:02:40,980 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-11-20 04:02:46,143 - INFO - 10.0.0.23 2025-11-20 04:02:46,147 - INFO - api-server:initializing 2025-11-20 04:02:46,148 - DEBUG - defaultdict(, {'10.0.0.23': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-11-20 04:02:46,148 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-11-20 04:02:51,261 - INFO - 10.0.0.23 2025-11-20 04:02:51,266 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart] connection down) 2025-11-20 04:02:51,266 - DEBUG - defaultdict(, {'10.0.0.23': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart] connection down'}}}) 2025-11-20 04:02:51,266 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-11-20 04:02:56,377 - INFO - 10.0.0.23 2025-11-20 04:02:56,381 - INFO - api-server:active 2025-11-20 04:02:56,382 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.23 2025-11-20 04:02:57,052 - INFO - 10.0.0.50 2025-11-20 04:02:57,056 - INFO - api-server:initializing 2025-11-20 04:02:57,056 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-11-20 04:02:57,056 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-11-20 04:03:02,170 - INFO - 10.0.0.50 2025-11-20 04:03:02,175 - INFO - api-server:initializing 2025-11-20 04:03:02,175 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-11-20 04:03:02,175 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-11-20 04:03:07,275 - INFO - 10.0.0.50 2025-11-20 04:03:07,280 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2025-11-20 04:03:07,280 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2025-11-20 04:03:07,281 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-11-20 04:03:12,380 - INFO - 10.0.0.50 2025-11-20 04:03:12,384 - INFO - api-server:active 2025-11-20 04:03:12,385 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.50 2025-11-20 04:03:13,086 - INFO - 10.0.0.22 2025-11-20 04:03:13,090 - INFO - api-server:initializing 2025-11-20 04:03:13,090 - DEBUG - defaultdict(, {'10.0.0.22': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-11-20 04:03:13,090 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-11-20 04:03:18,200 - INFO - 10.0.0.22 2025-11-20 04:03:18,204 - INFO - api-server:initializing 2025-11-20 04:03:18,204 - DEBUG - defaultdict(, {'10.0.0.22': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-11-20 04:03:18,204 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-11-20 04:03:23,295 - INFO - 10.0.0.22 2025-11-20 04:03:23,303 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2025-11-20 04:03:23,303 - DEBUG - defaultdict(, {'10.0.0.22': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2025-11-20 04:03:23,303 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-11-20 04:03:28,391 - INFO - 10.0.0.22 2025-11-20 04:03:28,396 - INFO - api-server:active 2025-11-20 04:03:28,397 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.22 2025-11-20 04:03:28,497 - INFO - 10.0.0.23 2025-11-20 04:03:28,497 - INFO - config 2025-11-20 04:03:28,502 - INFO - config-nodemgr:active 2025-11-20 04:03:28,507 - INFO - api-server:active 2025-11-20 04:03:28,511 - INFO - schema:active 2025-11-20 04:03:28,514 - INFO - svc-monitor:active 2025-11-20 04:03:28,517 - INFO - device-manager:backup 2025-11-20 04:03:28,517 - INFO - config-database 2025-11-20 04:03:28,517 - INFO - config-cassandra:active 2025-11-20 04:03:28,517 - INFO - config-zookeeper:active 2025-11-20 04:03:28,517 - INFO - config-rabbitmq:active 2025-11-20 04:03:28,518 - INFO - control 2025-11-20 04:03:28,522 - INFO - control-nodemgr:active 2025-11-20 04:03:28,527 - INFO - control:active 2025-11-20 04:03:28,528 - INFO - named:active 2025-11-20 04:03:28,533 - INFO - dns:active 2025-11-20 04:03:28,533 - INFO - analytics 2025-11-20 04:03:28,538 - INFO - analytics-nodemgr:active 2025-11-20 04:03:28,542 - INFO - analytics-api:active 2025-11-20 04:03:28,547 - INFO - collector:active 2025-11-20 04:03:28,547 - INFO - analytics-database 2025-11-20 04:03:28,547 - INFO - analytics-cassandra:active 2025-11-20 04:03:28,551 - INFO - analyticsdb-nodemgr:active 2025-11-20 04:03:28,557 - INFO - query-engine:active 2025-11-20 04:03:28,557 - INFO - webui 2025-11-20 04:03:28,558 - INFO - webui:active 2025-11-20 04:03:28,558 - INFO - webui-middleware:active 2025-11-20 04:03:28,558 - INFO - redis:active 2025-11-20 04:03:28,558 - INFO - analytics_snmp 2025-11-20 04:03:28,562 - INFO - snmp-collector:active 2025-11-20 04:03:28,562 - INFO - snmp-topology:active 2025-11-20 04:03:28,562 - INFO - analytics_alarm 2025-11-20 04:03:28,562 - INFO - alarmgen:active 2025-11-20 04:03:28,660 - INFO - 10.0.0.50 2025-11-20 04:03:28,660 - INFO - config 2025-11-20 04:03:28,667 - INFO - config-nodemgr:active 2025-11-20 04:03:28,671 - INFO - api-server:active 2025-11-20 04:03:28,675 - INFO - schema:backup 2025-11-20 04:03:28,678 - INFO - svc-monitor:backup 2025-11-20 04:03:28,682 - INFO - device-manager:active 2025-11-20 04:03:28,682 - INFO - config-database 2025-11-20 04:03:28,682 - INFO - config-cassandra:active 2025-11-20 04:03:28,682 - INFO - config-zookeeper:active 2025-11-20 04:03:28,683 - INFO - config-rabbitmq:active 2025-11-20 04:03:28,683 - INFO - control 2025-11-20 04:03:28,689 - INFO - control-nodemgr:active 2025-11-20 04:03:28,694 - INFO - control:active 2025-11-20 04:03:28,694 - INFO - named:active 2025-11-20 04:03:28,699 - INFO - dns:active 2025-11-20 04:03:28,700 - INFO - analytics 2025-11-20 04:03:28,706 - INFO - analytics-nodemgr:active 2025-11-20 04:03:28,711 - INFO - analytics-api:active 2025-11-20 04:03:28,717 - INFO - collector:active 2025-11-20 04:03:28,717 - INFO - analytics-database 2025-11-20 04:03:28,717 - INFO - analytics-cassandra:active 2025-11-20 04:03:28,723 - INFO - analyticsdb-nodemgr:active 2025-11-20 04:03:28,729 - INFO - query-engine:active 2025-11-20 04:03:28,729 - INFO - webui 2025-11-20 04:03:28,729 - INFO - webui:active 2025-11-20 04:03:28,729 - INFO - webui-middleware:active 2025-11-20 04:03:28,729 - INFO - redis:active 2025-11-20 04:03:28,730 - INFO - analytics_snmp 2025-11-20 04:03:28,736 - INFO - snmp-collector:active 2025-11-20 04:03:28,736 - INFO - snmp-topology:active 2025-11-20 04:03:28,736 - INFO - analytics_alarm 2025-11-20 04:03:28,736 - INFO - alarmgen:active 2025-11-20 04:03:28,833 - INFO - 10.0.0.22 2025-11-20 04:03:28,833 - INFO - config 2025-11-20 04:03:28,840 - INFO - config-nodemgr:active 2025-11-20 04:03:28,844 - INFO - api-server:active 2025-11-20 04:03:28,848 - INFO - schema:backup 2025-11-20 04:03:28,850 - INFO - svc-monitor:backup 2025-11-20 04:03:28,853 - INFO - device-manager:backup 2025-11-20 04:03:28,853 - INFO - config-database 2025-11-20 04:03:28,853 - INFO - config-cassandra:active 2025-11-20 04:03:28,854 - INFO - config-zookeeper:active 2025-11-20 04:03:28,854 - INFO - config-rabbitmq:active 2025-11-20 04:03:28,854 - INFO - control 2025-11-20 04:03:28,859 - INFO - control-nodemgr:active 2025-11-20 04:03:28,865 - INFO - control:active 2025-11-20 04:03:28,865 - INFO - named:active 2025-11-20 04:03:28,870 - INFO - dns:active 2025-11-20 04:03:28,870 - INFO - analytics 2025-11-20 04:03:28,876 - INFO - analytics-nodemgr:active 2025-11-20 04:03:28,881 - INFO - analytics-api:active 2025-11-20 04:03:28,886 - INFO - collector:active 2025-11-20 04:03:28,887 - INFO - analytics-database 2025-11-20 04:03:28,887 - INFO - analytics-cassandra:active 2025-11-20 04:03:28,892 - INFO - analyticsdb-nodemgr:active 2025-11-20 04:03:28,899 - INFO - query-engine:active 2025-11-20 04:03:28,899 - INFO - webui 2025-11-20 04:03:28,899 - INFO - webui:active 2025-11-20 04:03:28,899 - INFO - webui-middleware:active 2025-11-20 04:03:28,899 - INFO - redis:active 2025-11-20 04:03:28,899 - INFO - analytics_snmp 2025-11-20 04:03:28,904 - INFO - snmp-collector:active 2025-11-20 04:03:28,904 - INFO - snmp-topology:active 2025-11-20 04:03:28,904 - INFO - analytics_alarm 2025-11-20 04:03:28,904 - INFO - alarmgen:active 2025-11-20 04:03:28,990 - INFO - 10.0.0.131 2025-11-20 04:03:28,991 - INFO - vrouter 2025-11-20 04:03:28,995 - INFO - vrouter-nodemgr:active 2025-11-20 04:03:29,001 - INFO - agent:active 2025-11-20 04:03:29,086 - INFO - 10.0.0.48 2025-11-20 04:03:29,086 - INFO - vrouter 2025-11-20 04:03:29,092 - INFO - vrouter-nodemgr:active 2025-11-20 04:03:29,098 - INFO - agent:active 2025-11-20 04:03:29,098 - INFO - All the contrail services are up on all nodes 2025-11-20 04:03:29,098 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-1?flat 2025-11-20 04:03:29,516 - INFO - an-jenkins-deploy-platform-ansible-os-4768-1:contrail-vrouter-agent module connection to ['10.20.0.129:5269'] servers UP 2025-11-20 04:03:29,516 - INFO - an-jenkins-deploy-platform-ansible-os-4768-1 connected to xmpp 10.20.0.129 2025-11-20 04:03:29,516 - INFO - an-jenkins-deploy-platform-ansible-os-4768-1:contrail-vrouter-agent module connection to ['10.20.0.18:5269'] servers UP 2025-11-20 04:03:29,516 - INFO - an-jenkins-deploy-platform-ansible-os-4768-1 connected to xmpp 10.20.0.18 2025-11-20 04:03:29,516 - WARNING - an-jenkins-deploy-platform-ansible-os-4768-1:contrail-vrouter-agent module connection to ['10.20.0.201:5269'] servers NOT UP 2025-11-20 04:03:29,516 - INFO - an-jenkins-deploy-platform-ansible-os-4768-1:contrail-vrouter-agent module connection to ['10.20.0.129:53'] servers UP 2025-11-20 04:03:29,516 - INFO - an-jenkins-deploy-platform-ansible-os-4768-1 connected to dns 10.20.0.129 2025-11-20 04:03:29,517 - INFO - an-jenkins-deploy-platform-ansible-os-4768-1:contrail-vrouter-agent module connection to ['10.20.0.18:53'] servers UP 2025-11-20 04:03:29,517 - INFO - an-jenkins-deploy-platform-ansible-os-4768-1 connected to dns 10.20.0.18 2025-11-20 04:03:29,517 - WARNING - an-jenkins-deploy-platform-ansible-os-4768-1:contrail-vrouter-agent module connection to ['10.20.0.201:53'] servers NOT UP 2025-11-20 04:03:29,517 - WARNING - an-jenkins-deploy-platform-ansible-os-4768-1:contrail-vrouter-agent module connection to ['10.0.0.23:8086'] servers NOT UP 2025-11-20 04:03:29,517 - WARNING - an-jenkins-deploy-platform-ansible-os-4768-1:contrail-vrouter-agent module connection to ['10.0.0.50:8086'] servers NOT UP 2025-11-20 04:03:29,518 - INFO - an-jenkins-deploy-platform-ansible-os-4768-1:contrail-vrouter-agent module connection to ['10.0.0.22:8086'] servers UP 2025-11-20 04:03:29,518 - INFO - an-jenkins-deploy-platform-ansible-os-4768-1 connected to collector 10.0.0.22 2025-11-20 04:03:29,518 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4768-1?flat 2025-11-20 04:03:29,942 - INFO - cn-jenkins-deploy-platform-ansible-os-4768-1:contrail-control module connection to 10.0.0.23:9041 servers UP 2025-11-20 04:03:29,942 - INFO - cn-jenkins-deploy-platform-ansible-os-4768-1:contrail-control module connection to 10.0.0.23:5673 servers UP 2025-11-20 04:03:29,942 - INFO - cn-jenkins-deploy-platform-ansible-os-4768-1:contrail-control module connection to ['10.0.0.23:8086'] servers UP 2025-11-20 04:03:29,942 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4768-2?flat 2025-11-20 04:03:30,334 - INFO - cn-jenkins-deploy-platform-ansible-os-4768-2:contrail-control module connection to 10.0.0.23:9041 servers UP 2025-11-20 04:03:30,334 - INFO - cn-jenkins-deploy-platform-ansible-os-4768-2:contrail-control module connection to 10.0.0.23:5673 servers UP 2025-11-20 04:03:30,334 - INFO - cn-jenkins-deploy-platform-ansible-os-4768-2:contrail-control module connection to ['10.0.0.23:8086'] servers UP 2025-11-20 04:03:30,334 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4768-3?flat 2025-11-20 04:03:30,353 - INFO - cn-jenkins-deploy-platform-ansible-os-4768-3:contrail-control module connection to 10.0.0.23:9041 servers UP 2025-11-20 04:03:30,353 - INFO - cn-jenkins-deploy-platform-ansible-os-4768-3:contrail-control module connection to 10.0.0.23:5673 servers UP 2025-11-20 04:03:30,353 - INFO - cn-jenkins-deploy-platform-ansible-os-4768-3:contrail-control module connection to ['10.0.0.23:8086'] servers UP 2025-11-20 04:03:30,353 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4768-1.?flat 2025-11-20 04:03:30,377 - DEBUG - contrail-api is running 2025-11-20 04:03:30,377 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4768-2.?flat 2025-11-20 04:03:30,402 - DEBUG - contrail-api is running 2025-11-20 04:03:30,402 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4768-3.?flat 2025-11-20 04:03:30,429 - DEBUG - contrail-api is running 2025-11-20 04:03:30,728 - INFO - Created VN ctest-vn3-21173747, UUID :6c7da220-f2bf-4638-aa73-544f525b2071 2025-11-20 04:03:31,136 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-11-20 04:03:31,173 - DEBUG - Requesting: http://10.0.0.23:8082/domain/785d7fdc-4b61-45a1-9add-76a6d523389b 2025-11-20 04:03:31,187 - DEBUG - Requesting: http://10.0.0.23:8082/project/3339bf5d-318a-4edf-9cb8-4a7fd32a3af6 2025-11-20 04:03:31,305 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/de5b015d-d5f7-42fe-a691-8c1cdd7e7a17 2025-11-20 04:03:31,327 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/de5b015d-d5f7-42fe-a691-8c1cdd7e7a17 2025-11-20 04:03:31,375 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/46baa226-2fa2-4a9d-98af-eceff3d49ff5 2025-11-20 04:03:31,385 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/46baa226-2fa2-4a9d-98af-eceff3d49ff5 2025-11-20 04:03:31,394 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/7f938436-71dd-48a6-aa91-d10dac176928 2025-11-20 04:03:31,402 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-11-20 04:03:31,402 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/de5b015d-d5f7-42fe-a691-8c1cdd7e7a17 2025-11-20 04:03:31,414 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/46baa226-2fa2-4a9d-98af-eceff3d49ff5 2025-11-20 04:03:31,431 - INFO - Verified VN network id 15 for VN de5b015d-d5f7-42fe-a691-8c1cdd7e7a17 2025-11-20 04:03:31,432 - INFO - Verifications in API Server for VN ctest-vn1-60419904 passed 2025-11-20 04:03:31,432 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/de5b015d-d5f7-42fe-a691-8c1cdd7e7a17 2025-11-20 04:03:31,444 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/46baa226-2fa2-4a9d-98af-eceff3d49ff5 2025-11-20 04:03:31,456 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/46baa226-2fa2-4a9d-98af-eceff3d49ff5 2025-11-20 04:03:31,467 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/7f938436-71dd-48a6-aa91-d10dac176928 2025-11-20 04:03:31,482 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-17624131', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '16022401601834009342', 'uuid-lslong': '12002528537522108951'}, 'enable': 'true', 'created': '2025-11-20T03:58:47', 'last-modified': '2025-11-20T03:58:48', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:07.098059'} 2025-11-20 04:03:31,491 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-11-20 04:03:31,499 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-17624131', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '16022401601834009342', 'uuid-lslong': '12002528537522108951'}, 'enable': 'true', 'created': '2025-11-20T03:58:47', 'last-modified': '2025-11-20T03:58:48', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:01.595093'} 2025-11-20 04:03:31,506 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-11-20 04:03:31,512 - DEBUG - Control-node 10.0.0.22 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-17624131', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '16022401601834009342', 'uuid-lslong': '12002528537522108951'}, 'enable': 'true', 'created': '2025-11-20T03:58:47', 'last-modified': '2025-11-20T03:58:48', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:55.997562'} 2025-11-20 04:03:31,518 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-11-20 04:03:31,518 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-60419904 passed 2025-11-20 04:03:31,518 - DEBUG - ====Verifying policy data for ctest-vn1-60419904 in API_Server ====== 2025-11-20 04:03:31,518 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-11-20 04:03:31,525 - DEBUG - Requesting: http://10.0.0.23:8082/domain/785d7fdc-4b61-45a1-9add-76a6d523389b 2025-11-20 04:03:31,537 - DEBUG - Requesting: http://10.0.0.23:8082/project/3339bf5d-318a-4edf-9cb8-4a7fd32a3af6 2025-11-20 04:03:31,644 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/de5b015d-d5f7-42fe-a691-8c1cdd7e7a17 2025-11-20 04:03:31,657 - DEBUG - ==>Verifying data for policy with id: 62b6dcc5-9000-44b8-b9fa-b948b6eb09f3, fqn: ['default-domain', 'ctest-TestBasicVMVN0-17624131', 'policy1'] 2025-11-20 04:03:31,657 - INFO - VN ctest-vn1-60419904 Policy verification: verify_vn_policy_in_api_server, status: True 2025-11-20 04:03:31,657 - DEBUG - Verifying the vn in opserver 2025-11-20 04:03:31,657 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 virtual network link through opserver 10.0.0.23 2025-11-20 04:03:31,657 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-11-20 04:03:31,671 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904?flat'} 2025-11-20 04:03:31,671 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 virtual network link through opserver 10.0.0.50 2025-11-20 04:03:31,671 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2025-11-20 04:03:31,739 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904?flat'} 2025-11-20 04:03:31,739 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 virtual network link through opserver 10.0.0.22 2025-11-20 04:03:31,739 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-networks 2025-11-20 04:03:31,813 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904?flat'} 2025-11-20 04:03:31,813 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 is found in opserver 2025-11-20 04:03:31,825 - DEBUG - VRF ids for VN ctest-vn1-60419904: {'10.0.0.48': '2'} 2025-11-20 04:03:33,085 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4768-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4768-2) 2025-11-20 04:03:33,131 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-11-20 04:03:38,208 - DEBUG - VM is in ACTIVE state now 2025-11-20 04:03:38,208 - INFO - VM name : ctest-vn1_vm2-87721429 2025-11-20 04:03:38,284 - DEBUG - VM ctest-vn1_vm2-87721429 ID is 78c20816-8477-4524-be45-3f39c6c1958c 2025-11-20 04:03:38,358 - DEBUG - VM is in ACTIVE state now 2025-11-20 04:03:38,359 - DEBUG - VM ctest-vn1_vm2-87721429 launched on Node an-jenkins-deploy-platform-ansible-os-4768-2 2025-11-20 04:03:38,442 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/78c20816-8477-4524-be45-3f39c6c1958c 2025-11-20 04:03:38,478 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/78c20816-8477-4524-be45-3f39c6c1958c 2025-11-20 04:03:38,486 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine-interface/51d726ac-093b-4d31-b0a6-79e13854daf7 2025-11-20 04:03:38,496 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/78c20816-8477-4524-be45-3f39c6c1958c 2025-11-20 04:03:38,503 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine-interface/51d726ac-093b-4d31-b0a6-79e13854daf7 2025-11-20 04:03:38,512 - DEBUG - Requesting: http://10.0.0.22:8082/instance-ip/fd4c3b30-da11-4a99-a6e7-e07a1c6ec824 2025-11-20 04:03:38,523 - DEBUG - Verifying in api server 10.0.0.22 2025-11-20 04:03:38,523 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/78c20816-8477-4524-be45-3f39c6c1958c 2025-11-20 04:03:38,532 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine-interface/51d726ac-093b-4d31-b0a6-79e13854daf7 2025-11-20 04:03:38,546 - DEBUG - Requesting: http://10.0.0.22:8082/instance-ip/fd4c3b30-da11-4a99-a6e7-e07a1c6ec824 2025-11-20 04:03:38,553 - INFO - VM ctest-vn1_vm2-87721429 verfication in all API Servers passed 2025-11-20 04:03:38,571 - DEBUG - VM ctest-vn1_vm2-87721429 Tap interface: {'index': '4', 'name': 'tap51d726ac-09', 'uuid': '51d726ac-093b-4d31-b0a6-79e13854daf7', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904:ctest-vn1-60419904', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904', 'vm_uuid': '78c20816-8477-4524-be45-3f39c6c1958c', 'vm_name': 'ctest-vn1_vm2-87721429', 'ip_addr': '192.168.1.251', 'mac_addr': '02:51:d7:26:ac:09', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '167', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-17624131:51d726ac-093b-4d31-b0a6-79e13854daf7', 'sg_uuid_list': ['db81be13-9b62-47a3-a92b-985308dc1af8'], 'static_route_list': None, 'vm_project_uuid': '3339bf5d-318a-4edf-9cb8-4a7fd32a3af6', 'admin_state': 'Enabled', 'flow_key_idx': '25', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-11-20 04:03:38,571 - DEBUG - Agent 10.0.0.48 vrf name: default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904:ctest-vn1-60419904 2025-11-20 04:03:38,577 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904:ctest-vn1-60419904', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904', 'table_label': '-1', 'vxlan_id': '15', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.25:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-11-20 04:03:38,659 - DEBUG - VM ctest-vn1_vm2-87721429 : Tap interface tap51d726ac-09 is set to Active 2025-11-20 04:03:38,660 - DEBUG - Tap interface tap51d726ac-09 detail : {'index': '4', 'name': 'tap51d726ac-09', 'uuid': '51d726ac-093b-4d31-b0a6-79e13854daf7', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904:ctest-vn1-60419904', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904', 'vm_uuid': '78c20816-8477-4524-be45-3f39c6c1958c', 'vm_name': 'ctest-vn1_vm2-87721429', 'ip_addr': '192.168.1.251', 'mac_addr': '02:51:d7:26:ac:09', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '167', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-17624131:51d726ac-093b-4d31-b0a6-79e13854daf7', 'sg_uuid_list': ['db81be13-9b62-47a3-a92b-985308dc1af8'], 'static_route_list': None, 'vm_project_uuid': '3339bf5d-318a-4edf-9cb8-4a7fd32a3af6', 'admin_state': 'Enabled', 'flow_key_idx': '25', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-11-20 04:03:38,774 - DEBUG - Starting Layer 2 verification in Agent 2025-11-20 04:03:38,790 - DEBUG - Layer 2 path is seen for VM MAC 02:51:d7:26:ac:09 in agent 10.0.0.48 2025-11-20 04:03:38,790 - DEBUG - Active layer 2 route in agent is present for VMI tap51d726ac-09 2025-11-20 04:03:38,790 - DEBUG - L2 label(34) matches bw route table and interface table 2025-11-20 04:03:38,805 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 is found in Agent of node 10.0.0.48 2025-11-20 04:03:38,941 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 is consistent in agent 10.0.0.48 2025-11-20 04:03:38,941 - DEBUG - Route for VM IP ['192.168.1.251'] is consistent in agent 10.0.0.48 2025-11-20 04:03:38,941 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 verification for VM ctest-vn1_vm2-87721429 in Agent 10.0.0.48 passed 2025-11-20 04:03:39,058 - DEBUG - Starting all layer 2 verification in agent 10.0.0.48 2025-11-20 04:03:39,072 - DEBUG - Route for VM MAC 02:51:d7:26:ac:09 is consistent in agent 10.0.0.48 2025-11-20 04:03:42,272 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2025-11-20 04:03:42,272 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-87721429 failed! 2025-11-20 04:03:42,289 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 is 192.168.1.254 and allocation pool is NOT set 2025-11-20 04:03:46,373 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-11-20 04:03:46,373 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-87721429 failed! 2025-11-20 04:03:46,387 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 is 192.168.1.254 and allocation pool is NOT set 2025-11-20 04:03:50,460 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2025-11-20 04:03:50,460 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-87721429 failed! 2025-11-20 04:03:50,476 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 is 192.168.1.254 and allocation pool is NOT set 2025-11-20 04:03:54,560 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2025-11-20 04:03:54,560 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-87721429 failed! 2025-11-20 04:03:54,574 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 is 192.168.1.254 and allocation pool is NOT set 2025-11-20 04:03:58,656 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-11-20 04:03:58,657 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-87721429 failed! 2025-11-20 04:03:58,671 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 is 192.168.1.254 and allocation pool is NOT set 2025-11-20 04:04:02,755 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-11-20 04:04:02,755 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-87721429 failed! 2025-11-20 04:04:02,772 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 is 192.168.1.254 and allocation pool is NOT set 2025-11-20 04:04:06,848 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-11-20 04:04:06,849 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-87721429 failed! 2025-11-20 04:04:06,863 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 is 192.168.1.254 and allocation pool is NOT set 2025-11-20 04:04:10,943 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-11-20 04:04:10,944 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-87721429 failed! 2025-11-20 04:04:10,959 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 is 192.168.1.254 and allocation pool is NOT set 2025-11-20 04:04:15,040 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=6.18 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 1 received, 50% packet loss, time 1019ms\r\nrtt min/avg/max/mdev = 6.177/6.177/6.177/0.000 ms') 2025-11-20 04:04:15,040 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-87721429 passed 2025-11-20 04:04:15,040 - INFO - VM ctest-vn1_vm2-87721429 verifications in Compute nodes passed 2025-11-20 04:04:15,254 - DEBUG - Validated VM route 192.168.1.251 in vrouter of 10.0.0.48 2025-11-20 04:04:15,260 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-11-20 04:04:15,260 - INFO - Validated routes of VM ctest-vn1_vm2-87721429 in all vrouters 2025-11-20 04:04:15,687 - DEBUG - Starting all layer2 verification in 10.0.0.50 Control Node 2025-11-20 04:04:15,814 - DEBUG - Layer2 route found for VM MAC 02:51:d7:26:ac:09 in Control-node 10.0.0.50 2025-11-20 04:04:15,815 - DEBUG - L2 Label for VM ctest-vn1_vm2-87721429 same between Control-node 10.0.0.50 and Agent, Expected: 34, Seen: 34 2025-11-20 04:04:15,815 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2025-11-20 04:04:15,950 - DEBUG - Layer2 route found for VM MAC 02:51:d7:26:ac:09 in Control-node 10.0.0.23 2025-11-20 04:04:15,950 - DEBUG - L2 Label for VM ctest-vn1_vm2-87721429 same between Control-node 10.0.0.23 and Agent, Expected: 34, Seen: 34 2025-11-20 04:04:15,950 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-87721429 passed 2025-11-20 04:04:15,950 - DEBUG - Verifying the vm in opserver 2025-11-20 04:04:15,950 - DEBUG - Verifying in collector 10.0.0.22 ... 2025-11-20 04:04:15,950 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machine/78c20816-8477-4524-be45-3f39c6c1958c?flat 2025-11-20 04:04:16,037 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-17624131:51d726ac-093b-4d31-b0a6-79e13854daf7?flat 2025-11-20 04:04:16,060 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-17624131:51d726ac-093b-4d31-b0a6-79e13854daf7?flat 2025-11-20 04:04:16,080 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machine/78c20816-8477-4524-be45-3f39c6c1958c?flat 2025-11-20 04:04:16,164 - DEBUG - Verifying vm in vn uve 2025-11-20 04:04:16,165 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-17624131:51d726ac-093b-4d31-b0a6-79e13854daf7?flat 2025-11-20 04:04:16,195 - DEBUG - VM uve shows interface as {'vm_name': 'ctest-vn1_vm2-87721429', 'ip_address': '192.168.1.251', 'tx_vlan': 65535, 'admin_state': True, 'ip6_active': False, 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904', 'active': True, 'port_mirror_enabled': False, 'vhostuser_mode': 0, 'l2_active': True, 'gateway': '192.168.1.254', '__T': 1763611429033367, 'mac_address': '02:51:d7:26:ac:09', 'ip6_address': '::', 'is_health_check_active': True, 'rx_vlan': 65535, 'vm_uuid': '78c20816-8477-4524-be45-3f39c6c1958c', 'vn_uuid': 'de5b015d-d5f7-42fe-a691-8c1cdd7e7a17', 'label': 30, 'ip4_active': True, 'fixed_ip4_list': ['192.168.1.251'], 'uuid': '51d726ac-093b-4d31-b0a6-79e13854daf7'} 2025-11-20 04:04:16,195 - DEBUG - VM uve shows ip address as ['192.168.1.251', '::'] 2025-11-20 04:04:16,195 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 2025-11-20 04:04:16,195 - INFO - Verifying through opserver in 10.0.0.23 2025-11-20 04:04:16,196 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904?flat 2025-11-20 04:04:16,229 - DEBUG - expected vm list 78c20816-8477-4524-be45-3f39c6c1958c 2025-11-20 04:04:16,229 - DEBUG - Extracted vm list ['78c20816-8477-4524-be45-3f39c6c1958c'] 2025-11-20 04:04:16,229 - INFO - VM 78c20816-8477-4524-be45-3f39c6c1958c is present in default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 2025-11-20 04:04:16,230 - DEBUG - Verifying vm in vrouter uve 2025-11-20 04:04:16,230 - DEBUG - Getting info from collector 10.0.0.22.. 2025-11-20 04:04:16,230 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machine/78c20816-8477-4524-be45-3f39c6c1958c?flat 2025-11-20 04:04:16,249 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4768-2'] 2025-11-20 04:04:16,249 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-2:Compute:contrail-vrouter-agent:0?flat 2025-11-20 04:04:16,330 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-2?flat 2025-11-20 04:04:16,361 - DEBUG - VM 78c20816-8477-4524-be45-3f39c6c1958c is present in vrouter an-jenkins-deploy-platform-ansible-os-4768-2 uve 2025-11-20 04:04:16,361 - DEBUG - Validated that VM ctest-vn1_vm2-87721429 is in Vrouter an-jenkins-deploy-platform-ansible-os-4768-2 UVE 2025-11-20 04:04:16,361 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-11-20 04:04:16,368 - DEBUG - Expected tap interface of VM uuid 78c20816-8477-4524-be45-3f39c6c1958c is default-domain:ctest-TestBasicVMVN0-17624131:51d726ac-093b-4d31-b0a6-79e13854daf7 2025-11-20 04:04:16,368 - DEBUG - Expected VN of VM uuid 78c20816-8477-4524-be45-3f39c6c1958c is default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 2025-11-20 04:04:16,368 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-2:Compute:contrail-vrouter-agent:0?flat 2025-11-20 04:04:16,449 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-2?flat 2025-11-20 04:04:16,481 - DEBUG - VM 78c20816-8477-4524-be45-3f39c6c1958c is present in vrouter an-jenkins-deploy-platform-ansible-os-4768-2 uve 2025-11-20 04:04:16,481 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-17624131:51d726ac-093b-4d31-b0a6-79e13854daf7 of vm 78c20816-8477-4524-be45-3f39c6c1958c is present in vrouter an-jenkins-deploy-platform-ansible-os-4768-2 uve 2025-11-20 04:04:16,481 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 of vm 78c20816-8477-4524-be45-3f39c6c1958c is present in vrouter an-jenkins-deploy-platform-ansible-os-4768-2 uve 2025-11-20 04:04:16,481 - DEBUG - Verifying the 78c20816-8477-4524-be45-3f39c6c1958c virtual network link through opserver 10.0.0.23 2025-11-20 04:04:16,481 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-11-20 04:04:16,495 - DEBUG - VM link and name as {'name': '78c20816-8477-4524-be45-3f39c6c1958c', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/78c20816-8477-4524-be45-3f39c6c1958c?flat'} 2025-11-20 04:04:16,495 - DEBUG - Verifying the 78c20816-8477-4524-be45-3f39c6c1958c virtual network link through opserver 10.0.0.50 2025-11-20 04:04:16,495 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2025-11-20 04:04:16,579 - DEBUG - VM link and name as {'name': '78c20816-8477-4524-be45-3f39c6c1958c', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-machine/78c20816-8477-4524-be45-3f39c6c1958c?flat'} 2025-11-20 04:04:16,579 - DEBUG - Verifying the 78c20816-8477-4524-be45-3f39c6c1958c virtual network link through opserver 10.0.0.22 2025-11-20 04:04:16,579 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines 2025-11-20 04:04:16,592 - DEBUG - VM link and name as {'name': '78c20816-8477-4524-be45-3f39c6c1958c', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-machine/78c20816-8477-4524-be45-3f39c6c1958c?flat'} 2025-11-20 04:04:16,592 - INFO - VM ctest-vn1_vm2-87721429 validations in Opserver passed 2025-11-20 04:04:17,859 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4768-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4768-1) 2025-11-20 04:04:17,905 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-11-20 04:04:22,987 - DEBUG - VM is in ACTIVE state now 2025-11-20 04:04:22,987 - INFO - VM name : ctest-vn2_vm2-04645910 2025-11-20 04:04:23,066 - DEBUG - VM ctest-vn2_vm2-04645910 ID is 97ba6f70-7606-4e54-ba0b-69c55895f10c 2025-11-20 04:04:23,151 - DEBUG - VM is in ACTIVE state now 2025-11-20 04:04:23,151 - DEBUG - VM ctest-vn2_vm2-04645910 launched on Node an-jenkins-deploy-platform-ansible-os-4768-1 2025-11-20 04:04:23,229 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/97ba6f70-7606-4e54-ba0b-69c55895f10c 2025-11-20 04:04:23,242 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/97ba6f70-7606-4e54-ba0b-69c55895f10c 2025-11-20 04:04:23,253 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine-interface/0f82e5bb-5664-4154-94f7-cc7007d048f7 2025-11-20 04:04:23,264 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/97ba6f70-7606-4e54-ba0b-69c55895f10c 2025-11-20 04:04:23,272 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine-interface/0f82e5bb-5664-4154-94f7-cc7007d048f7 2025-11-20 04:04:23,283 - DEBUG - Requesting: http://10.0.0.22:8082/instance-ip/a59a3fa9-98b4-465f-87ed-cd4219f21e51 2025-11-20 04:04:23,294 - DEBUG - Verifying in api server 10.0.0.22 2025-11-20 04:04:23,294 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/97ba6f70-7606-4e54-ba0b-69c55895f10c 2025-11-20 04:04:23,302 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine-interface/0f82e5bb-5664-4154-94f7-cc7007d048f7 2025-11-20 04:04:23,313 - DEBUG - Requesting: http://10.0.0.22:8082/instance-ip/a59a3fa9-98b4-465f-87ed-cd4219f21e51 2025-11-20 04:04:23,322 - INFO - VM ctest-vn2_vm2-04645910 verfication in all API Servers passed 2025-11-20 04:04:23,341 - DEBUG - VM ctest-vn2_vm2-04645910 Tap interface: {'index': '4', 'name': 'tap0f82e5bb-56', 'uuid': '0f82e5bb-5664-4154-94f7-cc7007d048f7', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304:ctest-vn2-02050304', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '16', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304', 'vm_uuid': '97ba6f70-7606-4e54-ba0b-69c55895f10c', 'vm_name': 'ctest-vn2_vm2-04645910', 'ip_addr': '192.168.2.251', 'mac_addr': '02:0f:82:e5:bb:56', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '129', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-17624131:0f82e5bb-5664-4154-94f7-cc7007d048f7', 'sg_uuid_list': ['db81be13-9b62-47a3-a92b-985308dc1af8'], 'static_route_list': None, 'vm_project_uuid': '3339bf5d-318a-4edf-9cb8-4a7fd32a3af6', 'admin_state': 'Enabled', 'flow_key_idx': '35', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-11-20 04:04:23,341 - DEBUG - Agent 10.0.0.131 vrf name: default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304:ctest-vn2-02050304 2025-11-20 04:04:23,347 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304:ctest-vn2-02050304', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304', 'table_label': '-1', 'vxlan_id': '16', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.242:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-11-20 04:04:23,434 - DEBUG - VM ctest-vn2_vm2-04645910 : Tap interface tap0f82e5bb-56 is set to Active 2025-11-20 04:04:23,434 - DEBUG - Tap interface tap0f82e5bb-56 detail : {'index': '4', 'name': 'tap0f82e5bb-56', 'uuid': '0f82e5bb-5664-4154-94f7-cc7007d048f7', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304:ctest-vn2-02050304', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '16', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304', 'vm_uuid': '97ba6f70-7606-4e54-ba0b-69c55895f10c', 'vm_name': 'ctest-vn2_vm2-04645910', 'ip_addr': '192.168.2.251', 'mac_addr': '02:0f:82:e5:bb:56', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '129', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-17624131:0f82e5bb-5664-4154-94f7-cc7007d048f7', 'sg_uuid_list': ['db81be13-9b62-47a3-a92b-985308dc1af8'], 'static_route_list': None, 'vm_project_uuid': '3339bf5d-318a-4edf-9cb8-4a7fd32a3af6', 'admin_state': 'Enabled', 'flow_key_idx': '35', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-11-20 04:04:23,545 - DEBUG - Starting Layer 2 verification in Agent 2025-11-20 04:04:23,558 - DEBUG - Layer 2 path is seen for VM MAC 02:0f:82:e5:bb:56 in agent 10.0.0.131 2025-11-20 04:04:23,558 - DEBUG - Active layer 2 route in agent is present for VMI tap0f82e5bb-56 2025-11-20 04:04:23,558 - DEBUG - L2 label(34) matches bw route table and interface table 2025-11-20 04:04:23,566 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 is found in Agent of node 10.0.0.131 2025-11-20 04:04:23,700 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 is consistent in agent 10.0.0.131 2025-11-20 04:04:23,700 - DEBUG - Route for VM IP ['192.168.2.251'] is consistent in agent 10.0.0.131 2025-11-20 04:04:23,700 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 verification for VM ctest-vn2_vm2-04645910 in Agent 10.0.0.131 passed 2025-11-20 04:04:23,814 - DEBUG - Starting all layer 2 verification in agent 10.0.0.131 2025-11-20 04:04:23,830 - DEBUG - Route for VM MAC 02:0f:82:e5:bb:56 is consistent in agent 10.0.0.131 2025-11-20 04:04:27,023 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1008ms') 2025-11-20 04:04:27,024 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-04645910 failed! 2025-11-20 04:04:27,038 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 is 192.168.2.254 and allocation pool is NOT set 2025-11-20 04:04:31,122 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-11-20 04:04:31,123 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-04645910 failed! 2025-11-20 04:04:31,138 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 is 192.168.2.254 and allocation pool is NOT set 2025-11-20 04:04:35,215 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1012ms') 2025-11-20 04:04:35,215 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-04645910 failed! 2025-11-20 04:04:35,228 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 is 192.168.2.254 and allocation pool is NOT set 2025-11-20 04:04:39,310 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-11-20 04:04:39,311 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-04645910 failed! 2025-11-20 04:04:39,328 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 is 192.168.2.254 and allocation pool is NOT set 2025-11-20 04:04:43,410 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-11-20 04:04:43,410 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-04645910 failed! 2025-11-20 04:04:43,425 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 is 192.168.2.254 and allocation pool is NOT set 2025-11-20 04:04:47,506 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms') 2025-11-20 04:04:47,506 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-04645910 failed! 2025-11-20 04:04:47,524 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 is 192.168.2.254 and allocation pool is NOT set 2025-11-20 04:04:51,593 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1009ms') 2025-11-20 04:04:51,593 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-04645910 failed! 2025-11-20 04:04:51,608 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 is 192.168.2.254 and allocation pool is NOT set 2025-11-20 04:04:55,697 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1026ms') 2025-11-20 04:04:55,697 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-04645910 failed! 2025-11-20 04:04:55,712 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 is 192.168.2.254 and allocation pool is NOT set 2025-11-20 04:04:59,794 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-11-20 04:04:59,794 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-04645910 failed! 2025-11-20 04:04:59,808 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 is 192.168.2.254 and allocation pool is NOT set 2025-11-20 04:05:03,881 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=9.89 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 1 received, 50% packet loss, time 1011ms\r\nrtt min/avg/max/mdev = 9.890/9.890/9.890/0.000 ms') 2025-11-20 04:05:03,881 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-04645910 passed 2025-11-20 04:05:03,882 - INFO - VM ctest-vn2_vm2-04645910 verifications in Compute nodes passed 2025-11-20 04:05:04,114 - DEBUG - Validated VM route 192.168.2.251 in vrouter of 10.0.0.131 2025-11-20 04:05:04,128 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-11-20 04:05:04,128 - INFO - Validated routes of VM ctest-vn2_vm2-04645910 in all vrouters 2025-11-20 04:05:04,583 - DEBUG - Starting all layer2 verification in 10.0.0.50 Control Node 2025-11-20 04:05:04,713 - DEBUG - Layer2 route found for VM MAC 02:0f:82:e5:bb:56 in Control-node 10.0.0.50 2025-11-20 04:05:04,713 - DEBUG - L2 Label for VM ctest-vn2_vm2-04645910 same between Control-node 10.0.0.50 and Agent, Expected: 34, Seen: 34 2025-11-20 04:05:04,713 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2025-11-20 04:05:04,841 - DEBUG - Layer2 route found for VM MAC 02:0f:82:e5:bb:56 in Control-node 10.0.0.23 2025-11-20 04:05:04,841 - DEBUG - L2 Label for VM ctest-vn2_vm2-04645910 same between Control-node 10.0.0.23 and Agent, Expected: 34, Seen: 34 2025-11-20 04:05:04,841 - INFO - Verification in Control-nodes for VM ctest-vn2_vm2-04645910 passed 2025-11-20 04:05:04,842 - DEBUG - Verifying the vm in opserver 2025-11-20 04:05:04,842 - DEBUG - Verifying in collector 10.0.0.22 ... 2025-11-20 04:05:04,842 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machine/97ba6f70-7606-4e54-ba0b-69c55895f10c?flat 2025-11-20 04:05:04,857 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-17624131:0f82e5bb-5664-4154-94f7-cc7007d048f7?flat 2025-11-20 04:05:04,878 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-17624131:0f82e5bb-5664-4154-94f7-cc7007d048f7?flat 2025-11-20 04:05:04,896 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machine/97ba6f70-7606-4e54-ba0b-69c55895f10c?flat 2025-11-20 04:05:04,912 - DEBUG - Verifying vm in vn uve 2025-11-20 04:05:04,912 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-17624131:0f82e5bb-5664-4154-94f7-cc7007d048f7?flat 2025-11-20 04:05:04,932 - DEBUG - VM uve shows interface as {'l2_active': True, 'ip6_active': False, 'mac_address': '02:0f:82:e5:bb:56', 'vm_name': 'ctest-vn2_vm2-04645910', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304', 'vhostuser_mode': 0, 'label': 30, 'rx_vlan': 65535, 'uuid': '0f82e5bb-5664-4154-94f7-cc7007d048f7', 'ip6_address': '::', 'is_health_check_active': True, 'ip_address': '192.168.2.251', 'admin_state': True, 'port_mirror_enabled': False, 'tx_vlan': 65535, '__T': 1763611480586156, 'active': True, 'vm_uuid': '97ba6f70-7606-4e54-ba0b-69c55895f10c', 'ip4_active': True, 'vn_uuid': '48185bc6-b2d0-44ad-9dad-8abd34951e77', 'gateway': '192.168.2.254', 'fixed_ip4_list': ['192.168.2.251']} 2025-11-20 04:05:04,933 - DEBUG - VM uve shows ip address as ['192.168.2.251', '::'] 2025-11-20 04:05:04,933 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 2025-11-20 04:05:04,933 - INFO - Verifying through opserver in 10.0.0.23 2025-11-20 04:05:04,933 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304?flat 2025-11-20 04:05:04,957 - DEBUG - expected vm list 97ba6f70-7606-4e54-ba0b-69c55895f10c 2025-11-20 04:05:04,957 - DEBUG - Extracted vm list ['97ba6f70-7606-4e54-ba0b-69c55895f10c'] 2025-11-20 04:05:04,957 - INFO - VM 97ba6f70-7606-4e54-ba0b-69c55895f10c is present in default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 2025-11-20 04:05:04,957 - DEBUG - Verifying vm in vrouter uve 2025-11-20 04:05:04,957 - DEBUG - Getting info from collector 10.0.0.22.. 2025-11-20 04:05:04,957 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machine/97ba6f70-7606-4e54-ba0b-69c55895f10c?flat 2025-11-20 04:05:04,975 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4768-1'] 2025-11-20 04:05:04,976 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-1:Compute:contrail-vrouter-agent:0?flat 2025-11-20 04:05:05,019 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-1?flat 2025-11-20 04:05:05,047 - DEBUG - VM 97ba6f70-7606-4e54-ba0b-69c55895f10c is present in vrouter an-jenkins-deploy-platform-ansible-os-4768-1 uve 2025-11-20 04:05:05,048 - DEBUG - Validated that VM ctest-vn2_vm2-04645910 is in Vrouter an-jenkins-deploy-platform-ansible-os-4768-1 UVE 2025-11-20 04:05:05,048 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-11-20 04:05:05,056 - DEBUG - Expected tap interface of VM uuid 97ba6f70-7606-4e54-ba0b-69c55895f10c is default-domain:ctest-TestBasicVMVN0-17624131:0f82e5bb-5664-4154-94f7-cc7007d048f7 2025-11-20 04:05:05,056 - DEBUG - Expected VN of VM uuid 97ba6f70-7606-4e54-ba0b-69c55895f10c is default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 2025-11-20 04:05:05,056 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-1:Compute:contrail-vrouter-agent:0?flat 2025-11-20 04:05:05,113 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-1?flat 2025-11-20 04:05:05,154 - DEBUG - VM 97ba6f70-7606-4e54-ba0b-69c55895f10c is present in vrouter an-jenkins-deploy-platform-ansible-os-4768-1 uve 2025-11-20 04:05:05,154 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-17624131:0f82e5bb-5664-4154-94f7-cc7007d048f7 of vm 97ba6f70-7606-4e54-ba0b-69c55895f10c is present in vrouter an-jenkins-deploy-platform-ansible-os-4768-1 uve 2025-11-20 04:05:05,154 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 of vm 97ba6f70-7606-4e54-ba0b-69c55895f10c is present in vrouter an-jenkins-deploy-platform-ansible-os-4768-1 uve 2025-11-20 04:05:05,154 - DEBUG - Verifying the 97ba6f70-7606-4e54-ba0b-69c55895f10c virtual network link through opserver 10.0.0.23 2025-11-20 04:05:05,155 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-11-20 04:05:05,168 - DEBUG - VM link and name as {'name': '97ba6f70-7606-4e54-ba0b-69c55895f10c', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/97ba6f70-7606-4e54-ba0b-69c55895f10c?flat'} 2025-11-20 04:05:05,168 - DEBUG - Verifying the 97ba6f70-7606-4e54-ba0b-69c55895f10c virtual network link through opserver 10.0.0.50 2025-11-20 04:05:05,168 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2025-11-20 04:05:05,186 - DEBUG - VM link and name as {'name': '97ba6f70-7606-4e54-ba0b-69c55895f10c', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-machine/97ba6f70-7606-4e54-ba0b-69c55895f10c?flat'} 2025-11-20 04:05:05,186 - DEBUG - Verifying the 97ba6f70-7606-4e54-ba0b-69c55895f10c virtual network link through opserver 10.0.0.22 2025-11-20 04:05:05,186 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines 2025-11-20 04:05:05,200 - DEBUG - VM link and name as {'name': '97ba6f70-7606-4e54-ba0b-69c55895f10c', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-machine/97ba6f70-7606-4e54-ba0b-69c55895f10c?flat'} 2025-11-20 04:05:05,200 - INFO - VM ctest-vn2_vm2-04645910 validations in Opserver passed 2025-11-20 04:05:05,200 - INFO - Waiting for VM ctest-vn1_vm2-87721429 to be up.. 2025-11-20 04:05:05,200 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:05:05,200 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/78c20816-8477-4524-be45-3f39c6c1958c 2025-11-20 04:05:05,209 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine-interface/51d726ac-093b-4d31-b0a6-79e13854daf7 2025-11-20 04:05:06,394 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=3.35 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.651 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.651/1.998/3.346/1.347 ms') 2025-11-20 04:05:06,394 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-87721429 passed 2025-11-20 04:05:06,466 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:05:06,466 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-87721429, IP 192.168.1.251, Port 22 2025-11-20 04:05:06,634 - DEBUG - VM ctest-vn1_vm2-87721429 is ready for SSH connections 2025-11-20 04:05:06,634 - INFO - Waiting for VM ctest-vn2_vm2-04645910 to be up.. 2025-11-20 04:05:06,634 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:05:06,634 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/97ba6f70-7606-4e54-ba0b-69c55895f10c 2025-11-20 04:05:06,644 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine-interface/0f82e5bb-5664-4154-94f7-cc7007d048f7 2025-11-20 04:05:07,815 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=5.30 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=1.40 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 1.404/3.350/5.296/1.946 ms') 2025-11-20 04:05:07,815 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-04645910 passed 2025-11-20 04:05:07,885 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:05:07,885 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-04645910, IP 192.168.2.251, Port 22 2025-11-20 04:05:07,950 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 04:05:08,032 - DEBUG - VM ctest-vn2_vm2-04645910 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 04:05:13,033 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:05:13,034 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-04645910, IP 192.168.2.251, Port 22 2025-11-20 04:05:13,098 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 04:05:13,177 - DEBUG - VM ctest-vn2_vm2-04645910 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 04:05:18,178 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:05:18,178 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-04645910, IP 192.168.2.251, Port 22 2025-11-20 04:05:18,246 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-20 04:05:18,335 - DEBUG - VM ctest-vn2_vm2-04645910 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-20 04:05:23,335 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:05:23,336 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-04645910, IP 192.168.2.251, Port 22 2025-11-20 04:05:23,515 - DEBUG - VM ctest-vn2_vm2-04645910 is ready for SSH connections 2025-11-20 04:05:23,515 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-20 04:05:23,515 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.251, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.48, gateway password: c0ntrail123 2025-11-20 04:05:23,515 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.251 2025-11-20 04:06:04,703 - DEBUG - PING 192.168.2.251 (192.168.2.251) 56(84) bytes of data. 64 bytes from 192.168.2.251: icmp_req=1 ttl=63 time=9.44 ms 64 bytes from 192.168.2.251: icmp_req=2 ttl=63 time=1.30 ms 64 bytes from 192.168.2.251: icmp_req=3 ttl=63 time=1.71 ms --- 192.168.2.251 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.304/4.154/9.444/3.744 ms 2025-11-20 04:06:04,704 - INFO - Ping to IP 192.168.2.251 from VM ctest-vn1_vm2-87721429 passed 2025-11-20 04:06:04,704 - INFO - Deleting VM ctest-vn2_vm2-04645910 2025-11-20 04:06:04,889 - DEBUG - Verifying in api server 10.0.0.22 2025-11-20 04:06:04,889 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/97ba6f70-7606-4e54-ba0b-69c55895f10c 2025-11-20 04:06:04,899 - DEBUG - VM ID 97ba6f70-7606-4e54-ba0b-69c55895f10c of VM ctest-vn2_vm2-04645910 is still found in API Server 2025-11-20 04:06:06,900 - DEBUG - Verifying in api server 10.0.0.22 2025-11-20 04:06:06,900 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/97ba6f70-7606-4e54-ba0b-69c55895f10c 2025-11-20 04:06:06,907 - DEBUG - Response Code: 404 2025-11-20 04:06:06,908 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/97ba6f70-7606-4e54-ba0b-69c55895f10c 2025-11-20 04:06:06,914 - DEBUG - Response Code: 404 2025-11-20 04:06:06,914 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/97ba6f70-7606-4e54-ba0b-69c55895f10c 2025-11-20 04:06:06,920 - DEBUG - Response Code: 404 2025-11-20 04:06:06,921 - INFO - VM ctest-vn2_vm2-04645910 is fully removed in API-Server 2025-11-20 04:06:07,076 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-11-20 04:06:07,257 - DEBUG - VM route 192.168.2.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 of compute 10.0.0.131 2025-11-20 04:06:07,264 - DEBUG - Validated that vrouter 10.0.0.131 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 2025-11-20 04:06:07,264 - DEBUG - Validated that vrouter 10.0.0.48 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 2025-11-20 04:06:07,264 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 2025-11-20 04:06:07,264 - INFO - VM ctest-vn2_vm2-04645910 is removed in Compute, and routes are removed in all compute nodes 2025-11-20 04:06:07,488 - INFO - Routes for VM ctest-vn2_vm2-04645910 is removed in all control-nodes 2025-11-20 04:06:07,488 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-1:Compute:contrail-vrouter-agent:0?flat 2025-11-20 04:06:07,540 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-1?flat 2025-11-20 04:06:07,568 - DEBUG - VM 97ba6f70-7606-4e54-ba0b-69c55895f10c is still present in vrouter an-jenkins-deploy-platform-ansible-os-4768-1 uve 2025-11-20 04:06:07,568 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-11-20 04:06:07,568 - DEBUG - interface for vm 97ba6f70-7606-4e54-ba0b-69c55895f10c deleted from vrouter uve ... 2025-11-20 04:06:07,568 - DEBUG - Deleted interface not in error interface list ... 2025-11-20 04:06:10,569 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-1:Compute:contrail-vrouter-agent:0?flat 2025-11-20 04:06:10,617 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-1?flat 2025-11-20 04:06:10,645 - DEBUG - VM 97ba6f70-7606-4e54-ba0b-69c55895f10c is not present in vrouter an-jenkins-deploy-platform-ansible-os-4768-1 uve 2025-11-20 04:06:10,645 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-11-20 04:06:10,645 - DEBUG - interface for vm 97ba6f70-7606-4e54-ba0b-69c55895f10c deleted from vrouter uve ... 2025-11-20 04:06:10,645 - DEBUG - Deleted interface not in error interface list ... 2025-11-20 04:06:10,645 - DEBUG - Validated that VM 97ba6f70-7606-4e54-ba0b-69c55895f10c is removed in Vrouter UVE 2025-11-20 04:06:10,645 - DEBUG - Verifying through opserver in 10.0.0.23 2025-11-20 04:06:10,646 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304?flat 2025-11-20 04:06:10,696 - DEBUG - 97ba6f70-7606-4e54-ba0b-69c55895f10c BM not in default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn2-02050304 uve 2025-11-20 04:06:10,696 - DEBUG - Validated that VM 97ba6f70-7606-4e54-ba0b-69c55895f10c is not present in the VN UVE 2025-11-20 04:06:10,696 - DEBUG - Verifying the 97ba6f70-7606-4e54-ba0b-69c55895f10c virtual network link through opserver 10.0.0.23 2025-11-20 04:06:10,696 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-11-20 04:06:10,712 - INFO - 97ba6f70-7606-4e54-ba0b-69c55895f10c vm uve deleted from opserver 2025-11-20 04:06:10,712 - DEBUG - Verifying the 97ba6f70-7606-4e54-ba0b-69c55895f10c virtual network link through opserver 10.0.0.50 2025-11-20 04:06:10,712 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2025-11-20 04:06:10,730 - INFO - 97ba6f70-7606-4e54-ba0b-69c55895f10c vm uve deleted from opserver 2025-11-20 04:06:10,730 - DEBUG - Verifying the 97ba6f70-7606-4e54-ba0b-69c55895f10c virtual network link through opserver 10.0.0.22 2025-11-20 04:06:10,730 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines 2025-11-20 04:06:10,743 - INFO - 97ba6f70-7606-4e54-ba0b-69c55895f10c vm uve deleted from opserver 2025-11-20 04:06:10,743 - INFO - Validated that VM 97ba6f70-7606-4e54-ba0b-69c55895f10c is removed from Opserver 2025-11-20 04:06:10,743 - INFO - Deleting VM ctest-vn1_vm2-87721429 2025-11-20 04:06:10,829 - DEBUG - Verifying in api server 10.0.0.22 2025-11-20 04:06:10,829 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/78c20816-8477-4524-be45-3f39c6c1958c 2025-11-20 04:06:10,837 - DEBUG - VM ID 78c20816-8477-4524-be45-3f39c6c1958c of VM ctest-vn1_vm2-87721429 is still found in API Server 2025-11-20 04:06:12,838 - DEBUG - Verifying in api server 10.0.0.22 2025-11-20 04:06:12,838 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/78c20816-8477-4524-be45-3f39c6c1958c 2025-11-20 04:06:12,845 - DEBUG - Response Code: 404 2025-11-20 04:06:12,846 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/78c20816-8477-4524-be45-3f39c6c1958c 2025-11-20 04:06:12,852 - DEBUG - Response Code: 404 2025-11-20 04:06:12,853 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-machine/78c20816-8477-4524-be45-3f39c6c1958c 2025-11-20 04:06:12,858 - DEBUG - Response Code: 404 2025-11-20 04:06:12,858 - INFO - VM ctest-vn1_vm2-87721429 is fully removed in API-Server 2025-11-20 04:06:13,016 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-11-20 04:06:13,049 - DEBUG - Validated that vrouter 10.0.0.131 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 2025-11-20 04:06:13,189 - DEBUG - VM route 192.168.1.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 of compute 10.0.0.48 2025-11-20 04:06:13,196 - DEBUG - Validated that vrouter 10.0.0.48 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 2025-11-20 04:06:13,196 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 2025-11-20 04:06:13,196 - INFO - VM ctest-vn1_vm2-87721429 is removed in Compute, and routes are removed in all compute nodes 2025-11-20 04:06:13,508 - INFO - Routes for VM ctest-vn1_vm2-87721429 is removed in all control-nodes 2025-11-20 04:06:13,509 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-2:Compute:contrail-vrouter-agent:0?flat 2025-11-20 04:06:13,583 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-2?flat 2025-11-20 04:06:13,614 - DEBUG - VM 78c20816-8477-4524-be45-3f39c6c1958c is still present in vrouter an-jenkins-deploy-platform-ansible-os-4768-2 uve 2025-11-20 04:06:13,614 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-11-20 04:06:13,614 - DEBUG - interface for vm 78c20816-8477-4524-be45-3f39c6c1958c deleted from vrouter uve ... 2025-11-20 04:06:13,614 - DEBUG - Deleted interface not in error interface list ... 2025-11-20 04:06:16,615 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-2:Compute:contrail-vrouter-agent:0?flat 2025-11-20 04:06:16,685 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-2?flat 2025-11-20 04:06:16,722 - DEBUG - VM 78c20816-8477-4524-be45-3f39c6c1958c is still present in vrouter an-jenkins-deploy-platform-ansible-os-4768-2 uve 2025-11-20 04:06:16,722 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-11-20 04:06:16,722 - DEBUG - interface for vm 78c20816-8477-4524-be45-3f39c6c1958c deleted from vrouter uve ... 2025-11-20 04:06:16,722 - DEBUG - Deleted interface not in error interface list ... 2025-11-20 04:06:19,723 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4768-2:Compute:contrail-vrouter-agent:0?flat 2025-11-20 04:06:19,781 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4768-2?flat 2025-11-20 04:06:19,813 - DEBUG - VM 78c20816-8477-4524-be45-3f39c6c1958c is not present in vrouter an-jenkins-deploy-platform-ansible-os-4768-2 uve 2025-11-20 04:06:19,814 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-11-20 04:06:19,814 - DEBUG - interface for vm 78c20816-8477-4524-be45-3f39c6c1958c deleted from vrouter uve ... 2025-11-20 04:06:19,814 - DEBUG - Deleted interface not in error interface list ... 2025-11-20 04:06:19,814 - DEBUG - Validated that VM 78c20816-8477-4524-be45-3f39c6c1958c is removed in Vrouter UVE 2025-11-20 04:06:19,814 - DEBUG - Verifying through opserver in 10.0.0.23 2025-11-20 04:06:19,814 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904?flat 2025-11-20 04:06:19,838 - DEBUG - 78c20816-8477-4524-be45-3f39c6c1958c BM not in default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 uve 2025-11-20 04:06:19,838 - DEBUG - Validated that VM 78c20816-8477-4524-be45-3f39c6c1958c is not present in the VN UVE 2025-11-20 04:06:19,838 - DEBUG - Verifying the 78c20816-8477-4524-be45-3f39c6c1958c virtual network link through opserver 10.0.0.23 2025-11-20 04:06:19,838 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-11-20 04:06:19,850 - INFO - 78c20816-8477-4524-be45-3f39c6c1958c vm uve deleted from opserver 2025-11-20 04:06:19,850 - DEBUG - Verifying the 78c20816-8477-4524-be45-3f39c6c1958c virtual network link through opserver 10.0.0.50 2025-11-20 04:06:19,850 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2025-11-20 04:06:19,864 - INFO - 78c20816-8477-4524-be45-3f39c6c1958c vm uve deleted from opserver 2025-11-20 04:06:19,864 - DEBUG - Verifying the 78c20816-8477-4524-be45-3f39c6c1958c virtual network link through opserver 10.0.0.22 2025-11-20 04:06:19,864 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines 2025-11-20 04:06:19,876 - INFO - 78c20816-8477-4524-be45-3f39c6c1958c vm uve deleted from opserver 2025-11-20 04:06:19,876 - INFO - Validated that VM 78c20816-8477-4524-be45-3f39c6c1958c is removed from Opserver 2025-11-20 04:06:19,877 - INFO - Deleting VN ctest-vn3-21173747 2025-11-20 04:06:19,995 - INFO - Deleting VM ctest-vn2_vm1-40788747 2025-11-20 04:06:20,086 - INFO - Deleting VM ctest-vn1_vm1-06994196 2025-11-20 04:06:20,338 - INFO - Deleting VN ctest-vn2-02050304 2025-11-20 04:06:20,392 - DEBUG - RefsExistError Delete when resource still referred: ['http://10.0.0.22:8082/virtual-machine-interface/987eb33b-d1d5-4a23-8a9d-6ebad070c276'] while deleting VN ctest-vn2-02050304..Will retry 2025-11-20 04:06:22,528 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/7c46d3be-8510-44b3-b6e5-5c50faeefb6f 2025-11-20 04:06:22,535 - DEBUG - Response Code: 404 2025-11-20 04:06:22,535 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-11-20 04:06:22,541 - DEBUG - Requesting: http://10.0.0.23:8082/domain/785d7fdc-4b61-45a1-9add-76a6d523389b 2025-11-20 04:06:22,556 - DEBUG - Requesting: http://10.0.0.23:8082/project/3339bf5d-318a-4edf-9cb8-4a7fd32a3af6 2025-11-20 04:06:22,639 - INFO - Validated that VN ctest-vn2-02050304 is not found in API Server 2025-11-20 04:06:22,649 - DEBUG - VN ctest-vn2-02050304 is not present in Agent 10.0.0.131 2025-11-20 04:06:22,662 - DEBUG - VN ctest-vn2-02050304 is not present in Agent 10.0.0.48 2025-11-20 04:06:22,662 - INFO - Validated that VN ctest-vn2-02050304 is not in any agent 2025-11-20 04:06:22,693 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn2-02050304 info 2025-11-20 04:06:22,846 - INFO - Deleting VN ctest-vn1-60419904 2025-11-20 04:06:22,968 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/46baa226-2fa2-4a9d-98af-eceff3d49ff5 2025-11-20 04:06:22,973 - DEBUG - Response Code: 404 2025-11-20 04:06:22,973 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-11-20 04:06:22,983 - DEBUG - Requesting: http://10.0.0.23:8082/domain/785d7fdc-4b61-45a1-9add-76a6d523389b 2025-11-20 04:06:22,995 - DEBUG - Requesting: http://10.0.0.23:8082/project/3339bf5d-318a-4edf-9cb8-4a7fd32a3af6 2025-11-20 04:06:23,075 - INFO - Validated that VN ctest-vn1-60419904 is not found in API Server 2025-11-20 04:06:23,090 - DEBUG - VN ctest-vn1-60419904 is not present in Agent 10.0.0.131 2025-11-20 04:06:23,102 - DEBUG - VN ctest-vn1-60419904 is not present in Agent 10.0.0.48 2025-11-20 04:06:23,102 - INFO - Validated that VN ctest-vn1-60419904 is not in any agent 2025-11-20 04:06:23,108 - DEBUG - VRF 2 is not seen in agent 10.0.0.48 2025-11-20 04:06:23,112 - DEBUG - Vrouter 10.0.0.48 does not have vrf 2 for VN ctest-vn1-60419904 2025-11-20 04:06:23,112 - INFO - Validated that all vrouters do not have the route table for VN default-domain:ctest-TestBasicVMVN0-17624131:ctest-vn1-60419904 2025-11-20 04:06:23,133 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-60419904 info 2025-11-20 04:06:23,170 - INFO - Deleted policy policy2 2025-11-20 04:06:23,206 - INFO - Deleted policy policy1 2025-11-20 04:06:24,465 - DEBUG - Skipping xmpp flap check 2025-11-20 04:06:24,465 - INFO - END TEST : test_process_restart_in_policy_between_vns : PASSED[0:07:39] 2025-11-20 04:06:24,465 - INFO - -------------------------------------------------------------------------------- 2025-11-20 04:06:24,467 - INFO - ================================================================================ 2025-11-20 04:06:24,467 - INFO - STARTING TEST : test_underlay_broadcast_traffic_handling 2025-11-20 04:06:24,468 - INFO - TEST DESCRIPTION : Test the underlay brocast traffic handling by vrouter. (Bug-1545229). 1. Send broadcast traffic from one compute node. 2. Other compute in same subnet should receive that traffic. 3. Receiving compute should treat this traffic as underlay. 4. Compute should not replicate the packet and send the copy back. Pass criteria: Step 3-4 should pass Maintainer : chhandak@juniper.net 2025-11-20 04:06:25,750 - DEBUG - Skipping xmpp flap check 2025-11-20 04:06:25,750 - INFO - Initial checks done. Running the testcase now 2025-11-20 04:06:25,750 - INFO - 2025-11-20 04:06:26,144 - INFO - Agent interface name: ens6 2025-11-20 04:06:26,226 - DEBUG - Executing command: sudo tcpdump -nni ens6 -U host 10.20.0.255 -w /tmp/ens6_ctest-random-44428696.pcap 2025-11-20 04:06:31,591 - INFO - Pinging broacast address 10.20.0.255 from compute 10.20.0.242 2025-11-20 04:06:45,880 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-11-20 04:06:48,018 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-44428696.pcap | grep -c length 2025-11-20 04:06:48,033 - DEBUG - STDOUT: 10 2025-11-20 04:06:48,033 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-44428696.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-11-20 04:06:48,033 - INFO - 10 packets are found in tcpdump output as expected 2025-11-20 04:06:48,033 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-11-20 04:06:50,078 - INFO - Packet count matched: Compute 10.0.0.48 has receive only 10 packet from source IP 10.20.0.255. No duplicate packet seen 2025-11-20 04:06:51,372 - DEBUG - Skipping xmpp flap check 2025-11-20 04:06:51,373 - INFO - END TEST : test_underlay_broadcast_traffic_handling : PASSED[0:00:27] 2025-11-20 04:06:51,373 - INFO - -------------------------------------------------------------------------------- 2025-11-20 04:06:52,102 - INFO - Deleted project: ctest-TestBasicVMVN0-17624131, ID : 3339bf5d-318a-4edf-9cb8-4a7fd32a3af6