2026-01-19 04:06:13,103 - INFO - Domain Default found not creating 2026-01-19 04:06:13,266 - INFO - Project ctest-TestBasicVMVN0-83017551 not found, creating it 2026-01-19 04:06:13,894 - INFO - Created Project:ctest-TestBasicVMVN0-83017551, ID : 7b723255-fa49-4769-b023-38f28f67d984 2026-01-19 04:06:15,443 - INFO - ================================================================================ 2026-01-19 04:06:15,443 - INFO - STARTING TEST : test_control_node_switchover 2026-01-19 04:06:15,443 - INFO - TEST DESCRIPTION : Stop the control node and check peering with agent fallback to other control node. 1. Pick one VN from respource pool which has 2 VM's in it 2. Verify ping between VM's 3. Find active control node in cluster by agent inspect 4. Stop control service on active control node 5. Verify agents are connected to new active control-node using xmpp connections 6. Bring back control service on previous active node 7. Verify ping between VM's again after bringing up control serveice Pass criteria: Step 2,5 and 7 should pass 2026-01-19 04:06:16,692 - DEBUG - Skipping xmpp flap check 2026-01-19 04:06:16,692 - INFO - Initial checks done. Running the testcase now 2026-01-19 04:06:16,692 - INFO - 2026-01-19 04:06:17,386 - DEBUG - Response for create_network : {'network': {'id': 'c7d84faf-aaf5-420b-b599-5e46fe5337c8', 'name': 'ctest-vn1-37701351', 'tenant_id': '7b723255fa494769b02338f28f67d984', 'project_id': '7b723255fa494769b02338f28f67d984', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestBasicVMVN0-83017551', 'ctest-vn1-37701351'], 'port_security_enabled': True, 'description': ''}} 2026-01-19 04:06:17,689 - DEBUG - Response for create_subnet : {'subnet': {'id': '64fdcb80-df23-485e-8e0d-9609c2850785', 'name': '', 'tenant_id': '7b723255fa494769b02338f28f67d984', 'network_id': 'c7d84faf-aaf5-420b-b599-5e46fe5337c8', 'ip_version': 4, 'cidr': '192.168.1.0/24', 'allocation_pools': [{'start': '192.168.1.2', 'end': '192.168.1.254'}], 'gateway_ip': '192.168.1.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '192.168.1.2', 'tags': [], 'project_id': '7b723255fa494769b02338f28f67d984'}} 2026-01-19 04:06:17,715 - INFO - Created VN ctest-vn1-37701351 2026-01-19 04:06:17,729 - DEBUG - VN ctest-vn1-37701351 UUID is c7d84faf-aaf5-420b-b599-5e46fe5337c8 2026-01-19 04:06:17,822 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-19 04:06:18,126 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-19 04:06:18,161 - DEBUG - Requesting: http://10.0.0.27:8082/domain/bc123c74-c45e-4d7a-a955-76f59d7a4cc7 2026-01-19 04:06:18,175 - DEBUG - Requesting: http://10.0.0.27:8082/project/7b723255-fa49-4769-b023-38f28f67d984 2026-01-19 04:06:18,271 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/c7d84faf-aaf5-420b-b599-5e46fe5337c8 2026-01-19 04:06:18,288 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/c7d84faf-aaf5-420b-b599-5e46fe5337c8 2026-01-19 04:06:18,300 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/ceb01fb1-9d6b-487c-afda-4795c923304d 2026-01-19 04:06:18,312 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/ceb01fb1-9d6b-487c-afda-4795c923304d 2026-01-19 04:06:18,321 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/952c210f-854d-4d36-83b4-f2112f5523d9 2026-01-19 04:06:18,329 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-19 04:06:18,329 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/c7d84faf-aaf5-420b-b599-5e46fe5337c8 2026-01-19 04:06:18,341 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/ceb01fb1-9d6b-487c-afda-4795c923304d 2026-01-19 04:06:18,361 - INFO - Verified VN network id 16 for VN c7d84faf-aaf5-420b-b599-5e46fe5337c8 2026-01-19 04:06:18,361 - INFO - Verifications in API Server for VN ctest-vn1-37701351 passed 2026-01-19 04:06:18,361 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/c7d84faf-aaf5-420b-b599-5e46fe5337c8 2026-01-19 04:06:18,374 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/ceb01fb1-9d6b-487c-afda-4795c923304d 2026-01-19 04:06:18,382 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/ceb01fb1-9d6b-487c-afda-4795c923304d 2026-01-19 04:06:18,388 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/952c210f-854d-4d36-83b4-f2112f5523d9 2026-01-19 04:06:18,401 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-83017551', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '14400347424423232011', 'uuid-lslong': '13085593851309275080'}, 'enable': 'true', 'created': '2026-01-19T04:06:17', 'last-modified': '2026-01-19T04:06:17', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.764456'} 2026-01-19 04:06:18,410 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-19 04:06:18,416 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-83017551', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '14400347424423232011', 'uuid-lslong': '13085593851309275080'}, 'enable': 'true', 'created': '2026-01-19T04:06:17', 'last-modified': '2026-01-19T04:06:17', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.774936'} 2026-01-19 04:06:18,422 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-19 04:06:18,428 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-83017551', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '14400347424423232011', 'uuid-lslong': '13085593851309275080'}, 'enable': 'true', 'created': '2026-01-19T04:06:17', 'last-modified': '2026-01-19T04:06:17', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.790636'} 2026-01-19 04:06:18,433 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-19 04:06:18,434 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-37701351 passed 2026-01-19 04:06:18,434 - DEBUG - ====Verifying policy data for ctest-vn1-37701351 in API_Server ====== 2026-01-19 04:06:18,434 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-19 04:06:18,441 - DEBUG - Requesting: http://10.0.0.27:8082/domain/bc123c74-c45e-4d7a-a955-76f59d7a4cc7 2026-01-19 04:06:18,453 - DEBUG - Requesting: http://10.0.0.27:8082/project/7b723255-fa49-4769-b023-38f28f67d984 2026-01-19 04:06:18,537 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/c7d84faf-aaf5-420b-b599-5e46fe5337c8 2026-01-19 04:06:18,553 - DEBUG - =>VN ctest-vn1-37701351 has no policy to be verified 2026-01-19 04:06:18,553 - DEBUG - Verifying the vn in opserver 2026-01-19 04:06:18,553 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 virtual network link through opserver 10.0.0.27 2026-01-19 04:06:18,553 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2026-01-19 04:06:18,863 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2026-01-19 04:06:18,930 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351?flat'} 2026-01-19 04:06:18,930 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 virtual network link through opserver 10.0.0.50 2026-01-19 04:06:18,930 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2026-01-19 04:06:19,233 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2026-01-19 04:06:19,294 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351?flat'} 2026-01-19 04:06:19,295 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 virtual network link through opserver 10.0.0.23 2026-01-19 04:06:19,295 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2026-01-19 04:06:19,597 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2026-01-19 04:06:19,659 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351?flat'} 2026-01-19 04:06:19,659 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 is found in opserver 2026-01-19 04:06:19,679 - DEBUG - Do not have enough data to verify VN in agent 2026-01-19 04:06:19,688 - DEBUG - VRF ids for VN ctest-vn1-37701351: {} 2026-01-19 04:06:19,748 - DEBUG - Services list from nova: [, , , ] 2026-01-19 04:06:20,813 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-5040-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-5040-1) 2026-01-19 04:06:22,163 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-5040-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-5040-2) 2026-01-19 04:06:22,163 - INFO - Waiting for VM ctest-vn1_vm1-67069065 to be up.. 2026-01-19 04:06:22,265 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2026-01-19 04:06:27,356 - DEBUG - VM is in ACTIVE state now 2026-01-19 04:06:27,356 - INFO - VM name : ctest-vn1_vm1-67069065 2026-01-19 04:06:27,440 - DEBUG - VM ctest-vn1_vm1-67069065 ID is c09da990-2c0f-405b-a690-309bea13b193 2026-01-19 04:06:27,464 - DEBUG - VM ctest-vn1_vm1-67069065 launched on Node an-jenkins-deploy-platform-ansible-os-5040-1 2026-01-19 04:06:27,543 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/c09da990-2c0f-405b-a690-309bea13b193 2026-01-19 04:06:27,846 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/c09da990-2c0f-405b-a690-309bea13b193 2026-01-19 04:06:27,894 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/c0e0fe75-a8c5-4368-9cb2-056118eddd0b 2026-01-19 04:06:31,085 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1026ms') 2026-01-19 04:06:31,085 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67069065 failed! 2026-01-19 04:06:31,100 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 is 192.168.1.1 and allocation pool is NOT set 2026-01-19 04:06:35,170 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2026-01-19 04:06:35,170 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67069065 failed! 2026-01-19 04:06:35,186 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 is 192.168.1.1 and allocation pool is NOT set 2026-01-19 04:06:39,272 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2026-01-19 04:06:39,272 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67069065 failed! 2026-01-19 04:06:39,289 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 is 192.168.1.1 and allocation pool is NOT set 2026-01-19 04:06:43,366 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2026-01-19 04:06:43,366 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67069065 failed! 2026-01-19 04:06:43,381 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 is 192.168.1.1 and allocation pool is NOT set 2026-01-19 04:06:47,461 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2026-01-19 04:06:47,461 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67069065 failed! 2026-01-19 04:06:47,475 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 is 192.168.1.1 and allocation pool is NOT set 2026-01-19 04:06:51,551 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2026-01-19 04:06:51,552 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67069065 failed! 2026-01-19 04:06:51,570 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 is 192.168.1.1 and allocation pool is NOT set 2026-01-19 04:06:55,654 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2026-01-19 04:06:55,654 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67069065 failed! 2026-01-19 04:06:55,669 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 is 192.168.1.1 and allocation pool is NOT set 2026-01-19 04:06:59,751 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2026-01-19 04:06:59,751 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67069065 failed! 2026-01-19 04:06:59,768 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 is 192.168.1.1 and allocation pool is NOT set 2026-01-19 04:07:01,836 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=8.09 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=9.75 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 8.085/8.919/9.753/0.834 ms') 2026-01-19 04:07:01,836 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67069065 passed 2026-01-19 04:07:01,909 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:07:01,909 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-67069065, IP 192.168.1.3, Port 22 2026-01-19 04:07:01,977 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-01-19 04:07:02,068 - DEBUG - VM ctest-vn1_vm1-67069065 is NOT ready for SSH connections, VM status: ACTIVE 2026-01-19 04:07:07,068 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:07:07,069 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-67069065, IP 192.168.1.3, Port 22 2026-01-19 04:07:07,136 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-01-19 04:07:07,213 - DEBUG - VM ctest-vn1_vm1-67069065 is NOT ready for SSH connections, VM status: ACTIVE 2026-01-19 04:07:12,214 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:07:12,214 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-67069065, IP 192.168.1.3, Port 22 2026-01-19 04:07:12,270 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-01-19 04:07:12,390 - DEBUG - VM ctest-vn1_vm1-67069065 is NOT ready for SSH connections, VM status: ACTIVE 2026-01-19 04:07:17,391 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:07:17,392 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-67069065, IP 192.168.1.3, Port 22 2026-01-19 04:07:17,566 - DEBUG - VM ctest-vn1_vm1-67069065 is ready for SSH connections 2026-01-19 04:07:17,566 - INFO - Waiting for VM ctest-vn1_vm2-11400849 to be up.. 2026-01-19 04:07:17,649 - DEBUG - VM is in ACTIVE state now 2026-01-19 04:07:17,649 - INFO - VM name : ctest-vn1_vm2-11400849 2026-01-19 04:07:17,733 - DEBUG - VM ctest-vn1_vm2-11400849 ID is a4e86957-9f3a-4cef-b9bf-e0a588a10abd 2026-01-19 04:07:17,733 - DEBUG - VM ctest-vn1_vm2-11400849 launched on Node an-jenkins-deploy-platform-ansible-os-5040-2 2026-01-19 04:07:17,826 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/a4e86957-9f3a-4cef-b9bf-e0a588a10abd 2026-01-19 04:07:17,837 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/0a71d292-335b-4a13-b6da-c7d957d309d8 2026-01-19 04:07:19,002 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=3.69 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=1.26 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 1.262/2.473/3.685/1.211 ms') 2026-01-19 04:07:19,002 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm2-11400849 passed 2026-01-19 04:07:19,071 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:07:19,071 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-11400849, IP 192.168.1.4, Port 22 2026-01-19 04:07:19,239 - DEBUG - VM ctest-vn1_vm2-11400849 is ready for SSH connections 2026-01-19 04:07:19,239 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:07:19,239 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.249, gateway password: c0ntrail123 2026-01-19 04:07:19,239 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4 2026-01-19 04:07:41,438 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data. 64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=12.7 ms 64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.23 ms 64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.36 ms --- 192.168.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 1.236/5.116/12.752/5.399 ms 2026-01-19 04:07:41,438 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-67069065 passed 2026-01-19 04:07:41,438 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:07:41,438 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.65, gateway password: c0ntrail123 2026-01-19 04:07:41,438 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3 2026-01-19 04:08:17,849 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. 64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=8.51 ms 64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.55 ms 64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.18 ms --- 192.168.1.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 1.180/3.750/8.514/3.372 ms 2026-01-19 04:08:17,850 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-11400849 passed 2026-01-19 04:08:17,856 - INFO - Active control node from the Agent 10.0.0.249 is 10.0.0.23 2026-01-19 04:08:17,856 - INFO - Stoping the Control service in 10.0.0.23 2026-01-19 04:08:18,267 - INFO - 10.0.0.23 2026-01-19 04:08:18,268 - INFO - control:inactive 2026-01-19 04:08:18,268 - INFO - Contrail services control are down on nodes 10.0.0.23 2026-01-19 04:08:23,276 - INFO - Active control node from the Agent 10.0.0.249 is 10.0.0.27 2026-01-19 04:08:23,276 - INFO - Starting the Control service in 10.0.0.23 2026-01-19 04:08:23,651 - INFO - 10.0.0.23 2026-01-19 04:08:23,655 - INFO - control:initializing 2026-01-19 04:08:23,655 - DEBUG - defaultdict(, {'10.0.0.23': {'control': {'status': 'initializing', 'description': None}}}) 2026-01-19 04:08:23,655 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-01-19 04:08:28,755 - INFO - 10.0.0.23 2026-01-19 04:08:28,762 - INFO - control:active 2026-01-19 04:08:28,762 - INFO - Contrail services ['control'] are up on nodes 10.0.0.23 2026-01-19 04:08:33,876 - DEBUG - VM is in ACTIVE state now 2026-01-19 04:08:33,876 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:08:33,877 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/c09da990-2c0f-405b-a690-309bea13b193 2026-01-19 04:08:33,885 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/c0e0fe75-a8c5-4368-9cb2-056118eddd0b 2026-01-19 04:08:33,896 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/c09da990-2c0f-405b-a690-309bea13b193 2026-01-19 04:08:33,904 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/c0e0fe75-a8c5-4368-9cb2-056118eddd0b 2026-01-19 04:08:33,917 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/5f5453ab-9e52-4d63-bfb3-f33cf3c3230b 2026-01-19 04:08:33,928 - DEBUG - Verifying in api server 10.0.0.23 2026-01-19 04:08:33,928 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/c09da990-2c0f-405b-a690-309bea13b193 2026-01-19 04:08:33,937 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/c0e0fe75-a8c5-4368-9cb2-056118eddd0b 2026-01-19 04:08:33,949 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/5f5453ab-9e52-4d63-bfb3-f33cf3c3230b 2026-01-19 04:08:33,960 - INFO - VM ctest-vn1_vm1-67069065 verfication in all API Servers passed 2026-01-19 04:08:33,982 - DEBUG - VM ctest-vn1_vm1-67069065 Tap interface: {'index': '3', 'name': 'tapc0e0fe75-a8', 'uuid': 'c0e0fe75-a8c5-4368-9cb2-056118eddd0b', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351:ctest-vn1-37701351', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '16', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351', 'vm_uuid': 'c09da990-2c0f-405b-a690-309bea13b193', 'vm_name': 'ctest-vn1_vm1-67069065', 'ip_addr': '192.168.1.3', 'mac_addr': '02:c0:e0:fe:75:a8', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '163', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-83017551:c0e0fe75-a8c5-4368-9cb2-056118eddd0b', 'sg_uuid_list': ['33cc7843-a8b2-4b63-ba30-a95791158dfd'], 'static_route_list': None, 'vm_project_uuid': '7b723255-fa49-4769-b023-38f28f67d984', 'admin_state': 'Enabled', 'flow_key_idx': '25', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-01-19 04:08:33,982 - DEBUG - Agent 10.0.0.249 vrf name: default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351:ctest-vn1-37701351 2026-01-19 04:08:33,989 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351:ctest-vn1-37701351', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351', 'table_label': '-1', 'vxlan_id': '16', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.18:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2026-01-19 04:08:34,075 - DEBUG - VM ctest-vn1_vm1-67069065 : Tap interface tapc0e0fe75-a8 is set to Active 2026-01-19 04:08:34,075 - DEBUG - Tap interface tapc0e0fe75-a8 detail : {'index': '3', 'name': 'tapc0e0fe75-a8', 'uuid': 'c0e0fe75-a8c5-4368-9cb2-056118eddd0b', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351:ctest-vn1-37701351', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '16', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351', 'vm_uuid': 'c09da990-2c0f-405b-a690-309bea13b193', 'vm_name': 'ctest-vn1_vm1-67069065', 'ip_addr': '192.168.1.3', 'mac_addr': '02:c0:e0:fe:75:a8', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '163', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-83017551:c0e0fe75-a8c5-4368-9cb2-056118eddd0b', 'sg_uuid_list': ['33cc7843-a8b2-4b63-ba30-a95791158dfd'], 'static_route_list': None, 'vm_project_uuid': '7b723255-fa49-4769-b023-38f28f67d984', 'admin_state': 'Enabled', 'flow_key_idx': '25', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-01-19 04:08:34,202 - DEBUG - Starting Layer 2 verification in Agent 2026-01-19 04:08:34,215 - DEBUG - Layer 2 path is seen for VM MAC 02:c0:e0:fe:75:a8 in agent 10.0.0.249 2026-01-19 04:08:34,216 - DEBUG - Active layer 2 route in agent is present for VMI tapc0e0fe75-a8 2026-01-19 04:08:34,216 - DEBUG - L2 label(29) matches bw route table and interface table 2026-01-19 04:08:34,222 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 is found in Agent of node 10.0.0.249 2026-01-19 04:08:34,359 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 is consistent in agent 10.0.0.249 2026-01-19 04:08:34,359 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.249 2026-01-19 04:08:34,359 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 verification for VM ctest-vn1_vm1-67069065 in Agent 10.0.0.249 passed 2026-01-19 04:08:34,480 - DEBUG - Starting all layer 2 verification in agent 10.0.0.249 2026-01-19 04:08:34,495 - DEBUG - Route for VM MAC 02:c0:e0:fe:75:a8 is consistent in agent 10.0.0.249 2026-01-19 04:08:34,503 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 is found in Agent of node 10.0.0.65 2026-01-19 04:08:34,635 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 is consistent in agent 10.0.0.65 2026-01-19 04:08:34,635 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.65 2026-01-19 04:08:34,635 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 verification for VM ctest-vn1_vm1-67069065 in Agent 10.0.0.65 passed 2026-01-19 04:08:34,749 - DEBUG - Starting all layer 2 verification in agent 10.0.0.65 2026-01-19 04:08:34,759 - DEBUG - Route for VM MAC 02:c0:e0:fe:75:a8 is consistent in agent 10.0.0.65 2026-01-19 04:08:34,759 - INFO - VM ctest-vn1_vm1-67069065 verifications in Compute nodes passed 2026-01-19 04:08:34,996 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.249 2026-01-19 04:08:35,003 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2026-01-19 04:08:35,110 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.65 2026-01-19 04:08:35,116 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.18, 24; Got 10.20.0.18,24 2026-01-19 04:08:35,116 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.65 2026-01-19 04:08:35,116 - INFO - Validated routes of VM ctest-vn1_vm1-67069065 in all vrouters 2026-01-19 04:08:35,645 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2026-01-19 04:08:35,778 - DEBUG - Layer2 route found for VM MAC 02:c0:e0:fe:75:a8 in Control-node 10.0.0.23 2026-01-19 04:08:35,779 - DEBUG - L2 Label for VM ctest-vn1_vm1-67069065 same between Control-node 10.0.0.23 and Agent, Expected: 29, Seen: 29 2026-01-19 04:08:35,779 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2026-01-19 04:08:35,915 - DEBUG - Layer2 route found for VM MAC 02:c0:e0:fe:75:a8 in Control-node 10.0.0.27 2026-01-19 04:08:35,915 - DEBUG - L2 Label for VM ctest-vn1_vm1-67069065 same between Control-node 10.0.0.27 and Agent, Expected: 29, Seen: 29 2026-01-19 04:08:35,915 - DEBUG - Starting all layer2 verification in 10.0.0.50 Control Node 2026-01-19 04:08:36,048 - DEBUG - Layer2 route found for VM MAC 02:c0:e0:fe:75:a8 in Control-node 10.0.0.50 2026-01-19 04:08:36,048 - DEBUG - L2 Label for VM ctest-vn1_vm1-67069065 same between Control-node 10.0.0.50 and Agent, Expected: 29, Seen: 29 2026-01-19 04:08:36,048 - INFO - Verification in Control-nodes for VM ctest-vn1_vm1-67069065 passed 2026-01-19 04:08:36,048 - DEBUG - Verifying the vm in opserver 2026-01-19 04:08:36,048 - DEBUG - Verifying in collector 10.0.0.23 ... 2026-01-19 04:08:36,048 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/c09da990-2c0f-405b-a690-309bea13b193?flat 2026-01-19 04:08:36,113 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-83017551:c0e0fe75-a8c5-4368-9cb2-056118eddd0b?flat 2026-01-19 04:08:36,133 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-83017551:c0e0fe75-a8c5-4368-9cb2-056118eddd0b?flat 2026-01-19 04:08:36,220 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/c09da990-2c0f-405b-a690-309bea13b193?flat 2026-01-19 04:08:36,240 - DEBUG - Verifying vm in vn uve 2026-01-19 04:08:36,240 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-83017551:c0e0fe75-a8c5-4368-9cb2-056118eddd0b?flat 2026-01-19 04:08:36,321 - DEBUG - VM uve shows interface as {'ip4_active': True, 'active': True, 'mac_address': '02:c0:e0:fe:75:a8', 'ip6_active': False, 'vhostuser_mode': 0, 'uuid': 'c0e0fe75-a8c5-4368-9cb2-056118eddd0b', 'tx_vlan': 65535, 'is_health_check_active': True, 'rx_vlan': 65535, 'ip_address': '192.168.1.3', 'vm_uuid': 'c09da990-2c0f-405b-a690-309bea13b193', 'l2_active': True, 'vm_name': 'ctest-vn1_vm1-67069065', 'fixed_ip4_list': ['192.168.1.3'], 'admin_state': True, 'ip6_address': '::', 'gateway': '192.168.1.1', '__T': 1768795712638365, 'port_mirror_enabled': False, 'vn_uuid': 'c7d84faf-aaf5-420b-b599-5e46fe5337c8', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351', 'label': 24} 2026-01-19 04:08:36,322 - DEBUG - VM uve shows ip address as ['192.168.1.3', '::'] 2026-01-19 04:08:36,322 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 2026-01-19 04:08:36,322 - INFO - Verifying through opserver in 10.0.0.27 2026-01-19 04:08:36,322 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351?flat 2026-01-19 04:08:36,351 - DEBUG - expected vm list c09da990-2c0f-405b-a690-309bea13b193 2026-01-19 04:08:36,351 - DEBUG - Extracted vm list ['c09da990-2c0f-405b-a690-309bea13b193'] 2026-01-19 04:08:36,351 - INFO - VM c09da990-2c0f-405b-a690-309bea13b193 is present in default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 2026-01-19 04:08:36,352 - DEBUG - Verifying vm in vrouter uve 2026-01-19 04:08:36,352 - DEBUG - Getting info from collector 10.0.0.23.. 2026-01-19 04:08:36,352 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/c09da990-2c0f-405b-a690-309bea13b193?flat 2026-01-19 04:08:36,425 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-5040-1'] 2026-01-19 04:08:36,426 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-1:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:08:36,472 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-1?flat 2026-01-19 04:08:36,511 - DEBUG - VM c09da990-2c0f-405b-a690-309bea13b193 is present in vrouter an-jenkins-deploy-platform-ansible-os-5040-1 uve 2026-01-19 04:08:36,511 - DEBUG - Validated that VM ctest-vn1_vm1-67069065 is in Vrouter an-jenkins-deploy-platform-ansible-os-5040-1 UVE 2026-01-19 04:08:36,511 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2026-01-19 04:08:36,521 - DEBUG - Expected tap interface of VM uuid c09da990-2c0f-405b-a690-309bea13b193 is default-domain:ctest-TestBasicVMVN0-83017551:c0e0fe75-a8c5-4368-9cb2-056118eddd0b 2026-01-19 04:08:36,521 - DEBUG - Expected VN of VM uuid c09da990-2c0f-405b-a690-309bea13b193 is default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 2026-01-19 04:08:36,521 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-1:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:08:36,564 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-1?flat 2026-01-19 04:08:36,603 - DEBUG - VM c09da990-2c0f-405b-a690-309bea13b193 is present in vrouter an-jenkins-deploy-platform-ansible-os-5040-1 uve 2026-01-19 04:08:36,603 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-83017551:c0e0fe75-a8c5-4368-9cb2-056118eddd0b of vm c09da990-2c0f-405b-a690-309bea13b193 is present in vrouter an-jenkins-deploy-platform-ansible-os-5040-1 uve 2026-01-19 04:08:36,603 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 of vm c09da990-2c0f-405b-a690-309bea13b193 is present in vrouter an-jenkins-deploy-platform-ansible-os-5040-1 uve 2026-01-19 04:08:36,603 - DEBUG - Verifying the c09da990-2c0f-405b-a690-309bea13b193 virtual network link through opserver 10.0.0.27 2026-01-19 04:08:36,603 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-01-19 04:08:36,616 - DEBUG - VM link and name as {'name': 'c09da990-2c0f-405b-a690-309bea13b193', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/c09da990-2c0f-405b-a690-309bea13b193?flat'} 2026-01-19 04:08:36,616 - DEBUG - Verifying the c09da990-2c0f-405b-a690-309bea13b193 virtual network link through opserver 10.0.0.50 2026-01-19 04:08:36,616 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2026-01-19 04:08:36,689 - DEBUG - VM link and name as {'name': 'c09da990-2c0f-405b-a690-309bea13b193', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-machine/c09da990-2c0f-405b-a690-309bea13b193?flat'} 2026-01-19 04:08:36,689 - DEBUG - Verifying the c09da990-2c0f-405b-a690-309bea13b193 virtual network link through opserver 10.0.0.23 2026-01-19 04:08:36,689 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-01-19 04:08:36,705 - DEBUG - VM link and name as {'name': 'c09da990-2c0f-405b-a690-309bea13b193', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/c09da990-2c0f-405b-a690-309bea13b193?flat'} 2026-01-19 04:08:36,706 - INFO - VM ctest-vn1_vm1-67069065 validations in Opserver passed 2026-01-19 04:08:36,814 - DEBUG - VM is in ACTIVE state now 2026-01-19 04:08:36,814 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:08:36,814 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/a4e86957-9f3a-4cef-b9bf-e0a588a10abd 2026-01-19 04:08:36,822 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/0a71d292-335b-4a13-b6da-c7d957d309d8 2026-01-19 04:08:36,837 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/a4e86957-9f3a-4cef-b9bf-e0a588a10abd 2026-01-19 04:08:36,846 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/0a71d292-335b-4a13-b6da-c7d957d309d8 2026-01-19 04:08:36,862 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/42cb3fba-fd10-446c-96a1-8303d6367623 2026-01-19 04:08:36,875 - DEBUG - Verifying in api server 10.0.0.23 2026-01-19 04:08:36,875 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/a4e86957-9f3a-4cef-b9bf-e0a588a10abd 2026-01-19 04:08:36,886 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/0a71d292-335b-4a13-b6da-c7d957d309d8 2026-01-19 04:08:36,901 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/42cb3fba-fd10-446c-96a1-8303d6367623 2026-01-19 04:08:36,914 - INFO - VM ctest-vn1_vm2-11400849 verfication in all API Servers passed 2026-01-19 04:08:36,931 - DEBUG - VM ctest-vn1_vm2-11400849 Tap interface: {'index': '3', 'name': 'tap0a71d292-33', 'uuid': '0a71d292-335b-4a13-b6da-c7d957d309d8', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351:ctest-vn1-37701351', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '16', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351', 'vm_uuid': 'a4e86957-9f3a-4cef-b9bf-e0a588a10abd', 'vm_name': 'ctest-vn1_vm2-11400849', 'ip_addr': '192.168.1.4', 'mac_addr': '02:0a:71:d2:92:33', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '124', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-83017551:0a71d292-335b-4a13-b6da-c7d957d309d8', 'sg_uuid_list': ['33cc7843-a8b2-4b63-ba30-a95791158dfd'], 'static_route_list': None, 'vm_project_uuid': '7b723255-fa49-4769-b023-38f28f67d984', 'admin_state': 'Enabled', 'flow_key_idx': '25', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-01-19 04:08:36,931 - DEBUG - Agent 10.0.0.65 vrf name: default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351:ctest-vn1-37701351 2026-01-19 04:08:36,937 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351:ctest-vn1-37701351', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351', 'table_label': '-1', 'vxlan_id': '16', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.192:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2026-01-19 04:08:37,017 - DEBUG - VM ctest-vn1_vm2-11400849 : Tap interface tap0a71d292-33 is set to Active 2026-01-19 04:08:37,018 - DEBUG - Tap interface tap0a71d292-33 detail : {'index': '3', 'name': 'tap0a71d292-33', 'uuid': '0a71d292-335b-4a13-b6da-c7d957d309d8', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351:ctest-vn1-37701351', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '16', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351', 'vm_uuid': 'a4e86957-9f3a-4cef-b9bf-e0a588a10abd', 'vm_name': 'ctest-vn1_vm2-11400849', 'ip_addr': '192.168.1.4', 'mac_addr': '02:0a:71:d2:92:33', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '124', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-83017551:0a71d292-335b-4a13-b6da-c7d957d309d8', 'sg_uuid_list': ['33cc7843-a8b2-4b63-ba30-a95791158dfd'], 'static_route_list': None, 'vm_project_uuid': '7b723255-fa49-4769-b023-38f28f67d984', 'admin_state': 'Enabled', 'flow_key_idx': '25', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-01-19 04:08:37,148 - DEBUG - Starting Layer 2 verification in Agent 2026-01-19 04:08:37,158 - DEBUG - Layer 2 path is seen for VM MAC 02:0a:71:d2:92:33 in agent 10.0.0.65 2026-01-19 04:08:37,159 - DEBUG - Active layer 2 route in agent is present for VMI tap0a71d292-33 2026-01-19 04:08:37,159 - DEBUG - L2 label(29) matches bw route table and interface table 2026-01-19 04:08:37,167 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 is found in Agent of node 10.0.0.249 2026-01-19 04:08:37,304 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 is consistent in agent 10.0.0.249 2026-01-19 04:08:37,304 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.249 2026-01-19 04:08:37,304 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 verification for VM ctest-vn1_vm2-11400849 in Agent 10.0.0.249 passed 2026-01-19 04:08:37,422 - DEBUG - Starting all layer 2 verification in agent 10.0.0.249 2026-01-19 04:08:37,434 - DEBUG - Route for VM MAC 02:0a:71:d2:92:33 is consistent in agent 10.0.0.249 2026-01-19 04:08:37,443 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 is found in Agent of node 10.0.0.65 2026-01-19 04:08:37,577 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 is consistent in agent 10.0.0.65 2026-01-19 04:08:37,577 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.65 2026-01-19 04:08:37,577 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 verification for VM ctest-vn1_vm2-11400849 in Agent 10.0.0.65 passed 2026-01-19 04:08:37,692 - DEBUG - Starting all layer 2 verification in agent 10.0.0.65 2026-01-19 04:08:37,702 - DEBUG - Route for VM MAC 02:0a:71:d2:92:33 is consistent in agent 10.0.0.65 2026-01-19 04:08:37,703 - INFO - VM ctest-vn1_vm2-11400849 verifications in Compute nodes passed 2026-01-19 04:08:37,932 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.249 2026-01-19 04:08:37,937 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.192, 24; Got 10.20.0.192,24 2026-01-19 04:08:37,937 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.249 2026-01-19 04:08:38,038 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.65 2026-01-19 04:08:38,042 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2026-01-19 04:08:38,042 - INFO - Validated routes of VM ctest-vn1_vm2-11400849 in all vrouters 2026-01-19 04:08:38,556 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2026-01-19 04:08:38,685 - DEBUG - Layer2 route found for VM MAC 02:0a:71:d2:92:33 in Control-node 10.0.0.23 2026-01-19 04:08:38,685 - DEBUG - L2 Label for VM ctest-vn1_vm2-11400849 same between Control-node 10.0.0.23 and Agent, Expected: 29, Seen: 29 2026-01-19 04:08:38,685 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2026-01-19 04:08:38,813 - DEBUG - Layer2 route found for VM MAC 02:0a:71:d2:92:33 in Control-node 10.0.0.27 2026-01-19 04:08:38,813 - DEBUG - L2 Label for VM ctest-vn1_vm2-11400849 same between Control-node 10.0.0.27 and Agent, Expected: 29, Seen: 29 2026-01-19 04:08:38,813 - DEBUG - Starting all layer2 verification in 10.0.0.50 Control Node 2026-01-19 04:08:38,948 - DEBUG - Layer2 route found for VM MAC 02:0a:71:d2:92:33 in Control-node 10.0.0.50 2026-01-19 04:08:38,948 - DEBUG - L2 Label for VM ctest-vn1_vm2-11400849 same between Control-node 10.0.0.50 and Agent, Expected: 29, Seen: 29 2026-01-19 04:08:38,948 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-11400849 passed 2026-01-19 04:08:38,948 - DEBUG - Verifying the vm in opserver 2026-01-19 04:08:38,948 - DEBUG - Verifying in collector 10.0.0.23 ... 2026-01-19 04:08:38,948 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/a4e86957-9f3a-4cef-b9bf-e0a588a10abd?flat 2026-01-19 04:08:38,970 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-83017551:0a71d292-335b-4a13-b6da-c7d957d309d8?flat 2026-01-19 04:08:38,988 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-83017551:0a71d292-335b-4a13-b6da-c7d957d309d8?flat 2026-01-19 04:08:39,009 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/a4e86957-9f3a-4cef-b9bf-e0a588a10abd?flat 2026-01-19 04:08:39,027 - DEBUG - Verifying vm in vn uve 2026-01-19 04:08:39,027 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-83017551:0a71d292-335b-4a13-b6da-c7d957d309d8?flat 2026-01-19 04:08:39,048 - DEBUG - VM uve shows interface as {'__T': 1768795601546041, 'l2_active': True, 'vn_uuid': 'c7d84faf-aaf5-420b-b599-5e46fe5337c8', 'ip6_active': False, 'ip4_active': True, 'vm_uuid': 'a4e86957-9f3a-4cef-b9bf-e0a588a10abd', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351', 'tx_vlan': 65535, 'label': 24, 'rx_vlan': 65535, 'vm_name': 'ctest-vn1_vm2-11400849', 'active': True, 'ip_address': '192.168.1.4', 'port_mirror_enabled': False, 'fixed_ip4_list': ['192.168.1.4'], 'mac_address': '02:0a:71:d2:92:33', 'admin_state': True, 'gateway': '192.168.1.1', 'uuid': '0a71d292-335b-4a13-b6da-c7d957d309d8', 'vhostuser_mode': 0, 'ip6_address': '::', 'is_health_check_active': True} 2026-01-19 04:08:39,048 - DEBUG - VM uve shows ip address as ['192.168.1.4', '::'] 2026-01-19 04:08:39,048 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 2026-01-19 04:08:39,048 - INFO - Verifying through opserver in 10.0.0.27 2026-01-19 04:08:39,048 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351?flat 2026-01-19 04:08:39,079 - DEBUG - expected vm list a4e86957-9f3a-4cef-b9bf-e0a588a10abd 2026-01-19 04:08:39,079 - DEBUG - Extracted vm list ['a4e86957-9f3a-4cef-b9bf-e0a588a10abd'] 2026-01-19 04:08:39,079 - INFO - VM a4e86957-9f3a-4cef-b9bf-e0a588a10abd is present in default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 2026-01-19 04:08:39,079 - DEBUG - Verifying vm in vrouter uve 2026-01-19 04:08:39,079 - DEBUG - Getting info from collector 10.0.0.23.. 2026-01-19 04:08:39,079 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/a4e86957-9f3a-4cef-b9bf-e0a588a10abd?flat 2026-01-19 04:08:39,097 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-5040-2'] 2026-01-19 04:08:39,097 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-2:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:08:39,143 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-2?flat 2026-01-19 04:08:39,178 - DEBUG - VM a4e86957-9f3a-4cef-b9bf-e0a588a10abd is present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:08:39,178 - DEBUG - Validated that VM ctest-vn1_vm2-11400849 is in Vrouter an-jenkins-deploy-platform-ansible-os-5040-2 UVE 2026-01-19 04:08:39,178 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2026-01-19 04:08:39,185 - DEBUG - Expected tap interface of VM uuid a4e86957-9f3a-4cef-b9bf-e0a588a10abd is default-domain:ctest-TestBasicVMVN0-83017551:0a71d292-335b-4a13-b6da-c7d957d309d8 2026-01-19 04:08:39,185 - DEBUG - Expected VN of VM uuid a4e86957-9f3a-4cef-b9bf-e0a588a10abd is default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 2026-01-19 04:08:39,185 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-2:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:08:39,232 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-2?flat 2026-01-19 04:08:39,262 - DEBUG - VM a4e86957-9f3a-4cef-b9bf-e0a588a10abd is present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:08:39,262 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-83017551:0a71d292-335b-4a13-b6da-c7d957d309d8 of vm a4e86957-9f3a-4cef-b9bf-e0a588a10abd is present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:08:39,262 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 of vm a4e86957-9f3a-4cef-b9bf-e0a588a10abd is present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:08:39,262 - DEBUG - Verifying the a4e86957-9f3a-4cef-b9bf-e0a588a10abd virtual network link through opserver 10.0.0.27 2026-01-19 04:08:39,262 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-01-19 04:08:39,277 - DEBUG - VM link and name as {'name': 'a4e86957-9f3a-4cef-b9bf-e0a588a10abd', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/a4e86957-9f3a-4cef-b9bf-e0a588a10abd?flat'} 2026-01-19 04:08:39,277 - DEBUG - Verifying the a4e86957-9f3a-4cef-b9bf-e0a588a10abd virtual network link through opserver 10.0.0.50 2026-01-19 04:08:39,277 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2026-01-19 04:08:39,289 - DEBUG - VM link and name as {'name': 'a4e86957-9f3a-4cef-b9bf-e0a588a10abd', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-machine/a4e86957-9f3a-4cef-b9bf-e0a588a10abd?flat'} 2026-01-19 04:08:39,289 - DEBUG - Verifying the a4e86957-9f3a-4cef-b9bf-e0a588a10abd virtual network link through opserver 10.0.0.23 2026-01-19 04:08:39,289 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-01-19 04:08:39,307 - DEBUG - VM link and name as {'name': 'a4e86957-9f3a-4cef-b9bf-e0a588a10abd', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/a4e86957-9f3a-4cef-b9bf-e0a588a10abd?flat'} 2026-01-19 04:08:39,307 - INFO - VM ctest-vn1_vm2-11400849 validations in Opserver passed 2026-01-19 04:08:39,307 - INFO - Checking the ping between the VM again 2026-01-19 04:08:39,307 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:08:39,307 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.249, gateway password: c0ntrail123 2026-01-19 04:08:39,307 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4 2026-01-19 04:08:41,775 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data. 64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=5.76 ms 64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.39 ms 64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=2.15 ms --- 192.168.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2005ms rtt min/avg/max/mdev = 1.397/3.106/5.765/1.905 ms 2026-01-19 04:08:41,775 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-67069065 passed 2026-01-19 04:08:41,775 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:08:41,775 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.65, gateway password: c0ntrail123 2026-01-19 04:08:41,775 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3 2026-01-19 04:08:44,223 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. 64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=2.54 ms 64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.28 ms 64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.24 ms --- 192.168.1.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2005ms rtt min/avg/max/mdev = 1.249/1.691/2.544/0.603 ms 2026-01-19 04:08:44,223 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-11400849 passed 2026-01-19 04:08:44,223 - INFO - Deleting VM ctest-vn1_vm2-11400849 2026-01-19 04:08:44,308 - DEBUG - Verifying in api server 10.0.0.23 2026-01-19 04:08:44,308 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/a4e86957-9f3a-4cef-b9bf-e0a588a10abd 2026-01-19 04:08:44,320 - DEBUG - VM ID a4e86957-9f3a-4cef-b9bf-e0a588a10abd of VM ctest-vn1_vm2-11400849 is still found in API Server 2026-01-19 04:08:46,320 - DEBUG - Verifying in api server 10.0.0.23 2026-01-19 04:08:46,321 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/a4e86957-9f3a-4cef-b9bf-e0a588a10abd 2026-01-19 04:08:46,328 - DEBUG - Response Code: 404 2026-01-19 04:08:46,328 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/a4e86957-9f3a-4cef-b9bf-e0a588a10abd 2026-01-19 04:08:46,336 - DEBUG - Response Code: 404 2026-01-19 04:08:46,337 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/a4e86957-9f3a-4cef-b9bf-e0a588a10abd 2026-01-19 04:08:46,343 - DEBUG - Response Code: 404 2026-01-19 04:08:46,343 - INFO - VM ctest-vn1_vm2-11400849 is fully removed in API-Server 2026-01-19 04:08:46,498 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2026-01-19 04:08:46,646 - DEBUG - VM route 192.168.1.4 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 of compute 10.0.0.249 2026-01-19 04:08:46,650 - DEBUG - Validated that vrouter 10.0.0.249 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 2026-01-19 04:08:46,650 - DEBUG - Validated that vrouter 10.0.0.65 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 2026-01-19 04:08:46,650 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 2026-01-19 04:08:46,650 - INFO - VM ctest-vn1_vm2-11400849 is removed in Compute, and routes are removed in all compute nodes 2026-01-19 04:08:46,908 - INFO - Routes for VM ctest-vn1_vm2-11400849 is removed in all control-nodes 2026-01-19 04:08:46,908 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-2:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:08:46,956 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-2?flat 2026-01-19 04:08:46,987 - DEBUG - VM a4e86957-9f3a-4cef-b9bf-e0a588a10abd is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:08:46,987 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:08:46,987 - DEBUG - interface for vm a4e86957-9f3a-4cef-b9bf-e0a588a10abd deleted from vrouter uve ... 2026-01-19 04:08:46,987 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:08:49,988 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-2:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:08:50,042 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-2?flat 2026-01-19 04:08:50,073 - DEBUG - VM a4e86957-9f3a-4cef-b9bf-e0a588a10abd is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:08:50,073 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:08:50,073 - DEBUG - interface for vm a4e86957-9f3a-4cef-b9bf-e0a588a10abd deleted from vrouter uve ... 2026-01-19 04:08:50,073 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:08:53,073 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-2:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:08:53,120 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-2?flat 2026-01-19 04:08:53,150 - DEBUG - VM a4e86957-9f3a-4cef-b9bf-e0a588a10abd is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:08:53,151 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:08:53,151 - DEBUG - interface for vm a4e86957-9f3a-4cef-b9bf-e0a588a10abd deleted from vrouter uve ... 2026-01-19 04:08:53,151 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:08:56,151 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-2:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:08:56,199 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-2?flat 2026-01-19 04:08:56,232 - DEBUG - VM a4e86957-9f3a-4cef-b9bf-e0a588a10abd is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:08:56,232 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:08:56,232 - DEBUG - interface for vm a4e86957-9f3a-4cef-b9bf-e0a588a10abd deleted from vrouter uve ... 2026-01-19 04:08:56,232 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:08:59,233 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-2:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:08:59,279 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-2?flat 2026-01-19 04:08:59,314 - DEBUG - VM a4e86957-9f3a-4cef-b9bf-e0a588a10abd is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:08:59,314 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:08:59,314 - DEBUG - interface for vm a4e86957-9f3a-4cef-b9bf-e0a588a10abd deleted from vrouter uve ... 2026-01-19 04:08:59,314 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:09:02,315 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-2:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:09:02,364 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-2?flat 2026-01-19 04:09:02,395 - DEBUG - VM a4e86957-9f3a-4cef-b9bf-e0a588a10abd is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:09:02,395 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:09:02,395 - DEBUG - interface for vm a4e86957-9f3a-4cef-b9bf-e0a588a10abd deleted from vrouter uve ... 2026-01-19 04:09:02,396 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:09:05,397 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-2:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:09:05,451 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-2?flat 2026-01-19 04:09:05,487 - DEBUG - VM a4e86957-9f3a-4cef-b9bf-e0a588a10abd is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:09:05,487 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:09:05,487 - DEBUG - interface for vm a4e86957-9f3a-4cef-b9bf-e0a588a10abd deleted from vrouter uve ... 2026-01-19 04:09:05,487 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:09:08,487 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-2:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:09:08,536 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-2?flat 2026-01-19 04:09:08,566 - DEBUG - VM a4e86957-9f3a-4cef-b9bf-e0a588a10abd is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:09:08,566 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:09:08,567 - DEBUG - interface for vm a4e86957-9f3a-4cef-b9bf-e0a588a10abd deleted from vrouter uve ... 2026-01-19 04:09:08,567 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:09:11,567 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-2:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:09:11,614 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-2?flat 2026-01-19 04:09:11,649 - DEBUG - VM a4e86957-9f3a-4cef-b9bf-e0a588a10abd is not present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:09:11,649 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:09:11,649 - DEBUG - interface for vm a4e86957-9f3a-4cef-b9bf-e0a588a10abd deleted from vrouter uve ... 2026-01-19 04:09:11,649 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:09:11,649 - DEBUG - Validated that VM a4e86957-9f3a-4cef-b9bf-e0a588a10abd is removed in Vrouter UVE 2026-01-19 04:09:11,649 - DEBUG - Verifying through opserver in 10.0.0.27 2026-01-19 04:09:11,649 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351?flat 2026-01-19 04:09:11,669 - DEBUG - a4e86957-9f3a-4cef-b9bf-e0a588a10abd BM not in default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 uve 2026-01-19 04:09:11,669 - DEBUG - Validated that VM a4e86957-9f3a-4cef-b9bf-e0a588a10abd is not present in the VN UVE 2026-01-19 04:09:11,669 - DEBUG - Verifying the a4e86957-9f3a-4cef-b9bf-e0a588a10abd virtual network link through opserver 10.0.0.27 2026-01-19 04:09:11,669 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-01-19 04:09:11,682 - INFO - a4e86957-9f3a-4cef-b9bf-e0a588a10abd vm uve deleted from opserver 2026-01-19 04:09:11,682 - DEBUG - Verifying the a4e86957-9f3a-4cef-b9bf-e0a588a10abd virtual network link through opserver 10.0.0.50 2026-01-19 04:09:11,682 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2026-01-19 04:09:11,694 - INFO - a4e86957-9f3a-4cef-b9bf-e0a588a10abd vm uve deleted from opserver 2026-01-19 04:09:11,694 - DEBUG - Verifying the a4e86957-9f3a-4cef-b9bf-e0a588a10abd virtual network link through opserver 10.0.0.23 2026-01-19 04:09:11,694 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-01-19 04:09:11,707 - INFO - a4e86957-9f3a-4cef-b9bf-e0a588a10abd vm uve deleted from opserver 2026-01-19 04:09:11,707 - INFO - Validated that VM a4e86957-9f3a-4cef-b9bf-e0a588a10abd is removed from Opserver 2026-01-19 04:09:11,707 - INFO - Deleting VM ctest-vn1_vm1-67069065 2026-01-19 04:09:11,776 - DEBUG - Verifying in api server 10.0.0.23 2026-01-19 04:09:11,777 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/c09da990-2c0f-405b-a690-309bea13b193 2026-01-19 04:09:11,786 - DEBUG - VM ID c09da990-2c0f-405b-a690-309bea13b193 of VM ctest-vn1_vm1-67069065 is still found in API Server 2026-01-19 04:09:13,787 - DEBUG - Verifying in api server 10.0.0.23 2026-01-19 04:09:13,787 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/c09da990-2c0f-405b-a690-309bea13b193 2026-01-19 04:09:13,794 - DEBUG - Response Code: 404 2026-01-19 04:09:13,794 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/c09da990-2c0f-405b-a690-309bea13b193 2026-01-19 04:09:13,800 - DEBUG - Response Code: 404 2026-01-19 04:09:13,801 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/c09da990-2c0f-405b-a690-309bea13b193 2026-01-19 04:09:13,807 - DEBUG - Response Code: 404 2026-01-19 04:09:13,808 - INFO - VM ctest-vn1_vm1-67069065 is fully removed in API-Server 2026-01-19 04:09:13,958 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2026-01-19 04:09:14,003 - DEBUG - Validated that vrouter 10.0.0.249 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 2026-01-19 04:09:14,015 - DEBUG - Validated that vrouter 10.0.0.65 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 2026-01-19 04:09:14,015 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 2026-01-19 04:09:14,015 - INFO - VM ctest-vn1_vm1-67069065 is removed in Compute, and routes are removed in all compute nodes 2026-01-19 04:09:14,245 - INFO - Routes for VM ctest-vn1_vm1-67069065 is removed in all control-nodes 2026-01-19 04:09:14,245 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-1:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:09:14,286 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-1?flat 2026-01-19 04:09:14,319 - DEBUG - VM c09da990-2c0f-405b-a690-309bea13b193 is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-1 uve 2026-01-19 04:09:14,320 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:09:14,320 - DEBUG - interface for vm c09da990-2c0f-405b-a690-309bea13b193 deleted from vrouter uve ... 2026-01-19 04:09:14,320 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:09:17,320 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-1:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:09:17,368 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-1?flat 2026-01-19 04:09:17,422 - DEBUG - VM c09da990-2c0f-405b-a690-309bea13b193 is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-1 uve 2026-01-19 04:09:17,422 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:09:17,422 - DEBUG - interface for vm c09da990-2c0f-405b-a690-309bea13b193 deleted from vrouter uve ... 2026-01-19 04:09:17,422 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:09:20,423 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-1:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:09:20,472 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-1?flat 2026-01-19 04:09:20,502 - DEBUG - VM c09da990-2c0f-405b-a690-309bea13b193 is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-1 uve 2026-01-19 04:09:20,502 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:09:20,502 - DEBUG - interface for vm c09da990-2c0f-405b-a690-309bea13b193 deleted from vrouter uve ... 2026-01-19 04:09:20,502 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:09:23,503 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-1:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:09:23,547 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-1?flat 2026-01-19 04:09:23,577 - DEBUG - VM c09da990-2c0f-405b-a690-309bea13b193 is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-1 uve 2026-01-19 04:09:23,577 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:09:23,577 - DEBUG - interface for vm c09da990-2c0f-405b-a690-309bea13b193 deleted from vrouter uve ... 2026-01-19 04:09:23,577 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:09:26,577 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-1:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:09:26,621 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-1?flat 2026-01-19 04:09:26,652 - DEBUG - VM c09da990-2c0f-405b-a690-309bea13b193 is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-1 uve 2026-01-19 04:09:26,652 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:09:26,652 - DEBUG - interface for vm c09da990-2c0f-405b-a690-309bea13b193 deleted from vrouter uve ... 2026-01-19 04:09:26,652 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:09:29,652 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-1:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:09:29,700 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-1?flat 2026-01-19 04:09:29,732 - DEBUG - VM c09da990-2c0f-405b-a690-309bea13b193 is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-1 uve 2026-01-19 04:09:29,732 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:09:29,732 - DEBUG - interface for vm c09da990-2c0f-405b-a690-309bea13b193 deleted from vrouter uve ... 2026-01-19 04:09:29,732 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:09:32,732 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-1:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:09:32,777 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-1?flat 2026-01-19 04:09:32,807 - DEBUG - VM c09da990-2c0f-405b-a690-309bea13b193 is not present in vrouter an-jenkins-deploy-platform-ansible-os-5040-1 uve 2026-01-19 04:09:32,807 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:09:32,807 - DEBUG - interface for vm c09da990-2c0f-405b-a690-309bea13b193 deleted from vrouter uve ... 2026-01-19 04:09:32,807 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:09:32,807 - DEBUG - Validated that VM c09da990-2c0f-405b-a690-309bea13b193 is removed in Vrouter UVE 2026-01-19 04:09:32,807 - DEBUG - Verifying through opserver in 10.0.0.27 2026-01-19 04:09:32,807 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351?flat 2026-01-19 04:09:32,823 - DEBUG - c09da990-2c0f-405b-a690-309bea13b193 BM not in default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-37701351 uve 2026-01-19 04:09:32,823 - DEBUG - Validated that VM c09da990-2c0f-405b-a690-309bea13b193 is not present in the VN UVE 2026-01-19 04:09:32,823 - DEBUG - Verifying the c09da990-2c0f-405b-a690-309bea13b193 virtual network link through opserver 10.0.0.27 2026-01-19 04:09:32,823 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-01-19 04:09:32,833 - INFO - c09da990-2c0f-405b-a690-309bea13b193 vm uve deleted from opserver 2026-01-19 04:09:32,833 - DEBUG - Verifying the c09da990-2c0f-405b-a690-309bea13b193 virtual network link through opserver 10.0.0.50 2026-01-19 04:09:32,833 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2026-01-19 04:09:32,846 - INFO - c09da990-2c0f-405b-a690-309bea13b193 vm uve deleted from opserver 2026-01-19 04:09:32,846 - DEBUG - Verifying the c09da990-2c0f-405b-a690-309bea13b193 virtual network link through opserver 10.0.0.23 2026-01-19 04:09:32,846 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-01-19 04:09:32,862 - INFO - c09da990-2c0f-405b-a690-309bea13b193 vm uve deleted from opserver 2026-01-19 04:09:32,862 - INFO - Validated that VM c09da990-2c0f-405b-a690-309bea13b193 is removed from Opserver 2026-01-19 04:09:32,862 - INFO - Deleting VN ctest-vn1-37701351 2026-01-19 04:09:33,017 - DEBUG - Response for deleting network () 2026-01-19 04:09:33,017 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/ceb01fb1-9d6b-487c-afda-4795c923304d 2026-01-19 04:09:33,027 - DEBUG - Response Code: 404 2026-01-19 04:09:33,027 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-19 04:09:33,032 - DEBUG - Requesting: http://10.0.0.27:8082/domain/bc123c74-c45e-4d7a-a955-76f59d7a4cc7 2026-01-19 04:09:33,087 - DEBUG - Requesting: http://10.0.0.27:8082/project/7b723255-fa49-4769-b023-38f28f67d984 2026-01-19 04:09:33,174 - INFO - Validated that VN ctest-vn1-37701351 is not found in API Server 2026-01-19 04:09:33,187 - DEBUG - VN ctest-vn1-37701351 is not present in Agent 10.0.0.249 2026-01-19 04:09:33,197 - DEBUG - VN ctest-vn1-37701351 is not present in Agent 10.0.0.65 2026-01-19 04:09:33,198 - INFO - Validated that VN ctest-vn1-37701351 is not in any agent 2026-01-19 04:09:33,229 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-37701351 info 2026-01-19 04:09:34,504 - DEBUG - Skipping xmpp flap check

2026-01-19 04:09:34,505 - INFO - END TEST : test_control_node_switchover : PASSED[0:03:19]

2026-01-19 04:09:34,505 - INFO - -------------------------------------------------------------------------------- 2026-01-19 04:09:34,508 - INFO - ================================================================================ 2026-01-19 04:09:34,508 - INFO - STARTING TEST : test_process_restart_in_policy_between_vns 2026-01-19 04:09:34,508 - INFO - TEST DESCRIPTION : Test to validate that with policy having rule to check icmp fwding between VMs on different VNs , ping between VMs should pass with process restarts 1. Pick 2 VN's from resource pool which has one VM each 2. Create policy with icmp allow rule between those VN's and bind it networks 3. Ping from one VM to another VM 4. Restart process 'vrouter' and 'control' on setup 5. Ping again between VM's after process restart Pass criteria: Step 2,3,4 and 5 should pass 2026-01-19 04:09:35,809 - DEBUG - Skipping xmpp flap check 2026-01-19 04:09:35,809 - INFO - Initial checks done. Running the testcase now 2026-01-19 04:09:35,809 - INFO - 2026-01-19 04:09:35,815 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2026-01-19 04:09:36,119 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2026-01-19 04:09:36,703 - INFO - Created VN ctest-vn1-56688583, UUID :2c816068-94e5-4286-bf22-33b52acca2fd 2026-01-19 04:09:37,050 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-19 04:09:37,056 - DEBUG - Requesting: http://10.0.0.27:8082/domain/bc123c74-c45e-4d7a-a955-76f59d7a4cc7 2026-01-19 04:09:37,066 - DEBUG - Requesting: http://10.0.0.27:8082/project/7b723255-fa49-4769-b023-38f28f67d984 2026-01-19 04:09:37,164 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2c816068-94e5-4286-bf22-33b52acca2fd 2026-01-19 04:09:37,178 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2c816068-94e5-4286-bf22-33b52acca2fd 2026-01-19 04:09:37,190 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/57e98c3f-a527-4240-beea-90b4a26bc842 2026-01-19 04:09:37,200 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/57e98c3f-a527-4240-beea-90b4a26bc842 2026-01-19 04:09:37,211 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/7076590c-68f4-46d8-8f91-4a732dddb932 2026-01-19 04:09:37,221 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-19 04:09:37,221 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2c816068-94e5-4286-bf22-33b52acca2fd 2026-01-19 04:09:37,234 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/57e98c3f-a527-4240-beea-90b4a26bc842 2026-01-19 04:09:37,253 - INFO - Verified VN network id 16 for VN 2c816068-94e5-4286-bf22-33b52acca2fd 2026-01-19 04:09:37,253 - INFO - Verifications in API Server for VN ctest-vn1-56688583 passed 2026-01-19 04:09:37,253 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2c816068-94e5-4286-bf22-33b52acca2fd 2026-01-19 04:09:37,265 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/57e98c3f-a527-4240-beea-90b4a26bc842 2026-01-19 04:09:37,274 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/57e98c3f-a527-4240-beea-90b4a26bc842 2026-01-19 04:09:37,288 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/7076590c-68f4-46d8-8f91-4a732dddb932 2026-01-19 04:09:37,303 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-83017551', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3206950411955421830', 'uuid-lslong': '13772627463652549373'}, 'enable': 'true', 'created': '2026-01-19T04:09:36', 'last-modified': '2026-01-19T04:09:36', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.443316'} 2026-01-19 04:09:37,313 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-19 04:09:37,319 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-83017551', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3206950411955421830', 'uuid-lslong': '13772627463652549373'}, 'enable': 'true', 'created': '2026-01-19T04:09:36', 'last-modified': '2026-01-19T04:09:36', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.462053'} 2026-01-19 04:09:37,327 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-19 04:09:37,332 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-83017551', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3206950411955421830', 'uuid-lslong': '13772627463652549373'}, 'enable': 'true', 'created': '2026-01-19T04:09:36', 'last-modified': '2026-01-19T04:09:36', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.474478'} 2026-01-19 04:09:37,339 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-19 04:09:37,339 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-56688583 passed 2026-01-19 04:09:37,339 - DEBUG - ====Verifying policy data for ctest-vn1-56688583 in API_Server ====== 2026-01-19 04:09:37,340 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-19 04:09:37,347 - DEBUG - Requesting: http://10.0.0.27:8082/domain/bc123c74-c45e-4d7a-a955-76f59d7a4cc7 2026-01-19 04:09:37,361 - DEBUG - Requesting: http://10.0.0.27:8082/project/7b723255-fa49-4769-b023-38f28f67d984 2026-01-19 04:09:37,463 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2c816068-94e5-4286-bf22-33b52acca2fd 2026-01-19 04:09:37,472 - DEBUG - =>VN ctest-vn1-56688583 has no policy to be verified 2026-01-19 04:09:37,473 - DEBUG - Verifying the vn in opserver 2026-01-19 04:09:37,473 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 virtual network link through opserver 10.0.0.27 2026-01-19 04:09:37,473 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2026-01-19 04:09:37,487 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583?flat'} 2026-01-19 04:09:37,487 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 virtual network link through opserver 10.0.0.50 2026-01-19 04:09:37,487 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2026-01-19 04:09:37,561 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583?flat'} 2026-01-19 04:09:37,561 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 virtual network link through opserver 10.0.0.23 2026-01-19 04:09:37,561 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2026-01-19 04:09:37,574 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583?flat'} 2026-01-19 04:09:37,574 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 is found in opserver 2026-01-19 04:09:37,598 - DEBUG - Do not have enough data to verify VN in agent 2026-01-19 04:09:37,609 - DEBUG - VRF ids for VN ctest-vn1-56688583: {} 2026-01-19 04:09:38,311 - INFO - Created VN ctest-vn2-55794601, UUID :d47bd86e-aebf-48ab-86c6-ba296b4b455b 2026-01-19 04:09:38,637 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-19 04:09:38,644 - DEBUG - Requesting: http://10.0.0.27:8082/domain/bc123c74-c45e-4d7a-a955-76f59d7a4cc7 2026-01-19 04:09:38,656 - DEBUG - Requesting: http://10.0.0.27:8082/project/7b723255-fa49-4769-b023-38f28f67d984 2026-01-19 04:09:38,746 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/d47bd86e-aebf-48ab-86c6-ba296b4b455b 2026-01-19 04:09:38,756 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/d47bd86e-aebf-48ab-86c6-ba296b4b455b 2026-01-19 04:09:38,764 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/977aced7-e3e1-4139-b0f0-474a734fd8ac 2026-01-19 04:09:38,770 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/977aced7-e3e1-4139-b0f0-474a734fd8ac 2026-01-19 04:09:38,778 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/7995bab3-eb7e-4a47-b198-a8bf0a989916 2026-01-19 04:09:38,785 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-01-19 04:09:38,785 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/d47bd86e-aebf-48ab-86c6-ba296b4b455b 2026-01-19 04:09:38,793 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/977aced7-e3e1-4139-b0f0-474a734fd8ac 2026-01-19 04:09:38,809 - INFO - Verified VN network id 17 for VN d47bd86e-aebf-48ab-86c6-ba296b4b455b 2026-01-19 04:09:38,809 - INFO - Verifications in API Server for VN ctest-vn2-55794601 passed 2026-01-19 04:09:38,809 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/d47bd86e-aebf-48ab-86c6-ba296b4b455b 2026-01-19 04:09:38,820 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/977aced7-e3e1-4139-b0f0-474a734fd8ac 2026-01-19 04:09:38,827 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/977aced7-e3e1-4139-b0f0-474a734fd8ac 2026-01-19 04:09:38,833 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/7995bab3-eb7e-4a47-b198-a8bf0a989916 2026-01-19 04:09:38,845 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-83017551', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '15311069328065906859', 'uuid-lslong': '9711654333527573851'}, 'enable': 'true', 'created': '2026-01-19T04:09:38', 'last-modified': '2026-01-19T04:09:38', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.392951'} 2026-01-19 04:09:38,850 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-01-19 04:09:38,854 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-83017551', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '15311069328065906859', 'uuid-lslong': '9711654333527573851'}, 'enable': 'true', 'created': '2026-01-19T04:09:38', 'last-modified': '2026-01-19T04:09:38', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.398711'} 2026-01-19 04:09:38,862 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-01-19 04:09:38,868 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-83017551', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '15311069328065906859', 'uuid-lslong': '9711654333527573851'}, 'enable': 'true', 'created': '2026-01-19T04:09:38', 'last-modified': '2026-01-19T04:09:38', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.414060'} 2026-01-19 04:09:38,874 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-01-19 04:09:38,874 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn2-55794601 passed 2026-01-19 04:09:38,874 - DEBUG - ====Verifying policy data for ctest-vn2-55794601 in API_Server ====== 2026-01-19 04:09:38,874 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-19 04:09:38,882 - DEBUG - Requesting: http://10.0.0.27:8082/domain/bc123c74-c45e-4d7a-a955-76f59d7a4cc7 2026-01-19 04:09:38,896 - DEBUG - Requesting: http://10.0.0.27:8082/project/7b723255-fa49-4769-b023-38f28f67d984 2026-01-19 04:09:38,991 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/d47bd86e-aebf-48ab-86c6-ba296b4b455b 2026-01-19 04:09:39,001 - DEBUG - =>VN ctest-vn2-55794601 has no policy to be verified 2026-01-19 04:09:39,001 - DEBUG - Verifying the vn in opserver 2026-01-19 04:09:39,001 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 virtual network link through opserver 10.0.0.27 2026-01-19 04:09:39,001 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2026-01-19 04:09:39,013 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601?flat'} 2026-01-19 04:09:39,013 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 virtual network link through opserver 10.0.0.50 2026-01-19 04:09:39,013 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2026-01-19 04:09:39,026 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601?flat'} 2026-01-19 04:09:39,026 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 virtual network link through opserver 10.0.0.23 2026-01-19 04:09:39,026 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2026-01-19 04:09:39,042 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601?flat'} 2026-01-19 04:09:39,042 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 is found in opserver 2026-01-19 04:09:39,064 - DEBUG - Do not have enough data to verify VN in agent 2026-01-19 04:09:39,076 - DEBUG - VRF ids for VN ctest-vn2-55794601: {} 2026-01-19 04:09:40,648 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-5040-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-5040-1) 2026-01-19 04:09:42,032 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-5040-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-5040-2) 2026-01-19 04:09:42,032 - INFO - Waiting for VM ctest-vn1_vm1-30025374 to be up.. 2026-01-19 04:09:42,139 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2026-01-19 04:09:47,228 - DEBUG - VM is in ACTIVE state now 2026-01-19 04:09:47,228 - INFO - VM name : ctest-vn1_vm1-30025374 2026-01-19 04:09:47,313 - DEBUG - VM ctest-vn1_vm1-30025374 ID is b40cbf40-d982-447d-abb5-7105ee65aedd 2026-01-19 04:09:47,314 - DEBUG - VM ctest-vn1_vm1-30025374 launched on Node an-jenkins-deploy-platform-ansible-os-5040-1 2026-01-19 04:09:47,391 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/b40cbf40-d982-447d-abb5-7105ee65aedd 2026-01-19 04:09:47,402 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/7d691ec3-c458-4e21-a40b-d4e45ea0d586 2026-01-19 04:09:50,568 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1003ms') 2026-01-19 04:09:50,568 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-30025374 failed! 2026-01-19 04:09:50,583 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 is 192.168.1.254 and allocation pool is NOT set 2026-01-19 04:09:54,662 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2026-01-19 04:09:54,662 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-30025374 failed! 2026-01-19 04:09:54,677 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 is 192.168.1.254 and allocation pool is NOT set 2026-01-19 04:09:58,767 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1021ms') 2026-01-19 04:09:58,767 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-30025374 failed! 2026-01-19 04:09:58,795 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 is 192.168.1.254 and allocation pool is NOT set 2026-01-19 04:10:02,885 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1027ms') 2026-01-19 04:10:02,885 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-30025374 failed! 2026-01-19 04:10:02,900 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 is 192.168.1.254 and allocation pool is NOT set 2026-01-19 04:10:06,984 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2026-01-19 04:10:06,984 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-30025374 failed! 2026-01-19 04:10:07,000 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 is 192.168.1.254 and allocation pool is NOT set 2026-01-19 04:10:11,084 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1016ms') 2026-01-19 04:10:11,084 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-30025374 failed! 2026-01-19 04:10:11,100 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 is 192.168.1.254 and allocation pool is NOT set 2026-01-19 04:10:15,169 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1008ms') 2026-01-19 04:10:15,169 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-30025374 failed! 2026-01-19 04:10:15,188 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 is 192.168.1.254 and allocation pool is NOT set 2026-01-19 04:10:19,271 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2026-01-19 04:10:19,271 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-30025374 failed! 2026-01-19 04:10:19,288 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 is 192.168.1.254 and allocation pool is NOT set 2026-01-19 04:10:21,356 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=5.01 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=3.57 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 3.572/4.289/5.006/0.717 ms') 2026-01-19 04:10:21,356 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-30025374 passed 2026-01-19 04:10:21,429 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:10:21,429 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-30025374, IP 192.168.1.252, Port 22 2026-01-19 04:10:21,497 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-01-19 04:10:21,606 - DEBUG - VM ctest-vn1_vm1-30025374 is NOT ready for SSH connections, VM status: ACTIVE 2026-01-19 04:10:26,607 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:10:26,608 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-30025374, IP 192.168.1.252, Port 22 2026-01-19 04:10:26,666 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-01-19 04:10:26,751 - DEBUG - VM ctest-vn1_vm1-30025374 is NOT ready for SSH connections, VM status: ACTIVE 2026-01-19 04:10:31,752 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:10:31,752 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-30025374, IP 192.168.1.252, Port 22 2026-01-19 04:10:31,821 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-01-19 04:10:31,909 - DEBUG - VM ctest-vn1_vm1-30025374 is NOT ready for SSH connections, VM status: ACTIVE 2026-01-19 04:10:36,909 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:10:36,909 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-30025374, IP 192.168.1.252, Port 22 2026-01-19 04:10:36,976 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-01-19 04:10:37,064 - DEBUG - VM ctest-vn1_vm1-30025374 is NOT ready for SSH connections, VM status: ACTIVE 2026-01-19 04:10:42,065 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:10:42,066 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-30025374, IP 192.168.1.252, Port 22 2026-01-19 04:10:42,233 - DEBUG - VM ctest-vn1_vm1-30025374 is ready for SSH connections 2026-01-19 04:10:42,233 - INFO - Waiting for VM ctest-vn2_vm1-35050972 to be up.. 2026-01-19 04:10:42,322 - DEBUG - VM is in ACTIVE state now 2026-01-19 04:10:42,323 - INFO - VM name : ctest-vn2_vm1-35050972 2026-01-19 04:10:42,406 - DEBUG - VM ctest-vn2_vm1-35050972 ID is d66c7689-0f18-41bc-b13b-2a8e77eef215 2026-01-19 04:10:42,406 - DEBUG - VM ctest-vn2_vm1-35050972 launched on Node an-jenkins-deploy-platform-ansible-os-5040-2 2026-01-19 04:10:42,487 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/d66c7689-0f18-41bc-b13b-2a8e77eef215 2026-01-19 04:10:42,499 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/82a828cb-d02c-4b90-9e1d-5f954dc8f8d8 2026-01-19 04:10:43,666 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=3.24 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.499 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.499/1.867/3.236/1.368 ms') 2026-01-19 04:10:43,666 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn2_vm1-35050972 passed 2026-01-19 04:10:43,740 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:10:43,740 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-35050972, IP 192.168.2.252, Port 22 2026-01-19 04:10:43,900 - DEBUG - VM ctest-vn2_vm1-35050972 is ready for SSH connections 2026-01-19 04:10:43,900 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:10:43,900 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.252, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.249, gateway password: c0ntrail123 2026-01-19 04:10:43,900 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.252 2026-01-19 04:11:04,278 - DEBUG - PING 192.168.2.252 (192.168.2.252) 56(84) bytes of data. 64 bytes from 192.168.2.252: icmp_req=1 ttl=63 time=10.2 ms 64 bytes from 192.168.2.252: icmp_req=2 ttl=63 time=1.69 ms 64 bytes from 192.168.2.252: icmp_req=3 ttl=63 time=1.23 ms --- 192.168.2.252 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 1.239/4.405/10.279/4.158 ms 2026-01-19 04:11:04,278 - INFO - Ping to IP 192.168.2.252 from VM ctest-vn1_vm1-30025374 passed 2026-01-19 04:11:07,686 - INFO - 10.0.0.249 2026-01-19 04:11:07,689 - INFO - agent:initializing 2026-01-19 04:11:07,690 - DEBUG - defaultdict(, {'10.0.0.249': {'agent': {'status': 'initializing', 'description': None}}}) 2026-01-19 04:11:07,690 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-01-19 04:11:12,781 - INFO - 10.0.0.249 2026-01-19 04:11:12,786 - INFO - agent:active 2026-01-19 04:11:12,787 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.249 2026-01-19 04:11:16,159 - INFO - 10.0.0.65 2026-01-19 04:11:16,163 - INFO - agent:initializing 2026-01-19 04:11:16,163 - DEBUG - defaultdict(, {'10.0.0.65': {'agent': {'status': 'initializing', 'description': None}}}) 2026-01-19 04:11:16,163 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-01-19 04:11:21,251 - INFO - 10.0.0.65 2026-01-19 04:11:21,257 - INFO - agent:active 2026-01-19 04:11:21,257 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.65 2026-01-19 04:11:21,862 - INFO - 10.0.0.27 2026-01-19 04:11:21,866 - INFO - control:initializing 2026-01-19 04:11:21,866 - DEBUG - defaultdict(, {'10.0.0.27': {'control': {'status': 'initializing', 'description': None}}}) 2026-01-19 04:11:21,866 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-01-19 04:11:26,970 - INFO - 10.0.0.27 2026-01-19 04:11:26,976 - INFO - control:active 2026-01-19 04:11:26,976 - INFO - Contrail services ['control'] are up on nodes 10.0.0.27 2026-01-19 04:11:27,367 - INFO - 10.0.0.50 2026-01-19 04:11:27,372 - INFO - control:initializing 2026-01-19 04:11:27,372 - DEBUG - defaultdict(, {'10.0.0.50': {'control': {'status': 'initializing', 'description': None}}}) 2026-01-19 04:11:27,372 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-01-19 04:11:32,476 - INFO - 10.0.0.50 2026-01-19 04:11:32,483 - INFO - control:active 2026-01-19 04:11:32,484 - INFO - Contrail services ['control'] are up on nodes 10.0.0.50 2026-01-19 04:11:32,931 - INFO - 10.0.0.23 2026-01-19 04:11:32,938 - INFO - control:initializing 2026-01-19 04:11:32,939 - DEBUG - defaultdict(, {'10.0.0.23': {'control': {'status': 'initializing', 'description': None}}}) 2026-01-19 04:11:32,939 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-01-19 04:11:38,040 - INFO - 10.0.0.23 2026-01-19 04:11:38,046 - INFO - control:active 2026-01-19 04:11:38,046 - INFO - Contrail services ['control'] are up on nodes 10.0.0.23 2026-01-19 04:11:38,792 - INFO - 10.0.0.27 2026-01-19 04:11:38,795 - INFO - api-server:initializing 2026-01-19 04:11:38,796 - DEBUG - defaultdict(, {'10.0.0.27': {'api-server': {'status': 'initializing', 'description': None}}}) 2026-01-19 04:11:38,796 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-01-19 04:11:43,918 - INFO - 10.0.0.27 2026-01-19 04:11:43,921 - INFO - api-server:initializing 2026-01-19 04:11:43,922 - DEBUG - defaultdict(, {'10.0.0.27': {'api-server': {'status': 'initializing', 'description': None}}}) 2026-01-19 04:11:43,922 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2026-01-19 04:11:49,053 - INFO - 10.0.0.27 2026-01-19 04:11:49,058 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2026-01-19 04:11:49,059 - DEBUG - defaultdict(, {'10.0.0.27': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2026-01-19 04:11:49,059 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2026-01-19 04:11:54,202 - INFO - 10.0.0.27 2026-01-19 04:11:54,207 - INFO - api-server:active 2026-01-19 04:11:54,207 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.27 2026-01-19 04:11:54,898 - INFO - 10.0.0.50 2026-01-19 04:11:54,903 - INFO - api-server:initializing 2026-01-19 04:11:54,903 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': None}}}) 2026-01-19 04:11:54,903 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-01-19 04:12:00,002 - INFO - 10.0.0.50 2026-01-19 04:12:00,006 - INFO - api-server:initializing 2026-01-19 04:12:00,007 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': None}}}) 2026-01-19 04:12:00,007 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2026-01-19 04:12:05,095 - INFO - 10.0.0.50 2026-01-19 04:12:05,100 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2026-01-19 04:12:05,101 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2026-01-19 04:12:05,101 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2026-01-19 04:12:10,193 - INFO - 10.0.0.50 2026-01-19 04:12:10,198 - INFO - api-server:active 2026-01-19 04:12:10,198 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.50 2026-01-19 04:12:10,927 - INFO - 10.0.0.23 2026-01-19 04:12:10,933 - INFO - api-server:initializing 2026-01-19 04:12:10,933 - DEBUG - defaultdict(, {'10.0.0.23': {'api-server': {'status': 'initializing', 'description': None}}}) 2026-01-19 04:12:10,933 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2026-01-19 04:12:16,041 - INFO - 10.0.0.23 2026-01-19 04:12:16,046 - INFO - api-server:initializing 2026-01-19 04:12:16,046 - DEBUG - defaultdict(, {'10.0.0.23': {'api-server': {'status': 'initializing', 'description': None}}}) 2026-01-19 04:12:16,046 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2026-01-19 04:12:21,147 - INFO - 10.0.0.23 2026-01-19 04:12:21,154 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2026-01-19 04:12:21,154 - DEBUG - defaultdict(, {'10.0.0.23': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2026-01-19 04:12:21,154 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2026-01-19 04:12:26,254 - INFO - 10.0.0.23 2026-01-19 04:12:26,260 - INFO - api-server:active 2026-01-19 04:12:26,260 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.23 2026-01-19 04:12:26,361 - INFO - 10.0.0.27 2026-01-19 04:12:26,361 - INFO - config 2026-01-19 04:12:26,366 - INFO - config-nodemgr:active 2026-01-19 04:12:26,370 - INFO - api-server:active 2026-01-19 04:12:26,374 - INFO - schema:active 2026-01-19 04:12:26,377 - INFO - svc-monitor:active 2026-01-19 04:12:26,379 - INFO - device-manager:backup 2026-01-19 04:12:26,379 - INFO - config-database 2026-01-19 04:12:26,379 - INFO - config-cassandra:active 2026-01-19 04:12:26,379 - INFO - config-zookeeper:active 2026-01-19 04:12:26,380 - INFO - config-rabbitmq:active 2026-01-19 04:12:26,380 - INFO - control 2026-01-19 04:12:26,384 - INFO - control-nodemgr:active 2026-01-19 04:12:26,388 - INFO - control:active 2026-01-19 04:12:26,388 - INFO - named:active 2026-01-19 04:12:26,395 - INFO - dns:active 2026-01-19 04:12:26,396 - INFO - analytics 2026-01-19 04:12:26,400 - INFO - analytics-nodemgr:active 2026-01-19 04:12:26,404 - INFO - analytics-api:active 2026-01-19 04:12:26,408 - INFO - collector:active 2026-01-19 04:12:26,409 - INFO - analytics-database 2026-01-19 04:12:26,409 - INFO - analytics-cassandra:active 2026-01-19 04:12:26,413 - INFO - analyticsdb-nodemgr:active 2026-01-19 04:12:26,418 - INFO - query-engine:active 2026-01-19 04:12:26,418 - INFO - webui 2026-01-19 04:12:26,418 - INFO - webui:active 2026-01-19 04:12:26,419 - INFO - webui-middleware:active 2026-01-19 04:12:26,419 - INFO - redis:active 2026-01-19 04:12:26,419 - INFO - analytics_snmp 2026-01-19 04:12:26,423 - INFO - snmp-collector:active 2026-01-19 04:12:26,423 - INFO - snmp-topology:active 2026-01-19 04:12:26,423 - INFO - analytics_alarm 2026-01-19 04:12:26,423 - INFO - alarmgen:active 2026-01-19 04:12:26,524 - INFO - 10.0.0.50 2026-01-19 04:12:26,524 - INFO - config 2026-01-19 04:12:26,529 - INFO - config-nodemgr:active 2026-01-19 04:12:26,533 - INFO - api-server:active 2026-01-19 04:12:26,536 - INFO - schema:backup 2026-01-19 04:12:26,540 - INFO - svc-monitor:backup 2026-01-19 04:12:26,543 - INFO - device-manager:active 2026-01-19 04:12:26,544 - INFO - config-database 2026-01-19 04:12:26,544 - INFO - config-cassandra:active 2026-01-19 04:12:26,544 - INFO - config-zookeeper:active 2026-01-19 04:12:26,544 - INFO - config-rabbitmq:active 2026-01-19 04:12:26,544 - INFO - control 2026-01-19 04:12:26,548 - INFO - control-nodemgr:active 2026-01-19 04:12:26,553 - INFO - control:active 2026-01-19 04:12:26,553 - INFO - named:active 2026-01-19 04:12:26,558 - INFO - dns:active 2026-01-19 04:12:26,558 - INFO - analytics 2026-01-19 04:12:26,562 - INFO - analytics-nodemgr:active 2026-01-19 04:12:26,566 - INFO - analytics-api:active 2026-01-19 04:12:26,571 - INFO - collector:active 2026-01-19 04:12:26,571 - INFO - analytics-database 2026-01-19 04:12:26,571 - INFO - analytics-cassandra:active 2026-01-19 04:12:26,576 - INFO - analyticsdb-nodemgr:active 2026-01-19 04:12:26,581 - INFO - query-engine:active 2026-01-19 04:12:26,581 - INFO - webui 2026-01-19 04:12:26,581 - INFO - webui:active 2026-01-19 04:12:26,581 - INFO - webui-middleware:active 2026-01-19 04:12:26,581 - INFO - redis:active 2026-01-19 04:12:26,581 - INFO - analytics_snmp 2026-01-19 04:12:26,585 - INFO - snmp-collector:active 2026-01-19 04:12:26,585 - INFO - snmp-topology:active 2026-01-19 04:12:26,585 - INFO - analytics_alarm 2026-01-19 04:12:26,585 - INFO - alarmgen:active 2026-01-19 04:12:26,683 - INFO - 10.0.0.23 2026-01-19 04:12:26,683 - INFO - config 2026-01-19 04:12:26,688 - INFO - config-nodemgr:active 2026-01-19 04:12:26,693 - INFO - api-server:active 2026-01-19 04:12:26,696 - INFO - schema:backup 2026-01-19 04:12:26,699 - INFO - svc-monitor:backup 2026-01-19 04:12:26,703 - INFO - device-manager:backup 2026-01-19 04:12:26,703 - INFO - config-database 2026-01-19 04:12:26,703 - INFO - config-cassandra:active 2026-01-19 04:12:26,703 - INFO - config-zookeeper:active 2026-01-19 04:12:26,703 - INFO - config-rabbitmq:active 2026-01-19 04:12:26,703 - INFO - control 2026-01-19 04:12:26,708 - INFO - control-nodemgr:active 2026-01-19 04:12:26,713 - INFO - control:active 2026-01-19 04:12:26,713 - INFO - named:active 2026-01-19 04:12:26,718 - INFO - dns:active 2026-01-19 04:12:26,718 - INFO - analytics 2026-01-19 04:12:26,723 - INFO - analytics-nodemgr:active 2026-01-19 04:12:26,728 - INFO - analytics-api:active 2026-01-19 04:12:26,733 - INFO - collector:active 2026-01-19 04:12:26,733 - INFO - analytics-database 2026-01-19 04:12:26,733 - INFO - analytics-cassandra:active 2026-01-19 04:12:26,738 - INFO - analyticsdb-nodemgr:active 2026-01-19 04:12:26,743 - INFO - query-engine:active 2026-01-19 04:12:26,744 - INFO - webui 2026-01-19 04:12:26,744 - INFO - webui:active 2026-01-19 04:12:26,744 - INFO - webui-middleware:active 2026-01-19 04:12:26,744 - INFO - redis:active 2026-01-19 04:12:26,744 - INFO - analytics_snmp 2026-01-19 04:12:26,748 - INFO - snmp-collector:active 2026-01-19 04:12:26,748 - INFO - snmp-topology:active 2026-01-19 04:12:26,748 - INFO - analytics_alarm 2026-01-19 04:12:26,748 - INFO - alarmgen:active 2026-01-19 04:12:26,849 - INFO - 10.0.0.249 2026-01-19 04:12:26,849 - INFO - vrouter 2026-01-19 04:12:26,855 - INFO - vrouter-nodemgr:active 2026-01-19 04:12:26,860 - INFO - agent:active 2026-01-19 04:12:26,958 - INFO - 10.0.0.65 2026-01-19 04:12:26,959 - INFO - vrouter 2026-01-19 04:12:26,965 - INFO - vrouter-nodemgr:active 2026-01-19 04:12:26,970 - INFO - agent:active 2026-01-19 04:12:26,970 - INFO - All the contrail services are up on all nodes 2026-01-19 04:12:26,970 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-1?flat 2026-01-19 04:12:27,047 - INFO - an-jenkins-deploy-platform-ansible-os-5040-1:contrail-vrouter-agent module connection to ['10.20.0.25:5269'] servers UP 2026-01-19 04:12:27,047 - INFO - an-jenkins-deploy-platform-ansible-os-5040-1 connected to xmpp 10.20.0.25 2026-01-19 04:12:27,047 - INFO - an-jenkins-deploy-platform-ansible-os-5040-1:contrail-vrouter-agent module connection to ['10.20.0.17:5269'] servers UP 2026-01-19 04:12:27,047 - INFO - an-jenkins-deploy-platform-ansible-os-5040-1 connected to xmpp 10.20.0.17 2026-01-19 04:12:27,047 - WARNING - an-jenkins-deploy-platform-ansible-os-5040-1:contrail-vrouter-agent module connection to ['10.20.0.193:5269'] servers NOT UP 2026-01-19 04:12:27,047 - INFO - an-jenkins-deploy-platform-ansible-os-5040-1:contrail-vrouter-agent module connection to ['10.20.0.25:53'] servers UP 2026-01-19 04:12:27,047 - INFO - an-jenkins-deploy-platform-ansible-os-5040-1 connected to dns 10.20.0.25 2026-01-19 04:12:27,048 - INFO - an-jenkins-deploy-platform-ansible-os-5040-1:contrail-vrouter-agent module connection to ['10.20.0.17:53'] servers UP 2026-01-19 04:12:27,048 - INFO - an-jenkins-deploy-platform-ansible-os-5040-1 connected to dns 10.20.0.17 2026-01-19 04:12:27,048 - WARNING - an-jenkins-deploy-platform-ansible-os-5040-1:contrail-vrouter-agent module connection to ['10.20.0.193:53'] servers NOT UP 2026-01-19 04:12:27,048 - WARNING - an-jenkins-deploy-platform-ansible-os-5040-1:contrail-vrouter-agent module connection to ['10.0.0.27:8086'] servers NOT UP 2026-01-19 04:12:27,048 - WARNING - an-jenkins-deploy-platform-ansible-os-5040-1:contrail-vrouter-agent module connection to ['10.0.0.50:8086'] servers NOT UP 2026-01-19 04:12:27,048 - INFO - an-jenkins-deploy-platform-ansible-os-5040-1:contrail-vrouter-agent module connection to ['10.0.0.23:8086'] servers UP 2026-01-19 04:12:27,048 - INFO - an-jenkins-deploy-platform-ansible-os-5040-1 connected to collector 10.0.0.23 2026-01-19 04:12:27,048 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-5040-1?flat 2026-01-19 04:12:27,439 - INFO - cn-jenkins-deploy-platform-ansible-os-5040-1:contrail-control module connection to 10.0.0.27:9041 servers UP 2026-01-19 04:12:27,440 - INFO - cn-jenkins-deploy-platform-ansible-os-5040-1:contrail-control module connection to 10.0.0.27:5673 servers UP 2026-01-19 04:12:27,440 - INFO - cn-jenkins-deploy-platform-ansible-os-5040-1:contrail-control module connection to ['10.0.0.27:8086'] servers UP 2026-01-19 04:12:27,440 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-5040-2?flat 2026-01-19 04:12:27,851 - INFO - cn-jenkins-deploy-platform-ansible-os-5040-2:contrail-control module connection to 10.0.0.27:9041 servers UP 2026-01-19 04:12:27,851 - INFO - cn-jenkins-deploy-platform-ansible-os-5040-2:contrail-control module connection to 10.0.0.27:5673 servers UP 2026-01-19 04:12:27,852 - INFO - cn-jenkins-deploy-platform-ansible-os-5040-2:contrail-control module connection to ['10.0.0.27:8086'] servers UP 2026-01-19 04:12:27,852 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-5040-3?flat 2026-01-19 04:12:27,870 - INFO - cn-jenkins-deploy-platform-ansible-os-5040-3:contrail-control module connection to 10.0.0.27:9041 servers UP 2026-01-19 04:12:27,870 - INFO - cn-jenkins-deploy-platform-ansible-os-5040-3:contrail-control module connection to 10.0.0.27:5673 servers UP 2026-01-19 04:12:27,870 - INFO - cn-jenkins-deploy-platform-ansible-os-5040-3:contrail-control module connection to ['10.0.0.27:8086'] servers UP 2026-01-19 04:12:27,871 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-5040-1.?flat 2026-01-19 04:12:27,898 - DEBUG - contrail-api is running 2026-01-19 04:12:27,898 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-5040-2.?flat 2026-01-19 04:12:27,928 - DEBUG - contrail-api is running 2026-01-19 04:12:27,928 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-5040-3.?flat 2026-01-19 04:12:27,950 - DEBUG - contrail-api is running 2026-01-19 04:12:28,263 - INFO - Created VN ctest-vn3-83621045, UUID :a7e3b545-8cba-43f4-b5ec-6fd801beee63 2026-01-19 04:12:28,731 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-19 04:12:28,771 - DEBUG - Requesting: http://10.0.0.27:8082/domain/bc123c74-c45e-4d7a-a955-76f59d7a4cc7 2026-01-19 04:12:28,783 - DEBUG - Requesting: http://10.0.0.27:8082/project/7b723255-fa49-4769-b023-38f28f67d984 2026-01-19 04:12:28,890 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2c816068-94e5-4286-bf22-33b52acca2fd 2026-01-19 04:12:28,905 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2c816068-94e5-4286-bf22-33b52acca2fd 2026-01-19 04:12:28,915 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/57e98c3f-a527-4240-beea-90b4a26bc842 2026-01-19 04:12:28,924 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/57e98c3f-a527-4240-beea-90b4a26bc842 2026-01-19 04:12:28,932 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/7076590c-68f4-46d8-8f91-4a732dddb932 2026-01-19 04:12:28,941 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-19 04:12:28,941 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2c816068-94e5-4286-bf22-33b52acca2fd 2026-01-19 04:12:28,955 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/57e98c3f-a527-4240-beea-90b4a26bc842 2026-01-19 04:12:28,983 - INFO - Verified VN network id 16 for VN 2c816068-94e5-4286-bf22-33b52acca2fd 2026-01-19 04:12:28,983 - INFO - Verifications in API Server for VN ctest-vn1-56688583 passed 2026-01-19 04:12:28,983 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2c816068-94e5-4286-bf22-33b52acca2fd 2026-01-19 04:12:29,000 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/57e98c3f-a527-4240-beea-90b4a26bc842 2026-01-19 04:12:29,009 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/57e98c3f-a527-4240-beea-90b4a26bc842 2026-01-19 04:12:29,018 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/7076590c-68f4-46d8-8f91-4a732dddb932 2026-01-19 04:12:29,045 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-83017551', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3206950411955421830', 'uuid-lslong': '13772627463652549373'}, 'enable': 'true', 'created': '2026-01-19T04:09:36', 'last-modified': '2026-01-19T04:09:37', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:06.616461'} 2026-01-19 04:12:29,053 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-19 04:12:29,059 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-83017551', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3206950411955421830', 'uuid-lslong': '13772627463652549373'}, 'enable': 'true', 'created': '2026-01-19T04:09:36', 'last-modified': '2026-01-19T04:09:37', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:01.337600'} 2026-01-19 04:12:29,066 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-19 04:12:29,072 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-83017551', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3206950411955421830', 'uuid-lslong': '13772627463652549373'}, 'enable': 'true', 'created': '2026-01-19T04:09:36', 'last-modified': '2026-01-19T04:09:37', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:55.655716'} 2026-01-19 04:12:29,080 - DEBUG - Route Targets: ['target:64512:8000004'] 2026-01-19 04:12:29,080 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-56688583 passed 2026-01-19 04:12:29,080 - DEBUG - ====Verifying policy data for ctest-vn1-56688583 in API_Server ====== 2026-01-19 04:12:29,080 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-19 04:12:29,086 - DEBUG - Requesting: http://10.0.0.27:8082/domain/bc123c74-c45e-4d7a-a955-76f59d7a4cc7 2026-01-19 04:12:29,099 - DEBUG - Requesting: http://10.0.0.27:8082/project/7b723255-fa49-4769-b023-38f28f67d984 2026-01-19 04:12:29,193 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2c816068-94e5-4286-bf22-33b52acca2fd 2026-01-19 04:12:29,205 - DEBUG - ==>Verifying data for policy with id: a37dbaf1-8687-4eaa-8e41-964855cfcef5, fqn: ['default-domain', 'ctest-TestBasicVMVN0-83017551', 'policy1'] 2026-01-19 04:12:29,205 - INFO - VN ctest-vn1-56688583 Policy verification: verify_vn_policy_in_api_server, status: True 2026-01-19 04:12:29,205 - DEBUG - Verifying the vn in opserver 2026-01-19 04:12:29,205 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 virtual network link through opserver 10.0.0.27 2026-01-19 04:12:29,205 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2026-01-19 04:12:29,218 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583?flat'} 2026-01-19 04:12:29,218 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 virtual network link through opserver 10.0.0.50 2026-01-19 04:12:29,218 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2026-01-19 04:12:29,284 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583?flat'} 2026-01-19 04:12:29,284 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 virtual network link through opserver 10.0.0.23 2026-01-19 04:12:29,284 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2026-01-19 04:12:29,345 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583?flat'} 2026-01-19 04:12:29,345 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 is found in opserver 2026-01-19 04:12:29,356 - DEBUG - VRF ids for VN ctest-vn1-56688583: {'10.0.0.249': '2'} 2026-01-19 04:12:30,659 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-5040-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-5040-1) 2026-01-19 04:12:30,710 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2026-01-19 04:12:35,804 - DEBUG - VM is in ACTIVE state now 2026-01-19 04:12:35,804 - INFO - VM name : ctest-vn1_vm2-42009684 2026-01-19 04:12:35,905 - DEBUG - VM ctest-vn1_vm2-42009684 ID is ba553e1d-1e39-4c13-8a40-8eba58c297df 2026-01-19 04:12:35,991 - DEBUG - VM is in ACTIVE state now 2026-01-19 04:12:35,991 - DEBUG - VM ctest-vn1_vm2-42009684 launched on Node an-jenkins-deploy-platform-ansible-os-5040-1 2026-01-19 04:12:36,086 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/ba553e1d-1e39-4c13-8a40-8eba58c297df 2026-01-19 04:12:36,124 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/ba553e1d-1e39-4c13-8a40-8eba58c297df 2026-01-19 04:12:36,134 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/fabcddbe-b09b-44ea-b71f-1a1f5e7c836f 2026-01-19 04:12:36,147 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/ba553e1d-1e39-4c13-8a40-8eba58c297df 2026-01-19 04:12:36,159 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/fabcddbe-b09b-44ea-b71f-1a1f5e7c836f 2026-01-19 04:12:36,171 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/966d446f-5044-480f-9e66-7c0d907be506 2026-01-19 04:12:36,191 - DEBUG - Verifying in api server 10.0.0.23 2026-01-19 04:12:36,191 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/ba553e1d-1e39-4c13-8a40-8eba58c297df 2026-01-19 04:12:36,200 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/fabcddbe-b09b-44ea-b71f-1a1f5e7c836f 2026-01-19 04:12:36,210 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/966d446f-5044-480f-9e66-7c0d907be506 2026-01-19 04:12:36,220 - INFO - VM ctest-vn1_vm2-42009684 verfication in all API Servers passed 2026-01-19 04:12:36,237 - DEBUG - VM ctest-vn1_vm2-42009684 Tap interface: {'index': '4', 'name': 'tapfabcddbe-b0', 'uuid': 'fabcddbe-b09b-44ea-b71f-1a1f5e7c836f', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583:ctest-vn1-56688583', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '16', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583', 'vm_uuid': 'ba553e1d-1e39-4c13-8a40-8eba58c297df', 'vm_name': 'ctest-vn1_vm2-42009684', 'ip_addr': '192.168.1.251', 'mac_addr': '02:fa:bc:dd:be:b0', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '167', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-83017551:fabcddbe-b09b-44ea-b71f-1a1f5e7c836f', 'sg_uuid_list': ['33cc7843-a8b2-4b63-ba30-a95791158dfd'], 'static_route_list': None, 'vm_project_uuid': '7b723255-fa49-4769-b023-38f28f67d984', 'admin_state': 'Enabled', 'flow_key_idx': '35', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-01-19 04:12:36,237 - DEBUG - Agent 10.0.0.249 vrf name: default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583:ctest-vn1-56688583 2026-01-19 04:12:36,243 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583:ctest-vn1-56688583', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583', 'table_label': '-1', 'vxlan_id': '16', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.18:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2026-01-19 04:12:36,336 - DEBUG - VM ctest-vn1_vm2-42009684 : Tap interface tapfabcddbe-b0 is set to Active 2026-01-19 04:12:36,336 - DEBUG - Tap interface tapfabcddbe-b0 detail : {'index': '4', 'name': 'tapfabcddbe-b0', 'uuid': 'fabcddbe-b09b-44ea-b71f-1a1f5e7c836f', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583:ctest-vn1-56688583', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '16', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583', 'vm_uuid': 'ba553e1d-1e39-4c13-8a40-8eba58c297df', 'vm_name': 'ctest-vn1_vm2-42009684', 'ip_addr': '192.168.1.251', 'mac_addr': '02:fa:bc:dd:be:b0', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '167', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-83017551:fabcddbe-b09b-44ea-b71f-1a1f5e7c836f', 'sg_uuid_list': ['33cc7843-a8b2-4b63-ba30-a95791158dfd'], 'static_route_list': None, 'vm_project_uuid': '7b723255-fa49-4769-b023-38f28f67d984', 'admin_state': 'Enabled', 'flow_key_idx': '35', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-01-19 04:12:36,450 - DEBUG - Starting Layer 2 verification in Agent 2026-01-19 04:12:36,463 - DEBUG - Layer 2 path is seen for VM MAC 02:fa:bc:dd:be:b0 in agent 10.0.0.249 2026-01-19 04:12:36,464 - DEBUG - Active layer 2 route in agent is present for VMI tapfabcddbe-b0 2026-01-19 04:12:36,464 - DEBUG - L2 label(34) matches bw route table and interface table 2026-01-19 04:12:36,471 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 is found in Agent of node 10.0.0.249 2026-01-19 04:12:36,615 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 is consistent in agent 10.0.0.249 2026-01-19 04:12:36,616 - DEBUG - Route for VM IP ['192.168.1.251'] is consistent in agent 10.0.0.249 2026-01-19 04:12:36,616 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 verification for VM ctest-vn1_vm2-42009684 in Agent 10.0.0.249 passed 2026-01-19 04:12:36,736 - DEBUG - Starting all layer 2 verification in agent 10.0.0.249 2026-01-19 04:12:36,754 - DEBUG - Route for VM MAC 02:fa:bc:dd:be:b0 is consistent in agent 10.0.0.249 2026-01-19 04:12:39,972 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1031ms') 2026-01-19 04:12:39,972 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-42009684 failed! 2026-01-19 04:12:39,988 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 is 192.168.1.254 and allocation pool is NOT set 2026-01-19 04:12:44,070 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2026-01-19 04:12:44,070 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-42009684 failed! 2026-01-19 04:12:44,087 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 is 192.168.1.254 and allocation pool is NOT set 2026-01-19 04:12:48,166 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2026-01-19 04:12:48,166 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-42009684 failed! 2026-01-19 04:12:48,182 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 is 192.168.1.254 and allocation pool is NOT set 2026-01-19 04:12:52,256 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2026-01-19 04:12:52,256 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-42009684 failed! 2026-01-19 04:12:52,270 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 is 192.168.1.254 and allocation pool is NOT set 2026-01-19 04:12:56,359 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1026ms') 2026-01-19 04:12:56,359 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-42009684 failed! 2026-01-19 04:12:56,375 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 is 192.168.1.254 and allocation pool is NOT set 2026-01-19 04:13:00,453 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2026-01-19 04:13:00,453 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-42009684 failed! 2026-01-19 04:13:00,468 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 is 192.168.1.254 and allocation pool is NOT set 2026-01-19 04:13:04,553 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2026-01-19 04:13:04,553 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-42009684 failed! 2026-01-19 04:13:04,575 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 is 192.168.1.254 and allocation pool is NOT set 2026-01-19 04:13:08,650 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1009ms') 2026-01-19 04:13:08,650 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-42009684 failed! 2026-01-19 04:13:08,666 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 is 192.168.1.254 and allocation pool is NOT set 2026-01-19 04:13:10,732 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=7.79 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=1.53 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 1.532/4.661/7.790/3.129 ms') 2026-01-19 04:13:10,732 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-42009684 passed 2026-01-19 04:13:10,732 - INFO - VM ctest-vn1_vm2-42009684 verifications in Compute nodes passed 2026-01-19 04:13:10,942 - DEBUG - Validated VM route 192.168.1.251 in vrouter of 10.0.0.249 2026-01-19 04:13:10,948 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2026-01-19 04:13:10,948 - INFO - Validated routes of VM ctest-vn1_vm2-42009684 in all vrouters 2026-01-19 04:13:11,385 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2026-01-19 04:13:11,512 - DEBUG - Layer2 route found for VM MAC 02:fa:bc:dd:be:b0 in Control-node 10.0.0.23 2026-01-19 04:13:11,512 - DEBUG - L2 Label for VM ctest-vn1_vm2-42009684 same between Control-node 10.0.0.23 and Agent, Expected: 34, Seen: 34 2026-01-19 04:13:11,512 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2026-01-19 04:13:11,643 - DEBUG - Layer2 route found for VM MAC 02:fa:bc:dd:be:b0 in Control-node 10.0.0.27 2026-01-19 04:13:11,644 - DEBUG - L2 Label for VM ctest-vn1_vm2-42009684 same between Control-node 10.0.0.27 and Agent, Expected: 34, Seen: 34 2026-01-19 04:13:11,644 - DEBUG - Starting all layer2 verification in 10.0.0.50 Control Node 2026-01-19 04:13:11,774 - DEBUG - Layer2 route found for VM MAC 02:fa:bc:dd:be:b0 in Control-node 10.0.0.50 2026-01-19 04:13:11,775 - DEBUG - L2 Label for VM ctest-vn1_vm2-42009684 same between Control-node 10.0.0.50 and Agent, Expected: 34, Seen: 34 2026-01-19 04:13:11,775 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-42009684 passed 2026-01-19 04:13:11,775 - DEBUG - Verifying the vm in opserver 2026-01-19 04:13:11,775 - DEBUG - Verifying in collector 10.0.0.23 ... 2026-01-19 04:13:11,775 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/ba553e1d-1e39-4c13-8a40-8eba58c297df?flat 2026-01-19 04:13:11,846 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-83017551:fabcddbe-b09b-44ea-b71f-1a1f5e7c836f?flat 2026-01-19 04:13:11,864 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-83017551:fabcddbe-b09b-44ea-b71f-1a1f5e7c836f?flat 2026-01-19 04:13:11,883 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/ba553e1d-1e39-4c13-8a40-8eba58c297df?flat 2026-01-19 04:13:11,943 - DEBUG - Verifying vm in vn uve 2026-01-19 04:13:11,944 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-83017551:fabcddbe-b09b-44ea-b71f-1a1f5e7c836f?flat 2026-01-19 04:13:11,962 - DEBUG - VM uve shows interface as {'ip4_active': True, 'active': True, 'mac_address': '02:fa:bc:dd:be:b0', 'ip6_active': False, 'vhostuser_mode': 0, 'uuid': 'fabcddbe-b09b-44ea-b71f-1a1f5e7c836f', 'tx_vlan': 65535, 'is_health_check_active': True, 'rx_vlan': 65535, 'ip_address': '192.168.1.251', 'vm_uuid': 'ba553e1d-1e39-4c13-8a40-8eba58c297df', 'l2_active': True, 'vm_name': 'ctest-vn1_vm2-42009684', 'fixed_ip4_list': ['192.168.1.251'], 'admin_state': True, 'ip6_address': '::', 'gateway': '192.168.1.254', '__T': 1768795958281007, 'port_mirror_enabled': False, 'vn_uuid': '2c816068-94e5-4286-bf22-33b52acca2fd', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583', 'label': 30} 2026-01-19 04:13:11,962 - DEBUG - VM uve shows ip address as ['192.168.1.251', '::'] 2026-01-19 04:13:11,962 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 2026-01-19 04:13:11,962 - INFO - Verifying through opserver in 10.0.0.27 2026-01-19 04:13:11,963 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583?flat 2026-01-19 04:13:11,983 - DEBUG - expected vm list ba553e1d-1e39-4c13-8a40-8eba58c297df 2026-01-19 04:13:11,983 - DEBUG - Extracted vm list ['ba553e1d-1e39-4c13-8a40-8eba58c297df'] 2026-01-19 04:13:11,983 - INFO - VM ba553e1d-1e39-4c13-8a40-8eba58c297df is present in default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 2026-01-19 04:13:11,983 - DEBUG - Verifying vm in vrouter uve 2026-01-19 04:13:11,983 - DEBUG - Getting info from collector 10.0.0.23.. 2026-01-19 04:13:11,983 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/ba553e1d-1e39-4c13-8a40-8eba58c297df?flat 2026-01-19 04:13:11,999 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-5040-1'] 2026-01-19 04:13:11,999 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-1:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:13:12,042 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-1?flat 2026-01-19 04:13:12,074 - DEBUG - VM ba553e1d-1e39-4c13-8a40-8eba58c297df is present in vrouter an-jenkins-deploy-platform-ansible-os-5040-1 uve 2026-01-19 04:13:12,074 - DEBUG - Validated that VM ctest-vn1_vm2-42009684 is in Vrouter an-jenkins-deploy-platform-ansible-os-5040-1 UVE 2026-01-19 04:13:12,074 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2026-01-19 04:13:12,083 - DEBUG - Expected tap interface of VM uuid ba553e1d-1e39-4c13-8a40-8eba58c297df is default-domain:ctest-TestBasicVMVN0-83017551:fabcddbe-b09b-44ea-b71f-1a1f5e7c836f 2026-01-19 04:13:12,083 - DEBUG - Expected VN of VM uuid ba553e1d-1e39-4c13-8a40-8eba58c297df is default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 2026-01-19 04:13:12,083 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-1:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:13:12,129 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-1?flat 2026-01-19 04:13:12,161 - DEBUG - VM ba553e1d-1e39-4c13-8a40-8eba58c297df is present in vrouter an-jenkins-deploy-platform-ansible-os-5040-1 uve 2026-01-19 04:13:12,161 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-83017551:fabcddbe-b09b-44ea-b71f-1a1f5e7c836f of vm ba553e1d-1e39-4c13-8a40-8eba58c297df is present in vrouter an-jenkins-deploy-platform-ansible-os-5040-1 uve 2026-01-19 04:13:12,161 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 of vm ba553e1d-1e39-4c13-8a40-8eba58c297df is present in vrouter an-jenkins-deploy-platform-ansible-os-5040-1 uve 2026-01-19 04:13:12,162 - DEBUG - Verifying the ba553e1d-1e39-4c13-8a40-8eba58c297df virtual network link through opserver 10.0.0.27 2026-01-19 04:13:12,162 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-01-19 04:13:12,173 - DEBUG - VM link and name as {'name': 'ba553e1d-1e39-4c13-8a40-8eba58c297df', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/ba553e1d-1e39-4c13-8a40-8eba58c297df?flat'} 2026-01-19 04:13:12,173 - DEBUG - Verifying the ba553e1d-1e39-4c13-8a40-8eba58c297df virtual network link through opserver 10.0.0.50 2026-01-19 04:13:12,173 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2026-01-19 04:13:12,236 - DEBUG - VM link and name as {'name': 'ba553e1d-1e39-4c13-8a40-8eba58c297df', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-machine/ba553e1d-1e39-4c13-8a40-8eba58c297df?flat'} 2026-01-19 04:13:12,237 - DEBUG - Verifying the ba553e1d-1e39-4c13-8a40-8eba58c297df virtual network link through opserver 10.0.0.23 2026-01-19 04:13:12,237 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-01-19 04:13:12,252 - DEBUG - VM link and name as {'name': 'ba553e1d-1e39-4c13-8a40-8eba58c297df', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/ba553e1d-1e39-4c13-8a40-8eba58c297df?flat'} 2026-01-19 04:13:12,252 - INFO - VM ctest-vn1_vm2-42009684 validations in Opserver passed 2026-01-19 04:13:13,499 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-5040-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-5040-2) 2026-01-19 04:13:13,544 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2026-01-19 04:13:18,627 - DEBUG - VM is in ACTIVE state now 2026-01-19 04:13:18,627 - INFO - VM name : ctest-vn2_vm2-39790800 2026-01-19 04:13:18,716 - DEBUG - VM ctest-vn2_vm2-39790800 ID is 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 2026-01-19 04:13:18,798 - DEBUG - VM is in ACTIVE state now 2026-01-19 04:13:18,798 - DEBUG - VM ctest-vn2_vm2-39790800 launched on Node an-jenkins-deploy-platform-ansible-os-5040-2 2026-01-19 04:13:18,878 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/12c2d7af-d89f-4385-b6f1-0a09f2d56d17 2026-01-19 04:13:18,889 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/12c2d7af-d89f-4385-b6f1-0a09f2d56d17 2026-01-19 04:13:18,896 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/829d79ec-9889-4fd3-aeab-93c47d27501b 2026-01-19 04:13:18,908 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/12c2d7af-d89f-4385-b6f1-0a09f2d56d17 2026-01-19 04:13:18,916 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/829d79ec-9889-4fd3-aeab-93c47d27501b 2026-01-19 04:13:18,926 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/8ec93941-d8a9-4855-ba99-f6552db8c825 2026-01-19 04:13:18,939 - DEBUG - Verifying in api server 10.0.0.23 2026-01-19 04:13:18,939 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/12c2d7af-d89f-4385-b6f1-0a09f2d56d17 2026-01-19 04:13:18,947 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/829d79ec-9889-4fd3-aeab-93c47d27501b 2026-01-19 04:13:18,958 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/8ec93941-d8a9-4855-ba99-f6552db8c825 2026-01-19 04:13:18,969 - INFO - VM ctest-vn2_vm2-39790800 verfication in all API Servers passed 2026-01-19 04:13:18,992 - DEBUG - VM ctest-vn2_vm2-39790800 Tap interface: {'index': '4', 'name': 'tap829d79ec-98', 'uuid': '829d79ec-9889-4fd3-aeab-93c47d27501b', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601:ctest-vn2-55794601', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '17', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601', 'vm_uuid': '12c2d7af-d89f-4385-b6f1-0a09f2d56d17', 'vm_name': 'ctest-vn2_vm2-39790800', 'ip_addr': '192.168.2.251', 'mac_addr': '02:82:9d:79:ec:98', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '128', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-83017551:829d79ec-9889-4fd3-aeab-93c47d27501b', 'sg_uuid_list': ['33cc7843-a8b2-4b63-ba30-a95791158dfd'], 'static_route_list': None, 'vm_project_uuid': '7b723255-fa49-4769-b023-38f28f67d984', 'admin_state': 'Enabled', 'flow_key_idx': '37', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-01-19 04:13:18,992 - DEBUG - Agent 10.0.0.65 vrf name: default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601:ctest-vn2-55794601 2026-01-19 04:13:19,000 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601:ctest-vn2-55794601', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601', 'table_label': '-1', 'vxlan_id': '17', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.192:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2026-01-19 04:13:19,081 - DEBUG - VM ctest-vn2_vm2-39790800 : Tap interface tap829d79ec-98 is set to Active 2026-01-19 04:13:19,081 - DEBUG - Tap interface tap829d79ec-98 detail : {'index': '4', 'name': 'tap829d79ec-98', 'uuid': '829d79ec-9889-4fd3-aeab-93c47d27501b', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601:ctest-vn2-55794601', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '17', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601', 'vm_uuid': '12c2d7af-d89f-4385-b6f1-0a09f2d56d17', 'vm_name': 'ctest-vn2_vm2-39790800', 'ip_addr': '192.168.2.251', 'mac_addr': '02:82:9d:79:ec:98', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '128', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-83017551:829d79ec-9889-4fd3-aeab-93c47d27501b', 'sg_uuid_list': ['33cc7843-a8b2-4b63-ba30-a95791158dfd'], 'static_route_list': None, 'vm_project_uuid': '7b723255-fa49-4769-b023-38f28f67d984', 'admin_state': 'Enabled', 'flow_key_idx': '37', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2026-01-19 04:13:19,197 - DEBUG - Starting Layer 2 verification in Agent 2026-01-19 04:13:19,213 - DEBUG - Layer 2 path is seen for VM MAC 02:82:9d:79:ec:98 in agent 10.0.0.65 2026-01-19 04:13:19,214 - DEBUG - Active layer 2 route in agent is present for VMI tap829d79ec-98 2026-01-19 04:13:19,214 - DEBUG - L2 label(34) matches bw route table and interface table 2026-01-19 04:13:19,226 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 is found in Agent of node 10.0.0.65 2026-01-19 04:13:19,363 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 is consistent in agent 10.0.0.65 2026-01-19 04:13:19,364 - DEBUG - Route for VM IP ['192.168.2.251'] is consistent in agent 10.0.0.65 2026-01-19 04:13:19,364 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 verification for VM ctest-vn2_vm2-39790800 in Agent 10.0.0.65 passed 2026-01-19 04:13:19,479 - DEBUG - Starting all layer 2 verification in agent 10.0.0.65 2026-01-19 04:13:19,493 - DEBUG - Route for VM MAC 02:82:9d:79:ec:98 is consistent in agent 10.0.0.65 2026-01-19 04:13:22,683 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms') 2026-01-19 04:13:22,683 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-39790800 failed! 2026-01-19 04:13:22,698 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 is 192.168.2.254 and allocation pool is NOT set 2026-01-19 04:13:26,776 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2026-01-19 04:13:26,776 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-39790800 failed! 2026-01-19 04:13:26,795 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 is 192.168.2.254 and allocation pool is NOT set 2026-01-19 04:13:30,875 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2026-01-19 04:13:30,876 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-39790800 failed! 2026-01-19 04:13:30,892 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 is 192.168.2.254 and allocation pool is NOT set 2026-01-19 04:13:34,962 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms') 2026-01-19 04:13:34,962 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-39790800 failed! 2026-01-19 04:13:34,978 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 is 192.168.2.254 and allocation pool is NOT set 2026-01-19 04:13:39,065 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1021ms') 2026-01-19 04:13:39,065 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-39790800 failed! 2026-01-19 04:13:39,082 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 is 192.168.2.254 and allocation pool is NOT set 2026-01-19 04:13:43,161 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms') 2026-01-19 04:13:43,161 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-39790800 failed! 2026-01-19 04:13:43,179 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 is 192.168.2.254 and allocation pool is NOT set 2026-01-19 04:13:47,256 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms') 2026-01-19 04:13:47,256 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-39790800 failed! 2026-01-19 04:13:47,274 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 is 192.168.2.254 and allocation pool is NOT set 2026-01-19 04:13:51,354 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2026-01-19 04:13:51,354 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-39790800 failed! 2026-01-19 04:13:51,372 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 is 192.168.2.254 and allocation pool is NOT set 2026-01-19 04:13:53,437 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=7.18 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=2.08 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 2.082/4.628/7.175/2.546 ms') 2026-01-19 04:13:53,437 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-39790800 passed 2026-01-19 04:13:53,437 - INFO - VM ctest-vn2_vm2-39790800 verifications in Compute nodes passed 2026-01-19 04:13:53,654 - DEBUG - Validated VM route 192.168.2.251 in vrouter of 10.0.0.65 2026-01-19 04:13:53,660 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2026-01-19 04:13:53,660 - INFO - Validated routes of VM ctest-vn2_vm2-39790800 in all vrouters 2026-01-19 04:13:54,095 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2026-01-19 04:13:54,223 - DEBUG - Layer2 route found for VM MAC 02:82:9d:79:ec:98 in Control-node 10.0.0.23 2026-01-19 04:13:54,223 - DEBUG - L2 Label for VM ctest-vn2_vm2-39790800 same between Control-node 10.0.0.23 and Agent, Expected: 34, Seen: 34 2026-01-19 04:13:54,223 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2026-01-19 04:13:54,357 - DEBUG - Layer2 route found for VM MAC 02:82:9d:79:ec:98 in Control-node 10.0.0.27 2026-01-19 04:13:54,357 - DEBUG - L2 Label for VM ctest-vn2_vm2-39790800 same between Control-node 10.0.0.27 and Agent, Expected: 34, Seen: 34 2026-01-19 04:13:54,357 - DEBUG - Starting all layer2 verification in 10.0.0.50 Control Node 2026-01-19 04:13:54,488 - DEBUG - Layer2 route found for VM MAC 02:82:9d:79:ec:98 in Control-node 10.0.0.50 2026-01-19 04:13:54,488 - DEBUG - L2 Label for VM ctest-vn2_vm2-39790800 same between Control-node 10.0.0.50 and Agent, Expected: 34, Seen: 34 2026-01-19 04:13:54,488 - INFO - Verification in Control-nodes for VM ctest-vn2_vm2-39790800 passed 2026-01-19 04:13:54,488 - DEBUG - Verifying the vm in opserver 2026-01-19 04:13:54,488 - DEBUG - Verifying in collector 10.0.0.23 ... 2026-01-19 04:13:54,488 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/12c2d7af-d89f-4385-b6f1-0a09f2d56d17?flat 2026-01-19 04:13:54,508 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-83017551:829d79ec-9889-4fd3-aeab-93c47d27501b?flat 2026-01-19 04:13:54,530 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-83017551:829d79ec-9889-4fd3-aeab-93c47d27501b?flat 2026-01-19 04:13:54,546 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/12c2d7af-d89f-4385-b6f1-0a09f2d56d17?flat 2026-01-19 04:13:54,564 - DEBUG - Verifying vm in vn uve 2026-01-19 04:13:54,564 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-83017551:829d79ec-9889-4fd3-aeab-93c47d27501b?flat 2026-01-19 04:13:54,581 - DEBUG - VM uve shows interface as {'ip4_active': True, 'active': True, 'mac_address': '02:82:9d:79:ec:98', 'ip6_active': False, 'vhostuser_mode': 0, 'uuid': '829d79ec-9889-4fd3-aeab-93c47d27501b', 'tx_vlan': 65535, 'is_health_check_active': True, 'rx_vlan': 65535, 'ip_address': '192.168.2.251', 'vm_uuid': '12c2d7af-d89f-4385-b6f1-0a09f2d56d17', 'l2_active': True, 'vm_name': 'ctest-vn2_vm2-39790800', 'fixed_ip4_list': ['192.168.2.251'], 'admin_state': True, 'ip6_address': '::', 'gateway': '192.168.2.254', '__T': 1768795996704821, 'port_mirror_enabled': False, 'vn_uuid': 'd47bd86e-aebf-48ab-86c6-ba296b4b455b', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601', 'label': 30} 2026-01-19 04:13:54,581 - DEBUG - VM uve shows ip address as ['192.168.2.251', '::'] 2026-01-19 04:13:54,581 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 2026-01-19 04:13:54,581 - INFO - Verifying through opserver in 10.0.0.27 2026-01-19 04:13:54,581 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601?flat 2026-01-19 04:13:54,601 - DEBUG - expected vm list 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 2026-01-19 04:13:54,601 - DEBUG - Extracted vm list ['12c2d7af-d89f-4385-b6f1-0a09f2d56d17'] 2026-01-19 04:13:54,601 - INFO - VM 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 is present in default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 2026-01-19 04:13:54,601 - DEBUG - Verifying vm in vrouter uve 2026-01-19 04:13:54,601 - DEBUG - Getting info from collector 10.0.0.23.. 2026-01-19 04:13:54,602 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/12c2d7af-d89f-4385-b6f1-0a09f2d56d17?flat 2026-01-19 04:13:54,620 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-5040-2'] 2026-01-19 04:13:54,620 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-2:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:13:54,670 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-2?flat 2026-01-19 04:13:54,705 - DEBUG - VM 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 is present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:13:54,706 - DEBUG - Validated that VM ctest-vn2_vm2-39790800 is in Vrouter an-jenkins-deploy-platform-ansible-os-5040-2 UVE 2026-01-19 04:13:54,706 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2026-01-19 04:13:54,713 - DEBUG - Expected tap interface of VM uuid 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 is default-domain:ctest-TestBasicVMVN0-83017551:829d79ec-9889-4fd3-aeab-93c47d27501b 2026-01-19 04:13:54,713 - DEBUG - Expected VN of VM uuid 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 is default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 2026-01-19 04:13:54,713 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-2:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:13:54,767 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-2?flat 2026-01-19 04:13:54,795 - DEBUG - VM 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 is present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:13:54,796 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-83017551:829d79ec-9889-4fd3-aeab-93c47d27501b of vm 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 is present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:13:54,796 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 of vm 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 is present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:13:54,796 - DEBUG - Verifying the 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 virtual network link through opserver 10.0.0.27 2026-01-19 04:13:54,796 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-01-19 04:13:54,808 - DEBUG - VM link and name as {'name': '12c2d7af-d89f-4385-b6f1-0a09f2d56d17', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/12c2d7af-d89f-4385-b6f1-0a09f2d56d17?flat'} 2026-01-19 04:13:54,808 - DEBUG - Verifying the 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 virtual network link through opserver 10.0.0.50 2026-01-19 04:13:54,808 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2026-01-19 04:13:54,821 - DEBUG - VM link and name as {'name': '12c2d7af-d89f-4385-b6f1-0a09f2d56d17', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-machine/12c2d7af-d89f-4385-b6f1-0a09f2d56d17?flat'} 2026-01-19 04:13:54,821 - DEBUG - Verifying the 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 virtual network link through opserver 10.0.0.23 2026-01-19 04:13:54,821 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-01-19 04:13:54,836 - DEBUG - VM link and name as {'name': '12c2d7af-d89f-4385-b6f1-0a09f2d56d17', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/12c2d7af-d89f-4385-b6f1-0a09f2d56d17?flat'} 2026-01-19 04:13:54,836 - INFO - VM ctest-vn2_vm2-39790800 validations in Opserver passed 2026-01-19 04:13:54,836 - INFO - Waiting for VM ctest-vn1_vm2-42009684 to be up.. 2026-01-19 04:13:54,836 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:13:54,836 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/ba553e1d-1e39-4c13-8a40-8eba58c297df 2026-01-19 04:13:54,844 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/fabcddbe-b09b-44ea-b71f-1a1f5e7c836f 2026-01-19 04:13:56,013 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=3.55 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.539 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 0.539/2.044/3.549/1.505 ms') 2026-01-19 04:13:56,013 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-42009684 passed 2026-01-19 04:13:56,085 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:13:56,085 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-42009684, IP 192.168.1.251, Port 22 2026-01-19 04:13:56,258 - DEBUG - VM ctest-vn1_vm2-42009684 is ready for SSH connections 2026-01-19 04:13:56,259 - INFO - Waiting for VM ctest-vn2_vm2-39790800 to be up.. 2026-01-19 04:13:56,259 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:13:56,259 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/12c2d7af-d89f-4385-b6f1-0a09f2d56d17 2026-01-19 04:13:56,268 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/829d79ec-9889-4fd3-aeab-93c47d27501b 2026-01-19 04:13:57,427 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=2.42 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=2.83 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 2.422/2.625/2.828/0.203 ms') 2026-01-19 04:13:57,427 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-39790800 passed 2026-01-19 04:13:57,500 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:13:57,500 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-39790800, IP 192.168.2.251, Port 22 2026-01-19 04:13:57,557 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-01-19 04:13:57,649 - DEBUG - VM ctest-vn2_vm2-39790800 is NOT ready for SSH connections, VM status: ACTIVE 2026-01-19 04:14:02,650 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:14:02,650 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-39790800, IP 192.168.2.251, Port 22 2026-01-19 04:14:02,708 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-01-19 04:14:02,795 - DEBUG - VM ctest-vn2_vm2-39790800 is NOT ready for SSH connections, VM status: ACTIVE 2026-01-19 04:14:07,796 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:14:07,796 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-39790800, IP 192.168.2.251, Port 22 2026-01-19 04:14:07,863 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2026-01-19 04:14:07,951 - DEBUG - VM ctest-vn2_vm2-39790800 is NOT ready for SSH connections, VM status: ACTIVE 2026-01-19 04:14:12,952 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:14:12,952 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-39790800, IP 192.168.2.251, Port 22 2026-01-19 04:14:13,120 - DEBUG - VM ctest-vn2_vm2-39790800 is ready for SSH connections 2026-01-19 04:14:13,120 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2026-01-19 04:14:13,120 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.251, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.249, gateway password: c0ntrail123 2026-01-19 04:14:13,120 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.251 2026-01-19 04:14:51,849 - DEBUG - PING 192.168.2.251 (192.168.2.251) 56(84) bytes of data. 64 bytes from 192.168.2.251: icmp_req=1 ttl=63 time=7.79 ms 64 bytes from 192.168.2.251: icmp_req=2 ttl=63 time=1.78 ms 64 bytes from 192.168.2.251: icmp_req=3 ttl=63 time=1.37 ms --- 192.168.2.251 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.377/3.652/7.799/2.937 ms 2026-01-19 04:14:51,849 - INFO - Ping to IP 192.168.2.251 from VM ctest-vn1_vm2-42009684 passed 2026-01-19 04:14:51,849 - INFO - Deleting VM ctest-vn2_vm2-39790800 2026-01-19 04:14:51,922 - DEBUG - Verifying in api server 10.0.0.23 2026-01-19 04:14:51,922 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/12c2d7af-d89f-4385-b6f1-0a09f2d56d17 2026-01-19 04:14:51,932 - DEBUG - VM ID 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 of VM ctest-vn2_vm2-39790800 is still found in API Server 2026-01-19 04:14:53,932 - DEBUG - Verifying in api server 10.0.0.23 2026-01-19 04:14:53,932 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/12c2d7af-d89f-4385-b6f1-0a09f2d56d17 2026-01-19 04:14:53,940 - DEBUG - Response Code: 404 2026-01-19 04:14:53,941 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/12c2d7af-d89f-4385-b6f1-0a09f2d56d17 2026-01-19 04:14:53,948 - DEBUG - Response Code: 404 2026-01-19 04:14:53,948 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/12c2d7af-d89f-4385-b6f1-0a09f2d56d17 2026-01-19 04:14:53,954 - DEBUG - Response Code: 404 2026-01-19 04:14:53,954 - INFO - VM ctest-vn2_vm2-39790800 is fully removed in API-Server 2026-01-19 04:14:54,111 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2026-01-19 04:14:54,147 - DEBUG - Validated that vrouter 10.0.0.249 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 2026-01-19 04:14:54,283 - DEBUG - VM route 192.168.2.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 of compute 10.0.0.65 2026-01-19 04:14:54,289 - DEBUG - Validated that vrouter 10.0.0.65 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 2026-01-19 04:14:54,289 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 2026-01-19 04:14:54,289 - INFO - VM ctest-vn2_vm2-39790800 is removed in Compute, and routes are removed in all compute nodes 2026-01-19 04:14:54,517 - INFO - Routes for VM ctest-vn2_vm2-39790800 is removed in all control-nodes 2026-01-19 04:14:54,518 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-2:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:14:54,569 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-2?flat 2026-01-19 04:14:54,599 - DEBUG - VM 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:14:54,599 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:14:54,600 - DEBUG - interface for vm 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 deleted from vrouter uve ... 2026-01-19 04:14:54,600 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:14:57,600 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-2:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:14:57,653 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-2?flat 2026-01-19 04:14:57,690 - DEBUG - VM 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:14:57,691 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:14:57,691 - DEBUG - interface for vm 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 deleted from vrouter uve ... 2026-01-19 04:14:57,691 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:15:00,691 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-2:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:15:00,775 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-2?flat 2026-01-19 04:15:00,816 - DEBUG - VM 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:15:00,816 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:15:00,816 - DEBUG - interface for vm 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 deleted from vrouter uve ... 2026-01-19 04:15:00,816 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:15:03,817 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-2:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:15:03,887 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-2?flat 2026-01-19 04:15:03,926 - DEBUG - VM 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:15:03,926 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:15:03,926 - DEBUG - interface for vm 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 deleted from vrouter uve ... 2026-01-19 04:15:03,926 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:15:06,927 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-2:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:15:06,987 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-2?flat 2026-01-19 04:15:07,023 - DEBUG - VM 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:15:07,023 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:15:07,023 - DEBUG - interface for vm 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 deleted from vrouter uve ... 2026-01-19 04:15:07,023 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:15:10,024 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-2:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:15:10,084 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-2?flat 2026-01-19 04:15:10,121 - DEBUG - VM 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:15:10,121 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:15:10,121 - DEBUG - interface for vm 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 deleted from vrouter uve ... 2026-01-19 04:15:10,121 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:15:13,122 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-2:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:15:13,173 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-2?flat 2026-01-19 04:15:13,203 - DEBUG - VM 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:15:13,203 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:15:13,203 - DEBUG - interface for vm 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 deleted from vrouter uve ... 2026-01-19 04:15:13,203 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:15:16,203 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-2:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:15:16,266 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-2?flat 2026-01-19 04:15:16,297 - DEBUG - VM 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:15:16,297 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:15:16,297 - DEBUG - interface for vm 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 deleted from vrouter uve ... 2026-01-19 04:15:16,297 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:15:19,298 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-2:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:15:19,356 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-2?flat 2026-01-19 04:15:19,390 - DEBUG - VM 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 is not present in vrouter an-jenkins-deploy-platform-ansible-os-5040-2 uve 2026-01-19 04:15:19,390 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:15:19,390 - DEBUG - interface for vm 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 deleted from vrouter uve ... 2026-01-19 04:15:19,390 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:15:19,390 - DEBUG - Validated that VM 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 is removed in Vrouter UVE 2026-01-19 04:15:19,390 - DEBUG - Verifying through opserver in 10.0.0.27 2026-01-19 04:15:19,390 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601?flat 2026-01-19 04:15:19,417 - DEBUG - 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 BM not in default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn2-55794601 uve 2026-01-19 04:15:19,417 - DEBUG - Validated that VM 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 is not present in the VN UVE 2026-01-19 04:15:19,417 - DEBUG - Verifying the 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 virtual network link through opserver 10.0.0.27 2026-01-19 04:15:19,417 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-01-19 04:15:19,436 - INFO - 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 vm uve deleted from opserver 2026-01-19 04:15:19,436 - DEBUG - Verifying the 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 virtual network link through opserver 10.0.0.50 2026-01-19 04:15:19,436 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2026-01-19 04:15:19,450 - INFO - 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 vm uve deleted from opserver 2026-01-19 04:15:19,450 - DEBUG - Verifying the 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 virtual network link through opserver 10.0.0.23 2026-01-19 04:15:19,450 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-01-19 04:15:19,463 - INFO - 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 vm uve deleted from opserver 2026-01-19 04:15:19,463 - INFO - Validated that VM 12c2d7af-d89f-4385-b6f1-0a09f2d56d17 is removed from Opserver 2026-01-19 04:15:19,463 - INFO - Deleting VM ctest-vn1_vm2-42009684 2026-01-19 04:15:19,575 - DEBUG - Verifying in api server 10.0.0.23 2026-01-19 04:15:19,575 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/ba553e1d-1e39-4c13-8a40-8eba58c297df 2026-01-19 04:15:19,590 - DEBUG - VM ID ba553e1d-1e39-4c13-8a40-8eba58c297df of VM ctest-vn1_vm2-42009684 is still found in API Server 2026-01-19 04:15:21,590 - DEBUG - Verifying in api server 10.0.0.23 2026-01-19 04:15:21,591 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/ba553e1d-1e39-4c13-8a40-8eba58c297df 2026-01-19 04:15:21,599 - DEBUG - Response Code: 404 2026-01-19 04:15:21,599 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/ba553e1d-1e39-4c13-8a40-8eba58c297df 2026-01-19 04:15:21,605 - DEBUG - Response Code: 404 2026-01-19 04:15:21,605 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/ba553e1d-1e39-4c13-8a40-8eba58c297df 2026-01-19 04:15:21,613 - DEBUG - Response Code: 404 2026-01-19 04:15:21,613 - INFO - VM ctest-vn1_vm2-42009684 is fully removed in API-Server 2026-01-19 04:15:21,774 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2026-01-19 04:15:21,928 - DEBUG - VM route 192.168.1.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 of compute 10.0.0.249 2026-01-19 04:15:21,933 - DEBUG - Validated that vrouter 10.0.0.249 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 2026-01-19 04:15:21,933 - DEBUG - Validated that vrouter 10.0.0.65 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 2026-01-19 04:15:21,933 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 2026-01-19 04:15:21,933 - INFO - VM ctest-vn1_vm2-42009684 is removed in Compute, and routes are removed in all compute nodes 2026-01-19 04:15:22,167 - INFO - Routes for VM ctest-vn1_vm2-42009684 is removed in all control-nodes 2026-01-19 04:15:22,167 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-1:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:15:22,225 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-1?flat 2026-01-19 04:15:22,256 - DEBUG - VM ba553e1d-1e39-4c13-8a40-8eba58c297df is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-1 uve 2026-01-19 04:15:22,256 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:15:22,256 - DEBUG - interface for vm ba553e1d-1e39-4c13-8a40-8eba58c297df deleted from vrouter uve ... 2026-01-19 04:15:22,256 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:15:25,256 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-1:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:15:25,301 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-1?flat 2026-01-19 04:15:25,340 - DEBUG - VM ba553e1d-1e39-4c13-8a40-8eba58c297df is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-1 uve 2026-01-19 04:15:25,340 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:15:25,340 - DEBUG - interface for vm ba553e1d-1e39-4c13-8a40-8eba58c297df deleted from vrouter uve ... 2026-01-19 04:15:25,340 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:15:28,341 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-1:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:15:28,386 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-1?flat 2026-01-19 04:15:28,417 - DEBUG - VM ba553e1d-1e39-4c13-8a40-8eba58c297df is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-1 uve 2026-01-19 04:15:28,417 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:15:28,417 - DEBUG - interface for vm ba553e1d-1e39-4c13-8a40-8eba58c297df deleted from vrouter uve ... 2026-01-19 04:15:28,417 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:15:31,417 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-1:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:15:31,511 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-1?flat 2026-01-19 04:15:31,542 - DEBUG - VM ba553e1d-1e39-4c13-8a40-8eba58c297df is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-1 uve 2026-01-19 04:15:31,542 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:15:31,542 - DEBUG - interface for vm ba553e1d-1e39-4c13-8a40-8eba58c297df deleted from vrouter uve ... 2026-01-19 04:15:31,542 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:15:34,543 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-1:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:15:34,585 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-1?flat 2026-01-19 04:15:34,619 - DEBUG - VM ba553e1d-1e39-4c13-8a40-8eba58c297df is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-1 uve 2026-01-19 04:15:34,619 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:15:34,619 - DEBUG - interface for vm ba553e1d-1e39-4c13-8a40-8eba58c297df deleted from vrouter uve ... 2026-01-19 04:15:34,619 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:15:37,619 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-1:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:15:37,664 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-1?flat 2026-01-19 04:15:37,701 - DEBUG - VM ba553e1d-1e39-4c13-8a40-8eba58c297df is still present in vrouter an-jenkins-deploy-platform-ansible-os-5040-1 uve 2026-01-19 04:15:37,701 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:15:37,701 - DEBUG - interface for vm ba553e1d-1e39-4c13-8a40-8eba58c297df deleted from vrouter uve ... 2026-01-19 04:15:37,701 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:15:40,702 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-5040-1:Compute:contrail-vrouter-agent:0?flat 2026-01-19 04:15:40,745 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-5040-1?flat 2026-01-19 04:15:40,777 - DEBUG - VM ba553e1d-1e39-4c13-8a40-8eba58c297df is not present in vrouter an-jenkins-deploy-platform-ansible-os-5040-1 uve 2026-01-19 04:15:40,777 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2026-01-19 04:15:40,777 - DEBUG - interface for vm ba553e1d-1e39-4c13-8a40-8eba58c297df deleted from vrouter uve ... 2026-01-19 04:15:40,777 - DEBUG - Deleted interface not in error interface list ... 2026-01-19 04:15:40,777 - DEBUG - Validated that VM ba553e1d-1e39-4c13-8a40-8eba58c297df is removed in Vrouter UVE 2026-01-19 04:15:40,777 - DEBUG - Verifying through opserver in 10.0.0.27 2026-01-19 04:15:40,778 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583?flat 2026-01-19 04:15:40,800 - DEBUG - ba553e1d-1e39-4c13-8a40-8eba58c297df BM not in default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 uve 2026-01-19 04:15:40,800 - DEBUG - Validated that VM ba553e1d-1e39-4c13-8a40-8eba58c297df is not present in the VN UVE 2026-01-19 04:15:40,800 - DEBUG - Verifying the ba553e1d-1e39-4c13-8a40-8eba58c297df virtual network link through opserver 10.0.0.27 2026-01-19 04:15:40,800 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2026-01-19 04:15:40,811 - INFO - ba553e1d-1e39-4c13-8a40-8eba58c297df vm uve deleted from opserver 2026-01-19 04:15:40,811 - DEBUG - Verifying the ba553e1d-1e39-4c13-8a40-8eba58c297df virtual network link through opserver 10.0.0.50 2026-01-19 04:15:40,811 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2026-01-19 04:15:40,823 - INFO - ba553e1d-1e39-4c13-8a40-8eba58c297df vm uve deleted from opserver 2026-01-19 04:15:40,823 - DEBUG - Verifying the ba553e1d-1e39-4c13-8a40-8eba58c297df virtual network link through opserver 10.0.0.23 2026-01-19 04:15:40,823 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2026-01-19 04:15:40,836 - INFO - ba553e1d-1e39-4c13-8a40-8eba58c297df vm uve deleted from opserver 2026-01-19 04:15:40,836 - INFO - Validated that VM ba553e1d-1e39-4c13-8a40-8eba58c297df is removed from Opserver 2026-01-19 04:15:40,836 - INFO - Deleting VN ctest-vn3-83621045 2026-01-19 04:15:40,972 - INFO - Deleting VM ctest-vn2_vm1-35050972 2026-01-19 04:15:41,070 - INFO - Deleting VM ctest-vn1_vm1-30025374 2026-01-19 04:15:41,319 - INFO - Deleting VN ctest-vn2-55794601 2026-01-19 04:15:41,375 - DEBUG - RefsExistError Delete when resource still referred: ['http://10.0.0.23:8082/instance-ip/fa049818-5b0d-4833-b467-23cdaaa75879'] while deleting VN ctest-vn2-55794601..Will retry 2026-01-19 04:15:43,500 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/977aced7-e3e1-4139-b0f0-474a734fd8ac 2026-01-19 04:15:43,505 - DEBUG - Response Code: 404 2026-01-19 04:15:43,505 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-19 04:15:43,511 - DEBUG - Requesting: http://10.0.0.27:8082/domain/bc123c74-c45e-4d7a-a955-76f59d7a4cc7 2026-01-19 04:15:43,526 - DEBUG - Requesting: http://10.0.0.27:8082/project/7b723255-fa49-4769-b023-38f28f67d984 2026-01-19 04:15:43,614 - INFO - Validated that VN ctest-vn2-55794601 is not found in API Server 2026-01-19 04:15:43,625 - DEBUG - VN ctest-vn2-55794601 is not present in Agent 10.0.0.249 2026-01-19 04:15:43,636 - DEBUG - VN ctest-vn2-55794601 is not present in Agent 10.0.0.65 2026-01-19 04:15:43,636 - INFO - Validated that VN ctest-vn2-55794601 is not in any agent 2026-01-19 04:15:43,671 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn2-55794601 info 2026-01-19 04:15:43,837 - INFO - Deleting VN ctest-vn1-56688583 2026-01-19 04:15:43,974 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/57e98c3f-a527-4240-beea-90b4a26bc842 2026-01-19 04:15:43,979 - DEBUG - Response Code: 404 2026-01-19 04:15:43,980 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2026-01-19 04:15:43,986 - DEBUG - Requesting: http://10.0.0.27:8082/domain/bc123c74-c45e-4d7a-a955-76f59d7a4cc7 2026-01-19 04:15:44,006 - DEBUG - Requesting: http://10.0.0.27:8082/project/7b723255-fa49-4769-b023-38f28f67d984 2026-01-19 04:15:44,097 - INFO - Validated that VN ctest-vn1-56688583 is not found in API Server 2026-01-19 04:15:44,109 - DEBUG - VN ctest-vn1-56688583 is not present in Agent 10.0.0.249 2026-01-19 04:15:44,123 - DEBUG - VN ctest-vn1-56688583 is not present in Agent 10.0.0.65 2026-01-19 04:15:44,123 - INFO - Validated that VN ctest-vn1-56688583 is not in any agent 2026-01-19 04:15:44,132 - DEBUG - VRF 2 is not seen in agent 10.0.0.249 2026-01-19 04:15:44,139 - DEBUG - Vrouter 10.0.0.249 does not have vrf 2 for VN ctest-vn1-56688583 2026-01-19 04:15:44,139 - INFO - Validated that all vrouters do not have the route table for VN default-domain:ctest-TestBasicVMVN0-83017551:ctest-vn1-56688583 2026-01-19 04:15:44,166 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-56688583 info 2026-01-19 04:15:44,202 - INFO - Deleted policy policy2 2026-01-19 04:15:44,251 - INFO - Deleted policy policy1 2026-01-19 04:15:45,509 - DEBUG - Skipping xmpp flap check

2026-01-19 04:15:45,509 - INFO - END TEST : test_process_restart_in_policy_between_vns : PASSED[0:06:11]

2026-01-19 04:15:45,509 - INFO - -------------------------------------------------------------------------------- 2026-01-19 04:15:45,513 - INFO - ================================================================================ 2026-01-19 04:15:45,513 - INFO - STARTING TEST : test_underlay_broadcast_traffic_handling 2026-01-19 04:15:45,513 - INFO - TEST DESCRIPTION : Test the underlay brocast traffic handling by vrouter. (Bug-1545229). 1. Send broadcast traffic from one compute node. 2. Other compute in same subnet should receive that traffic. 3. Receiving compute should treat this traffic as underlay. 4. Compute should not replicate the packet and send the copy back. Pass criteria: Step 3-4 should pass Maintainer : chhandak@juniper.net 2026-01-19 04:15:46,804 - DEBUG - Skipping xmpp flap check 2026-01-19 04:15:46,805 - INFO - Initial checks done. Running the testcase now 2026-01-19 04:15:46,805 - INFO - 2026-01-19 04:15:47,195 - INFO - Agent interface name: ens6 2026-01-19 04:15:47,280 - DEBUG - Executing command: sudo tcpdump -nni ens6 -U host 10.20.0.255 -w /tmp/ens6_ctest-random-78964998.pcap 2026-01-19 04:15:52,636 - INFO - Pinging broacast address 10.20.0.255 from compute 10.20.0.18 2026-01-19 04:16:06,940 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2026-01-19 04:16:09,079 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-78964998.pcap | grep -c length 2026-01-19 04:16:09,092 - DEBUG - STDOUT: 10 2026-01-19 04:16:09,092 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-78964998.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2026-01-19 04:16:09,092 - INFO - 10 packets are found in tcpdump output as expected 2026-01-19 04:16:09,093 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2026-01-19 04:16:11,139 - INFO - Packet count matched: Compute 10.0.0.65 has receive only 10 packet from source IP 10.20.0.255. No duplicate packet seen 2026-01-19 04:16:12,421 - DEBUG - Skipping xmpp flap check

2026-01-19 04:16:12,421 - INFO - END TEST : test_underlay_broadcast_traffic_handling : PASSED[0:00:27]

2026-01-19 04:16:12,421 - INFO - -------------------------------------------------------------------------------- 2026-01-19 04:16:13,179 - INFO - Deleted project: ctest-TestBasicVMVN0-83017551, ID : 7b723255-fa49-4769-b023-38f28f67d984