2025-12-25 04:35:28,659 - INFO - Domain Default found not creating 2025-12-25 04:35:28,876 - INFO - Project ctest-TestBasicVMVN0-19229946 not found, creating it 2025-12-25 04:35:29,464 - INFO - Created Project:ctest-TestBasicVMVN0-19229946, ID : edba148a-64cc-4e4a-8216-4c7dbc6a7d87 2025-12-25 04:35:31,391 - INFO - ================================================================================ 2025-12-25 04:35:31,391 - INFO - STARTING TEST : test_control_node_switchover 2025-12-25 04:35:31,391 - INFO - TEST DESCRIPTION : Stop the control node and check peering with agent fallback to other control node. 1. Pick one VN from respource pool which has 2 VM's in it 2. Verify ping between VM's 3. Find active control node in cluster by agent inspect 4. Stop control service on active control node 5. Verify agents are connected to new active control-node using xmpp connections 6. Bring back control service on previous active node 7. Verify ping between VM's again after bringing up control serveice Pass criteria: Step 2,5 and 7 should pass 2025-12-25 04:35:32,684 - DEBUG - Skipping xmpp flap check 2025-12-25 04:35:32,684 - INFO - Initial checks done. Running the testcase now 2025-12-25 04:35:32,684 - INFO - 2025-12-25 04:35:33,532 - DEBUG - Response for create_network : {'network': {'id': '57c4ee51-9600-48e5-8782-069f92e5e892', 'name': 'ctest-vn1-88877144', 'tenant_id': 'edba148a64cc4e4a82164c7dbc6a7d87', 'project_id': 'edba148a64cc4e4a82164c7dbc6a7d87', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestBasicVMVN0-19229946', 'ctest-vn1-88877144'], 'port_security_enabled': True, 'description': ''}} 2025-12-25 04:35:33,899 - DEBUG - Response for create_subnet : {'subnet': {'id': 'add82984-cf97-4cc8-9aac-d4c4887da2ad', 'name': '', 'tenant_id': 'edba148a64cc4e4a82164c7dbc6a7d87', 'network_id': '57c4ee51-9600-48e5-8782-069f92e5e892', 'ip_version': 4, 'cidr': '192.168.1.0/24', 'allocation_pools': [{'start': '192.168.1.2', 'end': '192.168.1.254'}], 'gateway_ip': '192.168.1.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '192.168.1.2', 'tags': [], 'project_id': 'edba148a64cc4e4a82164c7dbc6a7d87'}} 2025-12-25 04:35:33,931 - INFO - Created VN ctest-vn1-88877144 2025-12-25 04:35:33,945 - DEBUG - VN ctest-vn1-88877144 UUID is 57c4ee51-9600-48e5-8782-069f92e5e892 2025-12-25 04:35:34,044 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-12-25 04:35:34,443 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-12-25 04:35:34,521 - DEBUG - Requesting: http://10.0.0.27:8082/domain/d4f5d7ea-1779-482f-9e93-03873d431e99 2025-12-25 04:35:34,549 - DEBUG - Requesting: http://10.0.0.27:8082/project/edba148a-64cc-4e4a-8216-4c7dbc6a7d87 2025-12-25 04:35:34,686 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/57c4ee51-9600-48e5-8782-069f92e5e892 2025-12-25 04:35:34,709 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/57c4ee51-9600-48e5-8782-069f92e5e892 2025-12-25 04:35:34,729 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/fb8238c0-da0c-41d3-8200-d4820d154826 2025-12-25 04:35:34,743 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/fb8238c0-da0c-41d3-8200-d4820d154826 2025-12-25 04:35:34,754 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/9a407557-cf8e-4c3c-b528-c95db06117a2 2025-12-25 04:35:34,765 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-25 04:35:34,765 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/57c4ee51-9600-48e5-8782-069f92e5e892 2025-12-25 04:35:34,777 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/fb8238c0-da0c-41d3-8200-d4820d154826 2025-12-25 04:35:34,799 - INFO - Verified VN network id 15 for VN 57c4ee51-9600-48e5-8782-069f92e5e892 2025-12-25 04:35:34,800 - INFO - Verifications in API Server for VN ctest-vn1-88877144 passed 2025-12-25 04:35:34,800 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/57c4ee51-9600-48e5-8782-069f92e5e892 2025-12-25 04:35:34,815 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/fb8238c0-da0c-41d3-8200-d4820d154826 2025-12-25 04:35:34,824 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/fb8238c0-da0c-41d3-8200-d4820d154826 2025-12-25 04:35:34,835 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/9a407557-cf8e-4c3c-b528-c95db06117a2 2025-12-25 04:35:34,854 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-19229946', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '6324441810911381733', 'uuid-lslong': '9764374224526764178'}, 'enable': 'true', 'created': '2025-12-25T04:35:33', 'last-modified': '2025-12-25T04:35:33', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:01.019581'} 2025-12-25 04:35:34,864 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-25 04:35:34,871 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-19229946', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '6324441810911381733', 'uuid-lslong': '9764374224526764178'}, 'enable': 'true', 'created': '2025-12-25T04:35:33', 'last-modified': '2025-12-25T04:35:33', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:01.041369'} 2025-12-25 04:35:34,878 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-25 04:35:34,884 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-19229946', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '6324441810911381733', 'uuid-lslong': '9764374224526764178'}, 'enable': 'true', 'created': '2025-12-25T04:35:33', 'last-modified': '2025-12-25T04:35:33', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:01.053053'} 2025-12-25 04:35:34,892 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-25 04:35:34,892 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-88877144 passed 2025-12-25 04:35:34,893 - DEBUG - ====Verifying policy data for ctest-vn1-88877144 in API_Server ====== 2025-12-25 04:35:34,893 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-12-25 04:35:34,901 - DEBUG - Requesting: http://10.0.0.27:8082/domain/d4f5d7ea-1779-482f-9e93-03873d431e99 2025-12-25 04:35:34,922 - DEBUG - Requesting: http://10.0.0.27:8082/project/edba148a-64cc-4e4a-8216-4c7dbc6a7d87 2025-12-25 04:35:35,056 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/57c4ee51-9600-48e5-8782-069f92e5e892 2025-12-25 04:35:35,068 - DEBUG - =>VN ctest-vn1-88877144 has no policy to be verified 2025-12-25 04:35:35,068 - DEBUG - Verifying the vn in opserver 2025-12-25 04:35:35,068 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 virtual network link through opserver 10.0.0.27 2025-12-25 04:35:35,069 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-12-25 04:35:35,425 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-12-25 04:35:35,518 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144?flat'} 2025-12-25 04:35:35,519 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 virtual network link through opserver 10.0.0.33 2025-12-25 04:35:35,519 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2025-12-25 04:35:35,891 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2025-12-25 04:35:35,983 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144?flat'} 2025-12-25 04:35:35,983 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 virtual network link through opserver 10.0.0.23 2025-12-25 04:35:35,984 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-12-25 04:35:36,360 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-12-25 04:35:36,449 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144?flat'} 2025-12-25 04:35:36,449 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 is found in opserver 2025-12-25 04:35:36,476 - DEBUG - Do not have enough data to verify VN in agent 2025-12-25 04:35:36,488 - DEBUG - VRF ids for VN ctest-vn1-88877144: {} 2025-12-25 04:35:36,570 - DEBUG - Services list from nova: [, , , ] 2025-12-25 04:35:37,908 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4927-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4927-1) 2025-12-25 04:35:39,529 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4927-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4927-2) 2025-12-25 04:35:39,529 - INFO - Waiting for VM ctest-vn1_vm1-30118421 to be up.. 2025-12-25 04:35:39,645 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-12-25 04:35:44,751 - DEBUG - VM is in ACTIVE state now 2025-12-25 04:35:44,751 - INFO - VM name : ctest-vn1_vm1-30118421 2025-12-25 04:35:44,854 - DEBUG - VM ctest-vn1_vm1-30118421 ID is 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 2025-12-25 04:35:44,887 - DEBUG - VM ctest-vn1_vm1-30118421 launched on Node an-jenkins-deploy-platform-ansible-os-4927-1 2025-12-25 04:35:45,027 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 2025-12-25 04:35:45,386 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 2025-12-25 04:35:45,432 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/b5858679-20bd-4ad0-b32d-efc7de28bd4e 2025-12-25 04:35:48,628 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-12-25 04:35:48,628 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-30118421 failed! 2025-12-25 04:35:48,647 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 is 192.168.1.1 and allocation pool is NOT set 2025-12-25 04:35:52,719 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1012ms') 2025-12-25 04:35:52,719 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-30118421 failed! 2025-12-25 04:35:52,736 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 is 192.168.1.1 and allocation pool is NOT set 2025-12-25 04:35:56,822 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1020ms') 2025-12-25 04:35:56,822 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-30118421 failed! 2025-12-25 04:35:56,846 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 is 192.168.1.1 and allocation pool is NOT set 2025-12-25 04:36:00,921 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1006ms') 2025-12-25 04:36:00,921 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-30118421 failed! 2025-12-25 04:36:00,941 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 is 192.168.1.1 and allocation pool is NOT set 2025-12-25 04:36:05,020 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1009ms') 2025-12-25 04:36:05,020 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-30118421 failed! 2025-12-25 04:36:05,040 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 is 192.168.1.1 and allocation pool is NOT set 2025-12-25 04:36:09,108 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1004ms') 2025-12-25 04:36:09,109 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-30118421 failed! 2025-12-25 04:36:09,131 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 is 192.168.1.1 and allocation pool is NOT set 2025-12-25 04:36:13,208 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1006ms') 2025-12-25 04:36:13,208 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-30118421 failed! 2025-12-25 04:36:13,225 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 is 192.168.1.1 and allocation pool is NOT set 2025-12-25 04:36:17,303 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms') 2025-12-25 04:36:17,303 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-30118421 failed! 2025-12-25 04:36:17,321 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 is 192.168.1.1 and allocation pool is NOT set 2025-12-25 04:36:21,392 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1009ms') 2025-12-25 04:36:21,392 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-30118421 failed! 2025-12-25 04:36:21,408 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 is 192.168.1.1 and allocation pool is NOT set 2025-12-25 04:36:25,494 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2025-12-25 04:36:25,494 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-30118421 failed! 2025-12-25 04:36:25,511 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 is 192.168.1.1 and allocation pool is NOT set 2025-12-25 04:36:29,590 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms') 2025-12-25 04:36:29,590 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-30118421 failed! 2025-12-25 04:36:29,608 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 is 192.168.1.1 and allocation pool is NOT set 2025-12-25 04:36:33,686 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1007ms') 2025-12-25 04:36:33,686 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-30118421 failed! 2025-12-25 04:36:33,703 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 is 192.168.1.1 and allocation pool is NOT set 2025-12-25 04:36:35,771 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=7.64 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=2.20 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 2.200/4.921/7.643/2.721 ms') 2025-12-25 04:36:35,771 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-30118421 passed 2025-12-25 04:36:35,842 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:36:35,843 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-30118421, IP 192.168.1.3, Port 22 2025-12-25 04:36:35,912 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-25 04:36:36,025 - DEBUG - VM ctest-vn1_vm1-30118421 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-25 04:36:41,026 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:36:41,026 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-30118421, IP 192.168.1.3, Port 22 2025-12-25 04:36:41,095 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-25 04:36:41,212 - DEBUG - VM ctest-vn1_vm1-30118421 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-25 04:36:46,213 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:36:46,213 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-30118421, IP 192.168.1.3, Port 22 2025-12-25 04:36:46,279 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-25 04:36:46,384 - DEBUG - VM ctest-vn1_vm1-30118421 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-25 04:36:51,386 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:36:51,386 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-30118421, IP 192.168.1.3, Port 22 2025-12-25 04:36:51,452 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-25 04:36:51,558 - DEBUG - VM ctest-vn1_vm1-30118421 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-25 04:36:56,559 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:36:56,559 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-30118421, IP 192.168.1.3, Port 22 2025-12-25 04:36:56,743 - DEBUG - VM ctest-vn1_vm1-30118421 is ready for SSH connections 2025-12-25 04:36:56,743 - INFO - Waiting for VM ctest-vn1_vm2-47382891 to be up.. 2025-12-25 04:36:56,889 - DEBUG - VM is in ACTIVE state now 2025-12-25 04:36:56,890 - INFO - VM name : ctest-vn1_vm2-47382891 2025-12-25 04:36:56,997 - DEBUG - VM ctest-vn1_vm2-47382891 ID is 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d 2025-12-25 04:36:56,997 - DEBUG - VM ctest-vn1_vm2-47382891 launched on Node an-jenkins-deploy-platform-ansible-os-4927-2 2025-12-25 04:36:57,106 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d 2025-12-25 04:36:57,140 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/2fa36043-ce42-4433-8555-5571fdbe0f18 2025-12-25 04:37:00,325 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms') 2025-12-25 04:37:00,325 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm2-47382891 failed! 2025-12-25 04:37:00,353 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 is 192.168.1.1 and allocation pool is NOT set 2025-12-25 04:37:04,450 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1032ms') 2025-12-25 04:37:04,450 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm2-47382891 failed! 2025-12-25 04:37:04,469 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 is 192.168.1.1 and allocation pool is NOT set 2025-12-25 04:37:06,541 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=7.04 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=4.31 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 4.310/5.676/7.042/1.366 ms') 2025-12-25 04:37:06,541 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm2-47382891 passed 2025-12-25 04:37:06,620 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:37:06,620 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-47382891, IP 192.168.1.4, Port 22 2025-12-25 04:37:06,686 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-25 04:37:06,795 - DEBUG - VM ctest-vn1_vm2-47382891 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-25 04:37:11,796 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:37:11,796 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-47382891, IP 192.168.1.4, Port 22 2025-12-25 04:37:11,865 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-25 04:37:11,975 - DEBUG - VM ctest-vn1_vm2-47382891 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-25 04:37:16,975 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:37:16,975 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-47382891, IP 192.168.1.4, Port 22 2025-12-25 04:37:17,042 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-25 04:37:17,151 - DEBUG - VM ctest-vn1_vm2-47382891 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-25 04:37:22,152 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:37:22,152 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-47382891, IP 192.168.1.4, Port 22 2025-12-25 04:37:22,222 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-25 04:37:22,330 - DEBUG - VM ctest-vn1_vm2-47382891 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-25 04:37:27,331 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:37:27,332 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-47382891, IP 192.168.1.4, Port 22 2025-12-25 04:37:27,506 - DEBUG - VM ctest-vn1_vm2-47382891 is ready for SSH connections 2025-12-25 04:37:27,506 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:37:27,506 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.19, gateway password: c0ntrail123 2025-12-25 04:37:27,506 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4 2025-12-25 04:38:11,382 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data. 64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=11.2 ms 64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.66 ms 64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.81 ms --- 192.168.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2005ms rtt min/avg/max/mdev = 1.662/4.897/11.218/4.470 ms 2025-12-25 04:38:11,383 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-30118421 passed 2025-12-25 04:38:11,383 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:38:11,383 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-12-25 04:38:11,383 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3 2025-12-25 04:38:57,191 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. 64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=7.46 ms 64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.71 ms 64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.29 ms --- 192.168.1.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2005ms rtt min/avg/max/mdev = 1.295/3.491/7.463/2.814 ms 2025-12-25 04:38:57,191 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-47382891 passed 2025-12-25 04:38:57,199 - INFO - Active control node from the Agent 10.0.0.19 is 10.0.0.23 2025-12-25 04:38:57,199 - INFO - Stoping the Control service in 10.0.0.23 2025-12-25 04:38:57,611 - INFO - 10.0.0.23 2025-12-25 04:38:57,611 - INFO - control:inactive 2025-12-25 04:38:57,611 - INFO - Contrail services control are down on nodes 10.0.0.23 2025-12-25 04:39:02,618 - INFO - Active control node from the Agent 10.0.0.19 is 10.0.0.33 2025-12-25 04:39:02,619 - INFO - Starting the Control service in 10.0.0.23 2025-12-25 04:39:03,041 - INFO - 10.0.0.23 2025-12-25 04:39:03,049 - INFO - control:initializing 2025-12-25 04:39:03,052 - DEBUG - defaultdict(, {'10.0.0.23': {'control': {'status': 'initializing', 'description': None}}}) 2025-12-25 04:39:03,052 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-12-25 04:39:08,154 - INFO - 10.0.0.23 2025-12-25 04:39:08,162 - INFO - control:active 2025-12-25 04:39:08,163 - INFO - Contrail services ['control'] are up on nodes 10.0.0.23 2025-12-25 04:39:13,296 - DEBUG - VM is in ACTIVE state now 2025-12-25 04:39:13,296 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:39:13,296 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 2025-12-25 04:39:13,306 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/b5858679-20bd-4ad0-b32d-efc7de28bd4e 2025-12-25 04:39:13,320 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 2025-12-25 04:39:13,331 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/b5858679-20bd-4ad0-b32d-efc7de28bd4e 2025-12-25 04:39:13,345 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/f2a594b3-a54a-40c8-b5a2-54900d9b7c8c 2025-12-25 04:39:13,360 - DEBUG - Verifying in api server 10.0.0.23 2025-12-25 04:39:13,360 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 2025-12-25 04:39:13,369 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/b5858679-20bd-4ad0-b32d-efc7de28bd4e 2025-12-25 04:39:13,382 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/f2a594b3-a54a-40c8-b5a2-54900d9b7c8c 2025-12-25 04:39:13,395 - INFO - VM ctest-vn1_vm1-30118421 verfication in all API Servers passed 2025-12-25 04:39:13,418 - DEBUG - VM ctest-vn1_vm1-30118421 Tap interface: {'index': '3', 'name': 'tapb5858679-20', 'uuid': 'b5858679-20bd-4ad0-b32d-efc7de28bd4e', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144:ctest-vn1-88877144', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144', 'vm_uuid': '564bbfa3-46a0-4e58-8dd7-86498e6cd1f6', 'vm_name': 'ctest-vn1_vm1-30118421', 'ip_addr': '192.168.1.3', 'mac_addr': '02:b5:85:86:79:20', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '160', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-19229946:b5858679-20bd-4ad0-b32d-efc7de28bd4e', 'sg_uuid_list': ['71e965cc-01d3-4f06-aa5d-b1e37addc289'], 'static_route_list': None, 'vm_project_uuid': 'edba148a-64cc-4e4a-8216-4c7dbc6a7d87', 'admin_state': 'Enabled', 'flow_key_idx': '28', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-25 04:39:13,418 - DEBUG - Agent 10.0.0.19 vrf name: default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144:ctest-vn1-88877144 2025-12-25 04:39:13,424 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144:ctest-vn1-88877144', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144', 'table_label': '-1', 'vxlan_id': '15', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.252:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-12-25 04:39:13,516 - DEBUG - VM ctest-vn1_vm1-30118421 : Tap interface tapb5858679-20 is set to Active 2025-12-25 04:39:13,516 - DEBUG - Tap interface tapb5858679-20 detail : {'index': '3', 'name': 'tapb5858679-20', 'uuid': 'b5858679-20bd-4ad0-b32d-efc7de28bd4e', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144:ctest-vn1-88877144', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144', 'vm_uuid': '564bbfa3-46a0-4e58-8dd7-86498e6cd1f6', 'vm_name': 'ctest-vn1_vm1-30118421', 'ip_addr': '192.168.1.3', 'mac_addr': '02:b5:85:86:79:20', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '160', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-19229946:b5858679-20bd-4ad0-b32d-efc7de28bd4e', 'sg_uuid_list': ['71e965cc-01d3-4f06-aa5d-b1e37addc289'], 'static_route_list': None, 'vm_project_uuid': 'edba148a-64cc-4e4a-8216-4c7dbc6a7d87', 'admin_state': 'Enabled', 'flow_key_idx': '28', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-25 04:39:13,647 - DEBUG - Starting Layer 2 verification in Agent 2025-12-25 04:39:13,662 - DEBUG - Layer 2 path is seen for VM MAC 02:b5:85:86:79:20 in agent 10.0.0.19 2025-12-25 04:39:13,662 - DEBUG - Active layer 2 route in agent is present for VMI tapb5858679-20 2025-12-25 04:39:13,662 - DEBUG - L2 label(29) matches bw route table and interface table 2025-12-25 04:39:13,671 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 is found in Agent of node 10.0.0.19 2025-12-25 04:39:13,812 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 is consistent in agent 10.0.0.19 2025-12-25 04:39:13,813 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.19 2025-12-25 04:39:13,813 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 verification for VM ctest-vn1_vm1-30118421 in Agent 10.0.0.19 passed 2025-12-25 04:39:13,939 - DEBUG - Starting all layer 2 verification in agent 10.0.0.19 2025-12-25 04:39:13,952 - DEBUG - Route for VM MAC 02:b5:85:86:79:20 is consistent in agent 10.0.0.19 2025-12-25 04:39:13,960 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 is found in Agent of node 10.0.0.130 2025-12-25 04:39:14,100 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 is consistent in agent 10.0.0.130 2025-12-25 04:39:14,100 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.130 2025-12-25 04:39:14,100 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 verification for VM ctest-vn1_vm1-30118421 in Agent 10.0.0.130 passed 2025-12-25 04:39:14,222 - DEBUG - Starting all layer 2 verification in agent 10.0.0.130 2025-12-25 04:39:14,235 - DEBUG - Route for VM MAC 02:b5:85:86:79:20 is consistent in agent 10.0.0.130 2025-12-25 04:39:14,236 - INFO - VM ctest-vn1_vm1-30118421 verifications in Compute nodes passed 2025-12-25 04:39:14,510 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.19 2025-12-25 04:39:14,516 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-12-25 04:39:14,644 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.130 2025-12-25 04:39:14,652 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.252, 25; Got 10.20.0.252,25 2025-12-25 04:39:14,652 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.130 2025-12-25 04:39:14,652 - INFO - Validated routes of VM ctest-vn1_vm1-30118421 in all vrouters 2025-12-25 04:39:15,197 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2025-12-25 04:39:15,350 - DEBUG - Layer2 route found for VM MAC 02:b5:85:86:79:20 in Control-node 10.0.0.27 2025-12-25 04:39:15,350 - DEBUG - L2 Label for VM ctest-vn1_vm1-30118421 same between Control-node 10.0.0.27 and Agent, Expected: 29, Seen: 29 2025-12-25 04:39:15,350 - DEBUG - Starting all layer2 verification in 10.0.0.33 Control Node 2025-12-25 04:39:15,488 - DEBUG - Layer2 route found for VM MAC 02:b5:85:86:79:20 in Control-node 10.0.0.33 2025-12-25 04:39:15,488 - DEBUG - L2 Label for VM ctest-vn1_vm1-30118421 same between Control-node 10.0.0.33 and Agent, Expected: 29, Seen: 29 2025-12-25 04:39:15,488 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2025-12-25 04:39:15,628 - DEBUG - Layer2 route found for VM MAC 02:b5:85:86:79:20 in Control-node 10.0.0.23 2025-12-25 04:39:15,628 - DEBUG - L2 Label for VM ctest-vn1_vm1-30118421 same between Control-node 10.0.0.23 and Agent, Expected: 29, Seen: 29 2025-12-25 04:39:15,628 - INFO - Verification in Control-nodes for VM ctest-vn1_vm1-30118421 passed 2025-12-25 04:39:15,628 - DEBUG - Verifying the vm in opserver 2025-12-25 04:39:15,628 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-12-25 04:39:15,628 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/564bbfa3-46a0-4e58-8dd7-86498e6cd1f6?flat 2025-12-25 04:39:15,647 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-19229946:b5858679-20bd-4ad0-b32d-efc7de28bd4e?flat 2025-12-25 04:39:15,673 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-19229946:b5858679-20bd-4ad0-b32d-efc7de28bd4e?flat 2025-12-25 04:39:15,761 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/564bbfa3-46a0-4e58-8dd7-86498e6cd1f6?flat 2025-12-25 04:39:15,849 - DEBUG - Verifying vm in vn uve 2025-12-25 04:39:15,849 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-19229946:b5858679-20bd-4ad0-b32d-efc7de28bd4e?flat 2025-12-25 04:39:15,937 - DEBUG - VM uve shows interface as {'active': True, 'rx_vlan': 65535, 'l2_active': True, 'vn_uuid': '57c4ee51-9600-48e5-8782-069f92e5e892', 'vm_uuid': '564bbfa3-46a0-4e58-8dd7-86498e6cd1f6', 'vhostuser_mode': 0, 'ip_address': '192.168.1.3', 'label': 25, 'port_mirror_enabled': False, 'ip6_active': False, 'admin_state': True, 'ip6_address': '::', 'vm_name': 'ctest-vn1_vm1-30118421', 'mac_address': '02:b5:85:86:79:20', 'uuid': 'b5858679-20bd-4ad0-b32d-efc7de28bd4e', 'is_health_check_active': True, 'tx_vlan': 65535, '__T': 1766637348247997, 'ip4_active': True, 'fixed_ip4_list': ['192.168.1.3'], 'gateway': '192.168.1.1', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144'} 2025-12-25 04:39:15,937 - DEBUG - VM uve shows ip address as ['192.168.1.3', '::'] 2025-12-25 04:39:15,937 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 2025-12-25 04:39:15,937 - INFO - Verifying through opserver in 10.0.0.27 2025-12-25 04:39:15,939 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144?flat 2025-12-25 04:39:15,970 - DEBUG - expected vm list 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 2025-12-25 04:39:15,970 - DEBUG - Extracted vm list ['564bbfa3-46a0-4e58-8dd7-86498e6cd1f6'] 2025-12-25 04:39:15,970 - INFO - VM 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 is present in default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 2025-12-25 04:39:15,970 - DEBUG - Verifying vm in vrouter uve 2025-12-25 04:39:15,970 - DEBUG - Getting info from collector 10.0.0.23.. 2025-12-25 04:39:15,971 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/564bbfa3-46a0-4e58-8dd7-86498e6cd1f6?flat 2025-12-25 04:39:16,054 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4927-1'] 2025-12-25 04:39:16,054 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-1:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:39:16,124 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-1?flat 2025-12-25 04:39:16,161 - DEBUG - VM 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 is present in vrouter an-jenkins-deploy-platform-ansible-os-4927-1 uve 2025-12-25 04:39:16,161 - DEBUG - Validated that VM ctest-vn1_vm1-30118421 is in Vrouter an-jenkins-deploy-platform-ansible-os-4927-1 UVE 2025-12-25 04:39:16,161 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-12-25 04:39:16,171 - DEBUG - Expected tap interface of VM uuid 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 is default-domain:ctest-TestBasicVMVN0-19229946:b5858679-20bd-4ad0-b32d-efc7de28bd4e 2025-12-25 04:39:16,171 - DEBUG - Expected VN of VM uuid 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 is default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 2025-12-25 04:39:16,171 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-1:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:39:16,229 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-1?flat 2025-12-25 04:39:16,264 - DEBUG - VM 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 is present in vrouter an-jenkins-deploy-platform-ansible-os-4927-1 uve 2025-12-25 04:39:16,264 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-19229946:b5858679-20bd-4ad0-b32d-efc7de28bd4e of vm 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 is present in vrouter an-jenkins-deploy-platform-ansible-os-4927-1 uve 2025-12-25 04:39:16,264 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 of vm 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 is present in vrouter an-jenkins-deploy-platform-ansible-os-4927-1 uve 2025-12-25 04:39:16,264 - DEBUG - Verifying the 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 virtual network link through opserver 10.0.0.27 2025-12-25 04:39:16,264 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-12-25 04:39:16,279 - DEBUG - VM link and name as {'name': '564bbfa3-46a0-4e58-8dd7-86498e6cd1f6', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/564bbfa3-46a0-4e58-8dd7-86498e6cd1f6?flat'} 2025-12-25 04:39:16,280 - DEBUG - Verifying the 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 virtual network link through opserver 10.0.0.33 2025-12-25 04:39:16,280 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-12-25 04:39:16,295 - DEBUG - VM link and name as {'name': '564bbfa3-46a0-4e58-8dd7-86498e6cd1f6', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-machine/564bbfa3-46a0-4e58-8dd7-86498e6cd1f6?flat'} 2025-12-25 04:39:16,296 - DEBUG - Verifying the 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 virtual network link through opserver 10.0.0.23 2025-12-25 04:39:16,296 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-12-25 04:39:16,318 - DEBUG - VM link and name as {'name': '564bbfa3-46a0-4e58-8dd7-86498e6cd1f6', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/564bbfa3-46a0-4e58-8dd7-86498e6cd1f6?flat'} 2025-12-25 04:39:16,318 - INFO - VM ctest-vn1_vm1-30118421 validations in Opserver passed 2025-12-25 04:39:16,424 - DEBUG - VM is in ACTIVE state now 2025-12-25 04:39:16,424 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:39:16,425 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d 2025-12-25 04:39:16,435 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/2fa36043-ce42-4433-8555-5571fdbe0f18 2025-12-25 04:39:16,446 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d 2025-12-25 04:39:16,457 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/2fa36043-ce42-4433-8555-5571fdbe0f18 2025-12-25 04:39:16,469 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/9a75bd3d-7ed0-4f1a-b820-5b156685b939 2025-12-25 04:39:16,482 - DEBUG - Verifying in api server 10.0.0.23 2025-12-25 04:39:16,482 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d 2025-12-25 04:39:16,493 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/2fa36043-ce42-4433-8555-5571fdbe0f18 2025-12-25 04:39:16,506 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/9a75bd3d-7ed0-4f1a-b820-5b156685b939 2025-12-25 04:39:16,520 - INFO - VM ctest-vn1_vm2-47382891 verfication in all API Servers passed 2025-12-25 04:39:16,541 - DEBUG - VM ctest-vn1_vm2-47382891 Tap interface: {'index': '3', 'name': 'tap2fa36043-ce', 'uuid': '2fa36043-ce42-4433-8555-5571fdbe0f18', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144:ctest-vn1-88877144', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144', 'vm_uuid': '1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d', 'vm_name': 'ctest-vn1_vm2-47382891', 'ip_addr': '192.168.1.4', 'mac_addr': '02:2f:a3:60:43:ce', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '127', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-19229946:2fa36043-ce42-4433-8555-5571fdbe0f18', 'sg_uuid_list': ['71e965cc-01d3-4f06-aa5d-b1e37addc289'], 'static_route_list': None, 'vm_project_uuid': 'edba148a-64cc-4e4a-8216-4c7dbc6a7d87', 'admin_state': 'Enabled', 'flow_key_idx': '29', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-25 04:39:16,541 - DEBUG - Agent 10.0.0.130 vrf name: default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144:ctest-vn1-88877144 2025-12-25 04:39:16,548 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144:ctest-vn1-88877144', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144', 'table_label': '-1', 'vxlan_id': '15', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.249:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-12-25 04:39:16,637 - DEBUG - VM ctest-vn1_vm2-47382891 : Tap interface tap2fa36043-ce is set to Active 2025-12-25 04:39:16,637 - DEBUG - Tap interface tap2fa36043-ce detail : {'index': '3', 'name': 'tap2fa36043-ce', 'uuid': '2fa36043-ce42-4433-8555-5571fdbe0f18', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144:ctest-vn1-88877144', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144', 'vm_uuid': '1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d', 'vm_name': 'ctest-vn1_vm2-47382891', 'ip_addr': '192.168.1.4', 'mac_addr': '02:2f:a3:60:43:ce', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '127', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-19229946:2fa36043-ce42-4433-8555-5571fdbe0f18', 'sg_uuid_list': ['71e965cc-01d3-4f06-aa5d-b1e37addc289'], 'static_route_list': None, 'vm_project_uuid': 'edba148a-64cc-4e4a-8216-4c7dbc6a7d87', 'admin_state': 'Enabled', 'flow_key_idx': '29', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-25 04:39:16,753 - DEBUG - Starting Layer 2 verification in Agent 2025-12-25 04:39:16,767 - DEBUG - Layer 2 path is seen for VM MAC 02:2f:a3:60:43:ce in agent 10.0.0.130 2025-12-25 04:39:16,767 - DEBUG - Active layer 2 route in agent is present for VMI tap2fa36043-ce 2025-12-25 04:39:16,767 - DEBUG - L2 label(29) matches bw route table and interface table 2025-12-25 04:39:16,773 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 is found in Agent of node 10.0.0.19 2025-12-25 04:39:16,911 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 is consistent in agent 10.0.0.19 2025-12-25 04:39:16,911 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.19 2025-12-25 04:39:16,911 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 verification for VM ctest-vn1_vm2-47382891 in Agent 10.0.0.19 passed 2025-12-25 04:39:17,031 - DEBUG - Starting all layer 2 verification in agent 10.0.0.19 2025-12-25 04:39:17,046 - DEBUG - Route for VM MAC 02:2f:a3:60:43:ce is consistent in agent 10.0.0.19 2025-12-25 04:39:17,054 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 is found in Agent of node 10.0.0.130 2025-12-25 04:39:17,197 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 is consistent in agent 10.0.0.130 2025-12-25 04:39:17,197 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.130 2025-12-25 04:39:17,197 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 verification for VM ctest-vn1_vm2-47382891 in Agent 10.0.0.130 passed 2025-12-25 04:39:17,323 - DEBUG - Starting all layer 2 verification in agent 10.0.0.130 2025-12-25 04:39:17,337 - DEBUG - Route for VM MAC 02:2f:a3:60:43:ce is consistent in agent 10.0.0.130 2025-12-25 04:39:17,337 - INFO - VM ctest-vn1_vm2-47382891 verifications in Compute nodes passed 2025-12-25 04:39:17,599 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.19 2025-12-25 04:39:17,605 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.249, 25; Got 10.20.0.249,25 2025-12-25 04:39:17,605 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.19 2025-12-25 04:39:17,734 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.130 2025-12-25 04:39:17,740 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-12-25 04:39:17,740 - INFO - Validated routes of VM ctest-vn1_vm2-47382891 in all vrouters 2025-12-25 04:39:18,279 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2025-12-25 04:39:18,425 - DEBUG - Layer2 route found for VM MAC 02:2f:a3:60:43:ce in Control-node 10.0.0.27 2025-12-25 04:39:18,425 - DEBUG - L2 Label for VM ctest-vn1_vm2-47382891 same between Control-node 10.0.0.27 and Agent, Expected: 29, Seen: 29 2025-12-25 04:39:18,425 - DEBUG - Starting all layer2 verification in 10.0.0.33 Control Node 2025-12-25 04:39:18,564 - DEBUG - Layer2 route found for VM MAC 02:2f:a3:60:43:ce in Control-node 10.0.0.33 2025-12-25 04:39:18,564 - DEBUG - L2 Label for VM ctest-vn1_vm2-47382891 same between Control-node 10.0.0.33 and Agent, Expected: 29, Seen: 29 2025-12-25 04:39:18,564 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2025-12-25 04:39:18,696 - DEBUG - Layer2 route found for VM MAC 02:2f:a3:60:43:ce in Control-node 10.0.0.23 2025-12-25 04:39:18,696 - DEBUG - L2 Label for VM ctest-vn1_vm2-47382891 same between Control-node 10.0.0.23 and Agent, Expected: 29, Seen: 29 2025-12-25 04:39:18,696 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-47382891 passed 2025-12-25 04:39:18,696 - DEBUG - Verifying the vm in opserver 2025-12-25 04:39:18,696 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-12-25 04:39:18,696 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d?flat 2025-12-25 04:39:18,716 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-19229946:2fa36043-ce42-4433-8555-5571fdbe0f18?flat 2025-12-25 04:39:18,739 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-19229946:2fa36043-ce42-4433-8555-5571fdbe0f18?flat 2025-12-25 04:39:18,760 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d?flat 2025-12-25 04:39:18,778 - DEBUG - Verifying vm in vn uve 2025-12-25 04:39:18,778 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-19229946:2fa36043-ce42-4433-8555-5571fdbe0f18?flat 2025-12-25 04:39:18,801 - DEBUG - VM uve shows interface as {'active': True, 'rx_vlan': 65535, 'l2_active': True, 'vn_uuid': '57c4ee51-9600-48e5-8782-069f92e5e892', 'vm_uuid': '1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d', 'vhostuser_mode': 0, 'ip_address': '192.168.1.4', 'label': 25, 'port_mirror_enabled': False, 'ip6_active': False, 'admin_state': True, 'ip6_address': '::', 'vm_name': 'ctest-vn1_vm2-47382891', 'mac_address': '02:2f:a3:60:43:ce', 'uuid': '2fa36043-ce42-4433-8555-5571fdbe0f18', 'is_health_check_active': True, 'tx_vlan': 65535, '__T': 1766637357328825, 'ip4_active': True, 'fixed_ip4_list': ['192.168.1.4'], 'gateway': '192.168.1.1', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144'} 2025-12-25 04:39:18,801 - DEBUG - VM uve shows ip address as ['192.168.1.4', '::'] 2025-12-25 04:39:18,801 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 2025-12-25 04:39:18,801 - INFO - Verifying through opserver in 10.0.0.27 2025-12-25 04:39:18,801 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144?flat 2025-12-25 04:39:18,829 - DEBUG - expected vm list 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d 2025-12-25 04:39:18,829 - DEBUG - Extracted vm list ['1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d'] 2025-12-25 04:39:18,829 - INFO - VM 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d is present in default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 2025-12-25 04:39:18,829 - DEBUG - Verifying vm in vrouter uve 2025-12-25 04:39:18,829 - DEBUG - Getting info from collector 10.0.0.23.. 2025-12-25 04:39:18,829 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d?flat 2025-12-25 04:39:18,848 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4927-2'] 2025-12-25 04:39:18,848 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-2:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:39:18,905 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-2?flat 2025-12-25 04:39:18,941 - DEBUG - VM 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d is present in vrouter an-jenkins-deploy-platform-ansible-os-4927-2 uve 2025-12-25 04:39:18,941 - DEBUG - Validated that VM ctest-vn1_vm2-47382891 is in Vrouter an-jenkins-deploy-platform-ansible-os-4927-2 UVE 2025-12-25 04:39:18,941 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-12-25 04:39:18,950 - DEBUG - Expected tap interface of VM uuid 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d is default-domain:ctest-TestBasicVMVN0-19229946:2fa36043-ce42-4433-8555-5571fdbe0f18 2025-12-25 04:39:18,950 - DEBUG - Expected VN of VM uuid 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d is default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 2025-12-25 04:39:18,950 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-2:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:39:19,003 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-2?flat 2025-12-25 04:39:19,043 - DEBUG - VM 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d is present in vrouter an-jenkins-deploy-platform-ansible-os-4927-2 uve 2025-12-25 04:39:19,043 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-19229946:2fa36043-ce42-4433-8555-5571fdbe0f18 of vm 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d is present in vrouter an-jenkins-deploy-platform-ansible-os-4927-2 uve 2025-12-25 04:39:19,044 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 of vm 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d is present in vrouter an-jenkins-deploy-platform-ansible-os-4927-2 uve 2025-12-25 04:39:19,044 - DEBUG - Verifying the 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d virtual network link through opserver 10.0.0.27 2025-12-25 04:39:19,044 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-12-25 04:39:19,065 - DEBUG - VM link and name as {'name': '1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d?flat'} 2025-12-25 04:39:19,065 - DEBUG - Verifying the 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d virtual network link through opserver 10.0.0.33 2025-12-25 04:39:19,065 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-12-25 04:39:19,154 - DEBUG - VM link and name as {'name': '1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-machine/1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d?flat'} 2025-12-25 04:39:19,154 - DEBUG - Verifying the 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d virtual network link through opserver 10.0.0.23 2025-12-25 04:39:19,154 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-12-25 04:39:19,169 - DEBUG - VM link and name as {'name': '1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d?flat'} 2025-12-25 04:39:19,169 - INFO - VM ctest-vn1_vm2-47382891 validations in Opserver passed 2025-12-25 04:39:19,169 - INFO - Checking the ping between the VM again 2025-12-25 04:39:19,169 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:39:19,169 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.19, gateway password: c0ntrail123 2025-12-25 04:39:19,170 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4 2025-12-25 04:39:21,743 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data. 64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=4.29 ms 64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.60 ms 64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.18 ms --- 192.168.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.183/2.359/4.293/1.378 ms 2025-12-25 04:39:21,743 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-30118421 passed 2025-12-25 04:39:21,743 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:39:21,743 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-12-25 04:39:21,743 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3 2025-12-25 04:39:24,295 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. 64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=2.63 ms 64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.60 ms 64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.53 ms --- 192.168.1.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2005ms rtt min/avg/max/mdev = 1.539/1.927/2.633/0.500 ms 2025-12-25 04:39:24,295 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-47382891 passed 2025-12-25 04:39:24,295 - INFO - Deleting VM ctest-vn1_vm2-47382891 2025-12-25 04:39:24,396 - DEBUG - Verifying in api server 10.0.0.23 2025-12-25 04:39:24,397 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d 2025-12-25 04:39:24,407 - DEBUG - VM ID 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d of VM ctest-vn1_vm2-47382891 is still found in API Server 2025-12-25 04:39:26,407 - DEBUG - Verifying in api server 10.0.0.23 2025-12-25 04:39:26,408 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d 2025-12-25 04:39:26,415 - DEBUG - Response Code: 404 2025-12-25 04:39:26,415 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d 2025-12-25 04:39:26,425 - DEBUG - Response Code: 404 2025-12-25 04:39:26,425 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d 2025-12-25 04:39:26,433 - DEBUG - Response Code: 404 2025-12-25 04:39:26,433 - INFO - VM ctest-vn1_vm2-47382891 is fully removed in API-Server 2025-12-25 04:39:26,590 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-12-25 04:39:26,789 - DEBUG - VM route 192.168.1.4 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 of compute 10.0.0.19 2025-12-25 04:39:26,793 - DEBUG - Validated that vrouter 10.0.0.19 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 2025-12-25 04:39:26,794 - DEBUG - Validated that vrouter 10.0.0.130 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 2025-12-25 04:39:26,794 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 2025-12-25 04:39:26,794 - INFO - VM ctest-vn1_vm2-47382891 is removed in Compute, and routes are removed in all compute nodes 2025-12-25 04:39:27,073 - INFO - Routes for VM ctest-vn1_vm2-47382891 is removed in all control-nodes 2025-12-25 04:39:27,073 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-2:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:39:27,146 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-2?flat 2025-12-25 04:39:27,179 - DEBUG - VM 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d is still present in vrouter an-jenkins-deploy-platform-ansible-os-4927-2 uve 2025-12-25 04:39:27,179 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:39:27,179 - DEBUG - interface for vm 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d deleted from vrouter uve ... 2025-12-25 04:39:27,179 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:39:30,180 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-2:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:39:30,242 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-2?flat 2025-12-25 04:39:30,276 - DEBUG - VM 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d is not present in vrouter an-jenkins-deploy-platform-ansible-os-4927-2 uve 2025-12-25 04:39:30,276 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:39:30,276 - DEBUG - interface for vm 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d deleted from vrouter uve ... 2025-12-25 04:39:30,276 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:39:30,276 - DEBUG - Validated that VM 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d is removed in Vrouter UVE 2025-12-25 04:39:30,276 - DEBUG - Verifying through opserver in 10.0.0.27 2025-12-25 04:39:30,276 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144?flat 2025-12-25 04:39:30,304 - DEBUG - 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d BM not in default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 uve 2025-12-25 04:39:30,304 - DEBUG - Validated that VM 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d is not present in the VN UVE 2025-12-25 04:39:30,304 - DEBUG - Verifying the 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d virtual network link through opserver 10.0.0.27 2025-12-25 04:39:30,304 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-12-25 04:39:30,323 - INFO - 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d vm uve deleted from opserver 2025-12-25 04:39:30,323 - DEBUG - Verifying the 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d virtual network link through opserver 10.0.0.33 2025-12-25 04:39:30,323 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-12-25 04:39:30,436 - INFO - 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d vm uve deleted from opserver 2025-12-25 04:39:30,436 - DEBUG - Verifying the 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d virtual network link through opserver 10.0.0.23 2025-12-25 04:39:30,436 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-12-25 04:39:30,453 - INFO - 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d vm uve deleted from opserver 2025-12-25 04:39:30,453 - INFO - Validated that VM 1764a1c5-ec1f-4f0f-b3af-b0bd9f695c8d is removed from Opserver 2025-12-25 04:39:30,453 - INFO - Deleting VM ctest-vn1_vm1-30118421 2025-12-25 04:39:30,560 - DEBUG - Verifying in api server 10.0.0.23 2025-12-25 04:39:30,560 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 2025-12-25 04:39:30,571 - DEBUG - VM ID 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 of VM ctest-vn1_vm1-30118421 is still found in API Server 2025-12-25 04:39:32,572 - DEBUG - Verifying in api server 10.0.0.23 2025-12-25 04:39:32,573 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 2025-12-25 04:39:32,582 - DEBUG - Response Code: 404 2025-12-25 04:39:32,583 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 2025-12-25 04:39:32,591 - DEBUG - Response Code: 404 2025-12-25 04:39:32,591 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 2025-12-25 04:39:32,600 - DEBUG - Response Code: 404 2025-12-25 04:39:32,600 - INFO - VM ctest-vn1_vm1-30118421 is fully removed in API-Server 2025-12-25 04:39:32,766 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-12-25 04:39:32,822 - DEBUG - Validated that vrouter 10.0.0.19 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 2025-12-25 04:39:32,835 - DEBUG - Validated that vrouter 10.0.0.130 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 2025-12-25 04:39:32,835 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 2025-12-25 04:39:32,835 - INFO - VM ctest-vn1_vm1-30118421 is removed in Compute, and routes are removed in all compute nodes 2025-12-25 04:39:33,091 - INFO - Routes for VM ctest-vn1_vm1-30118421 is removed in all control-nodes 2025-12-25 04:39:33,091 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-1:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:39:33,154 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-1?flat 2025-12-25 04:39:33,195 - DEBUG - VM 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4927-1 uve 2025-12-25 04:39:33,195 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:39:33,195 - DEBUG - interface for vm 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 deleted from vrouter uve ... 2025-12-25 04:39:33,195 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:39:36,196 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-1:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:39:36,250 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-1?flat 2025-12-25 04:39:36,290 - DEBUG - VM 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4927-1 uve 2025-12-25 04:39:36,290 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:39:36,290 - DEBUG - interface for vm 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 deleted from vrouter uve ... 2025-12-25 04:39:36,290 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:39:39,292 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-1:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:39:39,350 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-1?flat 2025-12-25 04:39:39,383 - DEBUG - VM 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4927-1 uve 2025-12-25 04:39:39,384 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:39:39,384 - DEBUG - interface for vm 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 deleted from vrouter uve ... 2025-12-25 04:39:39,384 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:39:42,384 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-1:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:39:42,441 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-1?flat 2025-12-25 04:39:42,475 - DEBUG - VM 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4927-1 uve 2025-12-25 04:39:42,475 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:39:42,475 - DEBUG - interface for vm 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 deleted from vrouter uve ... 2025-12-25 04:39:42,475 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:39:45,476 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-1:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:39:45,551 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-1?flat 2025-12-25 04:39:45,594 - DEBUG - VM 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4927-1 uve 2025-12-25 04:39:45,594 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:39:45,594 - DEBUG - interface for vm 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 deleted from vrouter uve ... 2025-12-25 04:39:45,594 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:39:48,595 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-1:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:39:48,653 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-1?flat 2025-12-25 04:39:48,688 - DEBUG - VM 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 is not present in vrouter an-jenkins-deploy-platform-ansible-os-4927-1 uve 2025-12-25 04:39:48,688 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:39:48,688 - DEBUG - interface for vm 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 deleted from vrouter uve ... 2025-12-25 04:39:48,688 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:39:48,688 - DEBUG - Validated that VM 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 is removed in Vrouter UVE 2025-12-25 04:39:48,688 - DEBUG - Verifying through opserver in 10.0.0.27 2025-12-25 04:39:48,688 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144?flat 2025-12-25 04:39:48,709 - DEBUG - 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 BM not in default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-88877144 uve 2025-12-25 04:39:48,709 - DEBUG - Validated that VM 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 is not present in the VN UVE 2025-12-25 04:39:48,709 - DEBUG - Verifying the 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 virtual network link through opserver 10.0.0.27 2025-12-25 04:39:48,709 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-12-25 04:39:48,725 - INFO - 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 vm uve deleted from opserver 2025-12-25 04:39:48,725 - DEBUG - Verifying the 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 virtual network link through opserver 10.0.0.33 2025-12-25 04:39:48,725 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-12-25 04:39:48,740 - INFO - 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 vm uve deleted from opserver 2025-12-25 04:39:48,740 - DEBUG - Verifying the 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 virtual network link through opserver 10.0.0.23 2025-12-25 04:39:48,740 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-12-25 04:39:48,756 - INFO - 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 vm uve deleted from opserver 2025-12-25 04:39:48,756 - INFO - Validated that VM 564bbfa3-46a0-4e58-8dd7-86498e6cd1f6 is removed from Opserver 2025-12-25 04:39:48,756 - INFO - Deleting VN ctest-vn1-88877144 2025-12-25 04:39:48,950 - DEBUG - Response for deleting network () 2025-12-25 04:39:48,950 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/fb8238c0-da0c-41d3-8200-d4820d154826 2025-12-25 04:39:48,961 - DEBUG - Response Code: 404 2025-12-25 04:39:48,961 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-12-25 04:39:48,970 - DEBUG - Requesting: http://10.0.0.27:8082/domain/d4f5d7ea-1779-482f-9e93-03873d431e99 2025-12-25 04:39:48,990 - DEBUG - Requesting: http://10.0.0.27:8082/project/edba148a-64cc-4e4a-8216-4c7dbc6a7d87 2025-12-25 04:39:49,096 - INFO - Validated that VN ctest-vn1-88877144 is not found in API Server 2025-12-25 04:39:49,108 - DEBUG - VN ctest-vn1-88877144 is not present in Agent 10.0.0.19 2025-12-25 04:39:49,125 - DEBUG - VN ctest-vn1-88877144 is not present in Agent 10.0.0.130 2025-12-25 04:39:49,125 - INFO - Validated that VN ctest-vn1-88877144 is not in any agent 2025-12-25 04:39:49,171 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-88877144 info 2025-12-25 04:39:50,465 - DEBUG - Skipping xmpp flap check

2025-12-25 04:39:50,465 - INFO - END TEST : test_control_node_switchover : PASSED[0:04:19]

2025-12-25 04:39:50,465 - INFO - -------------------------------------------------------------------------------- 2025-12-25 04:39:50,469 - INFO - ================================================================================ 2025-12-25 04:39:50,469 - INFO - STARTING TEST : test_process_restart_in_policy_between_vns 2025-12-25 04:39:50,469 - INFO - TEST DESCRIPTION : Test to validate that with policy having rule to check icmp fwding between VMs on different VNs , ping between VMs should pass with process restarts 1. Pick 2 VN's from resource pool which has one VM each 2. Create policy with icmp allow rule between those VN's and bind it networks 3. Ping from one VM to another VM 4. Restart process 'vrouter' and 'control' on setup 5. Ping again between VM's after process restart Pass criteria: Step 2,3,4 and 5 should pass 2025-12-25 04:39:51,762 - DEBUG - Skipping xmpp flap check 2025-12-25 04:39:51,762 - INFO - Initial checks done. Running the testcase now 2025-12-25 04:39:51,763 - INFO - 2025-12-25 04:39:51,769 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-12-25 04:39:52,060 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-12-25 04:39:52,661 - INFO - Created VN ctest-vn1-91861474, UUID :d07b1963-2a5b-47ff-8f83-4d94e5eac0ea 2025-12-25 04:39:53,029 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-12-25 04:39:53,037 - DEBUG - Requesting: http://10.0.0.27:8082/domain/d4f5d7ea-1779-482f-9e93-03873d431e99 2025-12-25 04:39:53,051 - DEBUG - Requesting: http://10.0.0.27:8082/project/edba148a-64cc-4e4a-8216-4c7dbc6a7d87 2025-12-25 04:39:53,169 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/d07b1963-2a5b-47ff-8f83-4d94e5eac0ea 2025-12-25 04:39:53,185 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/d07b1963-2a5b-47ff-8f83-4d94e5eac0ea 2025-12-25 04:39:53,198 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/226e8db5-3080-4ae5-944b-3627122b62a7 2025-12-25 04:39:53,210 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/226e8db5-3080-4ae5-944b-3627122b62a7 2025-12-25 04:39:53,220 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/ceef7a46-8d77-42c3-af18-68b90e1cda02 2025-12-25 04:39:53,230 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-25 04:39:53,231 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/d07b1963-2a5b-47ff-8f83-4d94e5eac0ea 2025-12-25 04:39:53,245 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/226e8db5-3080-4ae5-944b-3627122b62a7 2025-12-25 04:39:53,268 - INFO - Verified VN network id 15 for VN d07b1963-2a5b-47ff-8f83-4d94e5eac0ea 2025-12-25 04:39:53,268 - INFO - Verifications in API Server for VN ctest-vn1-91861474 passed 2025-12-25 04:39:53,269 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/d07b1963-2a5b-47ff-8f83-4d94e5eac0ea 2025-12-25 04:39:53,284 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/226e8db5-3080-4ae5-944b-3627122b62a7 2025-12-25 04:39:53,298 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/226e8db5-3080-4ae5-944b-3627122b62a7 2025-12-25 04:39:53,310 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/ceef7a46-8d77-42c3-af18-68b90e1cda02 2025-12-25 04:39:53,341 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-19229946', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '15022628895727503359', 'uuid-lslong': '10341194471280656618'}, 'enable': 'true', 'created': '2025-12-25T04:39:52', 'last-modified': '2025-12-25T04:39:52', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.519169'} 2025-12-25 04:39:53,353 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-25 04:39:53,360 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-19229946', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '15022628895727503359', 'uuid-lslong': '10341194471280656618'}, 'enable': 'true', 'created': '2025-12-25T04:39:52', 'last-modified': '2025-12-25T04:39:52', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.544277'} 2025-12-25 04:39:53,371 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-25 04:39:53,379 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-19229946', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '15022628895727503359', 'uuid-lslong': '10341194471280656618'}, 'enable': 'true', 'created': '2025-12-25T04:39:52', 'last-modified': '2025-12-25T04:39:52', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.563691'} 2025-12-25 04:39:53,388 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-25 04:39:53,388 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-91861474 passed 2025-12-25 04:39:53,389 - DEBUG - ====Verifying policy data for ctest-vn1-91861474 in API_Server ====== 2025-12-25 04:39:53,389 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-12-25 04:39:53,398 - DEBUG - Requesting: http://10.0.0.27:8082/domain/d4f5d7ea-1779-482f-9e93-03873d431e99 2025-12-25 04:39:53,416 - DEBUG - Requesting: http://10.0.0.27:8082/project/edba148a-64cc-4e4a-8216-4c7dbc6a7d87 2025-12-25 04:39:53,529 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/d07b1963-2a5b-47ff-8f83-4d94e5eac0ea 2025-12-25 04:39:53,548 - DEBUG - =>VN ctest-vn1-91861474 has no policy to be verified 2025-12-25 04:39:53,548 - DEBUG - Verifying the vn in opserver 2025-12-25 04:39:53,548 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 virtual network link through opserver 10.0.0.27 2025-12-25 04:39:53,548 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-12-25 04:39:53,564 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474?flat'} 2025-12-25 04:39:53,564 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 virtual network link through opserver 10.0.0.33 2025-12-25 04:39:53,564 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2025-12-25 04:39:53,578 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474?flat'} 2025-12-25 04:39:53,578 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 virtual network link through opserver 10.0.0.23 2025-12-25 04:39:53,578 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-12-25 04:39:53,594 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474?flat'} 2025-12-25 04:39:53,594 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 is found in opserver 2025-12-25 04:39:53,617 - DEBUG - Do not have enough data to verify VN in agent 2025-12-25 04:39:53,628 - DEBUG - VRF ids for VN ctest-vn1-91861474: {} 2025-12-25 04:39:54,368 - INFO - Created VN ctest-vn2-64314257, UUID :0a89949f-1f81-453a-b917-59c6379e0fba 2025-12-25 04:39:54,776 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-12-25 04:39:54,784 - DEBUG - Requesting: http://10.0.0.27:8082/domain/d4f5d7ea-1779-482f-9e93-03873d431e99 2025-12-25 04:39:54,802 - DEBUG - Requesting: http://10.0.0.27:8082/project/edba148a-64cc-4e4a-8216-4c7dbc6a7d87 2025-12-25 04:39:54,919 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/0a89949f-1f81-453a-b917-59c6379e0fba 2025-12-25 04:39:54,941 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/0a89949f-1f81-453a-b917-59c6379e0fba 2025-12-25 04:39:54,955 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/351ce0c9-7a04-4a7e-ac1b-b8600b343121 2025-12-25 04:39:54,970 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/351ce0c9-7a04-4a7e-ac1b-b8600b343121 2025-12-25 04:39:54,981 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/7f6c8574-fa9f-4bc1-90cc-b4dacc735050 2025-12-25 04:39:54,990 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-12-25 04:39:54,990 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/0a89949f-1f81-453a-b917-59c6379e0fba 2025-12-25 04:39:55,003 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/351ce0c9-7a04-4a7e-ac1b-b8600b343121 2025-12-25 04:39:55,025 - INFO - Verified VN network id 16 for VN 0a89949f-1f81-453a-b917-59c6379e0fba 2025-12-25 04:39:55,025 - INFO - Verifications in API Server for VN ctest-vn2-64314257 passed 2025-12-25 04:39:55,025 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/0a89949f-1f81-453a-b917-59c6379e0fba 2025-12-25 04:39:55,038 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/351ce0c9-7a04-4a7e-ac1b-b8600b343121 2025-12-25 04:39:55,049 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/351ce0c9-7a04-4a7e-ac1b-b8600b343121 2025-12-25 04:39:55,059 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/7f6c8574-fa9f-4bc1-90cc-b4dacc735050 2025-12-25 04:39:55,077 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-19229946', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '759301423337915706', 'uuid-lslong': '13337227529352515514'}, 'enable': 'true', 'created': '2025-12-25T04:39:54', 'last-modified': '2025-12-25T04:39:54', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.527358'} 2025-12-25 04:39:55,086 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-12-25 04:39:55,094 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-19229946', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '759301423337915706', 'uuid-lslong': '13337227529352515514'}, 'enable': 'true', 'created': '2025-12-25T04:39:54', 'last-modified': '2025-12-25T04:39:54', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.546118'} 2025-12-25 04:39:55,101 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-12-25 04:39:55,108 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-19229946', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '759301423337915706', 'uuid-lslong': '13337227529352515514'}, 'enable': 'true', 'created': '2025-12-25T04:39:54', 'last-modified': '2025-12-25T04:39:54', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.569274'} 2025-12-25 04:39:55,116 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-12-25 04:39:55,116 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn2-64314257 passed 2025-12-25 04:39:55,117 - DEBUG - ====Verifying policy data for ctest-vn2-64314257 in API_Server ====== 2025-12-25 04:39:55,117 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-12-25 04:39:55,125 - DEBUG - Requesting: http://10.0.0.27:8082/domain/d4f5d7ea-1779-482f-9e93-03873d431e99 2025-12-25 04:39:55,142 - DEBUG - Requesting: http://10.0.0.27:8082/project/edba148a-64cc-4e4a-8216-4c7dbc6a7d87 2025-12-25 04:39:55,244 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/0a89949f-1f81-453a-b917-59c6379e0fba 2025-12-25 04:39:55,261 - DEBUG - =>VN ctest-vn2-64314257 has no policy to be verified 2025-12-25 04:39:55,261 - DEBUG - Verifying the vn in opserver 2025-12-25 04:39:55,261 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 virtual network link through opserver 10.0.0.27 2025-12-25 04:39:55,261 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-12-25 04:39:55,276 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257?flat'} 2025-12-25 04:39:55,277 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 virtual network link through opserver 10.0.0.33 2025-12-25 04:39:55,277 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2025-12-25 04:39:55,294 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257?flat'} 2025-12-25 04:39:55,294 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 virtual network link through opserver 10.0.0.23 2025-12-25 04:39:55,294 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-12-25 04:39:55,310 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257?flat'} 2025-12-25 04:39:55,310 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 is found in opserver 2025-12-25 04:39:55,338 - DEBUG - Do not have enough data to verify VN in agent 2025-12-25 04:39:55,355 - DEBUG - VRF ids for VN ctest-vn2-64314257: {} 2025-12-25 04:39:57,212 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4927-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4927-1) 2025-12-25 04:39:59,058 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4927-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4927-2) 2025-12-25 04:39:59,060 - INFO - Waiting for VM ctest-vn1_vm1-31633523 to be up.. 2025-12-25 04:39:59,172 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-12-25 04:40:04,324 - DEBUG - VM is in ACTIVE state now 2025-12-25 04:40:04,324 - INFO - VM name : ctest-vn1_vm1-31633523 2025-12-25 04:40:04,490 - DEBUG - VM ctest-vn1_vm1-31633523 ID is 8e55db70-7fc5-4d43-bd90-1f7c4b807654 2025-12-25 04:40:04,490 - DEBUG - VM ctest-vn1_vm1-31633523 launched on Node an-jenkins-deploy-platform-ansible-os-4927-1 2025-12-25 04:40:04,677 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/8e55db70-7fc5-4d43-bd90-1f7c4b807654 2025-12-25 04:40:04,698 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/44bd40aa-76f9-4c22-9da3-09454432ded2 2025-12-25 04:40:07,926 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1030ms') 2025-12-25 04:40:07,926 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-31633523 failed! 2025-12-25 04:40:07,945 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 is 192.168.1.254 and allocation pool is NOT set 2025-12-25 04:40:12,024 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms') 2025-12-25 04:40:12,024 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-31633523 failed! 2025-12-25 04:40:12,044 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 is 192.168.1.254 and allocation pool is NOT set 2025-12-25 04:40:16,117 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1009ms') 2025-12-25 04:40:16,118 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-31633523 failed! 2025-12-25 04:40:16,135 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 is 192.168.1.254 and allocation pool is NOT set 2025-12-25 04:40:20,216 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms') 2025-12-25 04:40:20,216 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-31633523 failed! 2025-12-25 04:40:20,235 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 is 192.168.1.254 and allocation pool is NOT set 2025-12-25 04:40:24,344 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1034ms') 2025-12-25 04:40:24,344 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-31633523 failed! 2025-12-25 04:40:24,361 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 is 192.168.1.254 and allocation pool is NOT set 2025-12-25 04:40:28,440 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1007ms') 2025-12-25 04:40:28,440 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-31633523 failed! 2025-12-25 04:40:28,459 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 is 192.168.1.254 and allocation pool is NOT set 2025-12-25 04:40:32,535 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-12-25 04:40:32,535 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-31633523 failed! 2025-12-25 04:40:32,551 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 is 192.168.1.254 and allocation pool is NOT set 2025-12-25 04:40:36,629 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2025-12-25 04:40:36,629 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-31633523 failed! 2025-12-25 04:40:36,692 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 is 192.168.1.254 and allocation pool is NOT set 2025-12-25 04:40:40,790 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1033ms') 2025-12-25 04:40:40,790 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-31633523 failed! 2025-12-25 04:40:40,807 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 is 192.168.1.254 and allocation pool is NOT set 2025-12-25 04:40:42,872 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=5.54 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=2.08 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 2.080/3.808/5.537/1.728 ms') 2025-12-25 04:40:42,873 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-31633523 passed 2025-12-25 04:40:42,952 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:40:42,952 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-31633523, IP 192.168.1.252, Port 22 2025-12-25 04:40:43,021 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-25 04:40:43,235 - DEBUG - VM ctest-vn1_vm1-31633523 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-25 04:40:48,235 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:40:48,235 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-31633523, IP 192.168.1.252, Port 22 2025-12-25 04:40:48,304 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-25 04:40:48,487 - DEBUG - VM ctest-vn1_vm1-31633523 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-25 04:40:53,488 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:40:53,489 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-31633523, IP 192.168.1.252, Port 22 2025-12-25 04:40:53,555 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-25 04:40:53,663 - DEBUG - VM ctest-vn1_vm1-31633523 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-25 04:40:58,664 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:40:58,664 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-31633523, IP 192.168.1.252, Port 22 2025-12-25 04:40:58,732 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-25 04:40:58,878 - DEBUG - VM ctest-vn1_vm1-31633523 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-25 04:41:03,879 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:41:03,879 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-31633523, IP 192.168.1.252, Port 22 2025-12-25 04:41:04,052 - DEBUG - VM ctest-vn1_vm1-31633523 is ready for SSH connections 2025-12-25 04:41:04,052 - INFO - Waiting for VM ctest-vn2_vm1-00543289 to be up.. 2025-12-25 04:41:04,372 - DEBUG - VM is in ACTIVE state now 2025-12-25 04:41:04,372 - INFO - VM name : ctest-vn2_vm1-00543289 2025-12-25 04:41:04,479 - DEBUG - VM ctest-vn2_vm1-00543289 ID is 85a64a76-d82c-43be-a7e5-3512e9d70902 2025-12-25 04:41:04,479 - DEBUG - VM ctest-vn2_vm1-00543289 launched on Node an-jenkins-deploy-platform-ansible-os-4927-2 2025-12-25 04:41:04,600 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/85a64a76-d82c-43be-a7e5-3512e9d70902 2025-12-25 04:41:04,655 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/4410cce1-743a-4da7-a2ba-fae322db9f39 2025-12-25 04:41:05,841 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=2.63 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.995 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 0.995/1.812/2.630/0.817 ms') 2025-12-25 04:41:05,841 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn2_vm1-00543289 passed 2025-12-25 04:41:05,914 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:41:05,914 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-00543289, IP 192.168.2.252, Port 22 2025-12-25 04:41:05,981 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-25 04:41:06,100 - DEBUG - VM ctest-vn2_vm1-00543289 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-25 04:41:11,101 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:41:11,101 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-00543289, IP 192.168.2.252, Port 22 2025-12-25 04:41:11,273 - DEBUG - VM ctest-vn2_vm1-00543289 is ready for SSH connections 2025-12-25 04:41:11,273 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:41:11,273 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.252, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.19, gateway password: c0ntrail123 2025-12-25 04:41:11,273 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.252 2025-12-25 04:41:31,162 - DEBUG - PING 192.168.2.252 (192.168.2.252) 56(84) bytes of data. 64 bytes from 192.168.2.252: icmp_req=1 ttl=63 time=6.83 ms 64 bytes from 192.168.2.252: icmp_req=2 ttl=63 time=1.59 ms 64 bytes from 192.168.2.252: icmp_req=3 ttl=63 time=2.31 ms --- 192.168.2.252 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.598/3.584/6.838/2.320 ms 2025-12-25 04:41:31,163 - INFO - Ping to IP 192.168.2.252 from VM ctest-vn1_vm1-31633523 passed 2025-12-25 04:41:34,626 - INFO - 10.0.0.19 2025-12-25 04:41:34,633 - INFO - agent:initializing 2025-12-25 04:41:34,633 - DEBUG - defaultdict(, {'10.0.0.19': {'agent': {'status': 'initializing', 'description': None}}}) 2025-12-25 04:41:34,633 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-12-25 04:41:39,739 - INFO - 10.0.0.19 2025-12-25 04:41:39,746 - INFO - agent:active 2025-12-25 04:41:39,747 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.19 2025-12-25 04:41:43,164 - INFO - 10.0.0.130 2025-12-25 04:41:43,170 - INFO - agent:initializing 2025-12-25 04:41:43,171 - DEBUG - defaultdict(, {'10.0.0.130': {'agent': {'status': 'initializing', 'description': None}}}) 2025-12-25 04:41:43,171 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-12-25 04:41:48,269 - INFO - 10.0.0.130 2025-12-25 04:41:48,275 - INFO - agent:active 2025-12-25 04:41:48,276 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.130 2025-12-25 04:41:48,971 - INFO - 10.0.0.27 2025-12-25 04:41:48,975 - INFO - control:initializing 2025-12-25 04:41:48,975 - DEBUG - defaultdict(, {'10.0.0.27': {'control': {'status': 'initializing', 'description': None}}}) 2025-12-25 04:41:48,975 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-12-25 04:41:54,098 - INFO - 10.0.0.27 2025-12-25 04:41:54,105 - INFO - control:active 2025-12-25 04:41:54,105 - INFO - Contrail services ['control'] are up on nodes 10.0.0.27 2025-12-25 04:41:54,567 - INFO - 10.0.0.33 2025-12-25 04:41:54,574 - INFO - control:initializing 2025-12-25 04:41:54,574 - DEBUG - defaultdict(, {'10.0.0.33': {'control': {'status': 'initializing', 'description': None}}}) 2025-12-25 04:41:54,574 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-12-25 04:41:59,688 - INFO - 10.0.0.33 2025-12-25 04:41:59,694 - INFO - control:active 2025-12-25 04:41:59,695 - INFO - Contrail services ['control'] are up on nodes 10.0.0.33 2025-12-25 04:42:00,193 - INFO - 10.0.0.23 2025-12-25 04:42:00,199 - INFO - control:initializing 2025-12-25 04:42:00,199 - DEBUG - defaultdict(, {'10.0.0.23': {'control': {'status': 'initializing', 'description': None}}}) 2025-12-25 04:42:00,199 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-12-25 04:42:05,299 - INFO - 10.0.0.23 2025-12-25 04:42:05,305 - INFO - control:active 2025-12-25 04:42:05,306 - INFO - Contrail services ['control'] are up on nodes 10.0.0.23 2025-12-25 04:42:06,225 - INFO - 10.0.0.27 2025-12-25 04:42:06,233 - INFO - api-server:initializing 2025-12-25 04:42:06,235 - DEBUG - defaultdict(, {'10.0.0.27': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-12-25 04:42:06,235 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-12-25 04:42:11,410 - INFO - 10.0.0.27 2025-12-25 04:42:11,414 - INFO - api-server:initializing 2025-12-25 04:42:11,414 - DEBUG - defaultdict(, {'10.0.0.27': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-12-25 04:42:11,414 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-12-25 04:42:16,530 - INFO - 10.0.0.27 2025-12-25 04:42:16,536 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2025-12-25 04:42:16,536 - DEBUG - defaultdict(, {'10.0.0.27': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2025-12-25 04:42:16,536 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-12-25 04:42:21,680 - INFO - 10.0.0.27 2025-12-25 04:42:21,688 - INFO - api-server:active 2025-12-25 04:42:21,688 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.27 2025-12-25 04:42:22,433 - INFO - 10.0.0.33 2025-12-25 04:42:22,437 - INFO - api-server:initializing 2025-12-25 04:42:22,437 - DEBUG - defaultdict(, {'10.0.0.33': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-12-25 04:42:22,437 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-12-25 04:42:27,551 - INFO - 10.0.0.33 2025-12-25 04:42:27,556 - INFO - api-server:initializing 2025-12-25 04:42:27,556 - DEBUG - defaultdict(, {'10.0.0.33': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-12-25 04:42:27,556 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-12-25 04:42:32,656 - INFO - 10.0.0.33 2025-12-25 04:42:32,663 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2025-12-25 04:42:32,663 - DEBUG - defaultdict(, {'10.0.0.33': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2025-12-25 04:42:32,663 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-12-25 04:42:37,776 - INFO - 10.0.0.33 2025-12-25 04:42:37,781 - INFO - api-server:active 2025-12-25 04:42:37,782 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.33 2025-12-25 04:42:38,539 - INFO - 10.0.0.23 2025-12-25 04:42:38,544 - INFO - api-server:initializing 2025-12-25 04:42:38,544 - DEBUG - defaultdict(, {'10.0.0.23': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-12-25 04:42:38,544 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-12-25 04:42:43,662 - INFO - 10.0.0.23 2025-12-25 04:42:43,668 - INFO - api-server:initializing 2025-12-25 04:42:43,668 - DEBUG - defaultdict(, {'10.0.0.23': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-12-25 04:42:43,668 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-12-25 04:42:48,770 - INFO - 10.0.0.23 2025-12-25 04:42:48,777 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2025-12-25 04:42:48,777 - DEBUG - defaultdict(, {'10.0.0.23': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2025-12-25 04:42:48,777 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-12-25 04:42:53,891 - INFO - 10.0.0.23 2025-12-25 04:42:53,898 - INFO - api-server:active 2025-12-25 04:42:53,899 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.23 2025-12-25 04:42:54,034 - INFO - 10.0.0.27 2025-12-25 04:42:54,034 - INFO - config 2025-12-25 04:42:54,041 - INFO - config-nodemgr:active 2025-12-25 04:42:54,046 - INFO - api-server:active 2025-12-25 04:42:54,050 - INFO - schema:backup 2025-12-25 04:42:54,053 - INFO - svc-monitor:backup 2025-12-25 04:42:54,056 - INFO - device-manager:backup 2025-12-25 04:42:54,056 - INFO - config-database 2025-12-25 04:42:54,056 - INFO - config-cassandra:active 2025-12-25 04:42:54,056 - INFO - config-zookeeper:active 2025-12-25 04:42:54,056 - INFO - config-rabbitmq:active 2025-12-25 04:42:54,056 - INFO - control 2025-12-25 04:42:54,062 - INFO - control-nodemgr:active 2025-12-25 04:42:54,069 - INFO - control:active 2025-12-25 04:42:54,069 - INFO - named:active 2025-12-25 04:42:54,077 - INFO - dns:active 2025-12-25 04:42:54,078 - INFO - analytics 2025-12-25 04:42:54,084 - INFO - analytics-nodemgr:active 2025-12-25 04:42:54,091 - INFO - analytics-api:active 2025-12-25 04:42:54,098 - INFO - collector:active 2025-12-25 04:42:54,099 - INFO - analytics-database 2025-12-25 04:42:54,099 - INFO - analytics-cassandra:active 2025-12-25 04:42:54,104 - INFO - analyticsdb-nodemgr:active 2025-12-25 04:42:54,113 - INFO - query-engine:active 2025-12-25 04:42:54,113 - INFO - webui 2025-12-25 04:42:54,114 - INFO - webui:active 2025-12-25 04:42:54,114 - INFO - webui-middleware:active 2025-12-25 04:42:54,114 - INFO - redis:active 2025-12-25 04:42:54,114 - INFO - analytics_snmp 2025-12-25 04:42:54,119 - INFO - snmp-collector:active 2025-12-25 04:42:54,119 - INFO - snmp-topology:active 2025-12-25 04:42:54,119 - INFO - analytics_alarm 2025-12-25 04:42:54,119 - INFO - alarmgen:active 2025-12-25 04:42:54,219 - INFO - 10.0.0.33 2025-12-25 04:42:54,219 - INFO - config 2025-12-25 04:42:54,227 - INFO - config-nodemgr:active 2025-12-25 04:42:54,232 - INFO - api-server:active 2025-12-25 04:42:54,240 - INFO - schema:active 2025-12-25 04:42:54,245 - INFO - svc-monitor:active 2025-12-25 04:42:54,251 - INFO - device-manager:active 2025-12-25 04:42:54,251 - INFO - config-database 2025-12-25 04:42:54,251 - INFO - config-cassandra:active 2025-12-25 04:42:54,251 - INFO - config-zookeeper:active 2025-12-25 04:42:54,251 - INFO - config-rabbitmq:active 2025-12-25 04:42:54,251 - INFO - control 2025-12-25 04:42:54,257 - INFO - control-nodemgr:active 2025-12-25 04:42:54,263 - INFO - control:active 2025-12-25 04:42:54,263 - INFO - named:active 2025-12-25 04:42:54,270 - INFO - dns:active 2025-12-25 04:42:54,270 - INFO - analytics 2025-12-25 04:42:54,276 - INFO - analytics-nodemgr:active 2025-12-25 04:42:54,285 - INFO - analytics-api:active 2025-12-25 04:42:54,291 - INFO - collector:active 2025-12-25 04:42:54,292 - INFO - analytics-database 2025-12-25 04:42:54,292 - INFO - analytics-cassandra:active 2025-12-25 04:42:54,298 - INFO - analyticsdb-nodemgr:active 2025-12-25 04:42:54,305 - INFO - query-engine:active 2025-12-25 04:42:54,305 - INFO - webui 2025-12-25 04:42:54,305 - INFO - webui:active 2025-12-25 04:42:54,306 - INFO - webui-middleware:active 2025-12-25 04:42:54,306 - INFO - redis:active 2025-12-25 04:42:54,306 - INFO - analytics_snmp 2025-12-25 04:42:54,312 - INFO - snmp-collector:active 2025-12-25 04:42:54,312 - INFO - snmp-topology:active 2025-12-25 04:42:54,312 - INFO - analytics_alarm 2025-12-25 04:42:54,312 - INFO - alarmgen:active 2025-12-25 04:42:54,416 - INFO - 10.0.0.23 2025-12-25 04:42:54,416 - INFO - config 2025-12-25 04:42:54,424 - INFO - config-nodemgr:active 2025-12-25 04:42:54,430 - INFO - api-server:active 2025-12-25 04:42:54,436 - INFO - schema:backup 2025-12-25 04:42:54,440 - INFO - svc-monitor:backup 2025-12-25 04:42:54,444 - INFO - device-manager:backup 2025-12-25 04:42:54,444 - INFO - config-database 2025-12-25 04:42:54,444 - INFO - config-cassandra:active 2025-12-25 04:42:54,445 - INFO - config-zookeeper:active 2025-12-25 04:42:54,445 - INFO - config-rabbitmq:active 2025-12-25 04:42:54,445 - INFO - control 2025-12-25 04:42:54,450 - INFO - control-nodemgr:active 2025-12-25 04:42:54,457 - INFO - control:active 2025-12-25 04:42:54,457 - INFO - named:active 2025-12-25 04:42:54,464 - INFO - dns:active 2025-12-25 04:42:54,464 - INFO - analytics 2025-12-25 04:42:54,470 - INFO - analytics-nodemgr:active 2025-12-25 04:42:54,476 - INFO - analytics-api:active 2025-12-25 04:42:54,484 - INFO - collector:active 2025-12-25 04:42:54,484 - INFO - analytics-database 2025-12-25 04:42:54,484 - INFO - analytics-cassandra:active 2025-12-25 04:42:54,491 - INFO - analyticsdb-nodemgr:active 2025-12-25 04:42:54,498 - INFO - query-engine:active 2025-12-25 04:42:54,498 - INFO - webui 2025-12-25 04:42:54,498 - INFO - webui:active 2025-12-25 04:42:54,498 - INFO - webui-middleware:active 2025-12-25 04:42:54,498 - INFO - redis:active 2025-12-25 04:42:54,499 - INFO - analytics_snmp 2025-12-25 04:42:54,505 - INFO - snmp-collector:active 2025-12-25 04:42:54,505 - INFO - snmp-topology:active 2025-12-25 04:42:54,505 - INFO - analytics_alarm 2025-12-25 04:42:54,505 - INFO - alarmgen:active 2025-12-25 04:42:54,603 - INFO - 10.0.0.19 2025-12-25 04:42:54,604 - INFO - vrouter 2025-12-25 04:42:54,610 - INFO - vrouter-nodemgr:active 2025-12-25 04:42:54,618 - INFO - agent:active 2025-12-25 04:42:54,707 - INFO - 10.0.0.130 2025-12-25 04:42:54,708 - INFO - vrouter 2025-12-25 04:42:54,724 - INFO - vrouter-nodemgr:active 2025-12-25 04:42:54,732 - INFO - agent:active 2025-12-25 04:42:54,733 - INFO - All the contrail services are up on all nodes 2025-12-25 04:42:54,733 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-1?flat 2025-12-25 04:42:54,846 - INFO - an-jenkins-deploy-platform-ansible-os-4927-1:contrail-vrouter-agent module connection to ['10.20.0.209:5269'] servers UP 2025-12-25 04:42:54,847 - INFO - an-jenkins-deploy-platform-ansible-os-4927-1 connected to xmpp 10.20.0.209 2025-12-25 04:42:54,847 - INFO - an-jenkins-deploy-platform-ansible-os-4927-1:contrail-vrouter-agent module connection to ['10.20.0.14:5269'] servers UP 2025-12-25 04:42:54,847 - INFO - an-jenkins-deploy-platform-ansible-os-4927-1 connected to xmpp 10.20.0.14 2025-12-25 04:42:54,847 - WARNING - an-jenkins-deploy-platform-ansible-os-4927-1:contrail-vrouter-agent module connection to ['10.20.0.18:5269'] servers NOT UP 2025-12-25 04:42:54,847 - INFO - an-jenkins-deploy-platform-ansible-os-4927-1:contrail-vrouter-agent module connection to ['10.20.0.209:53'] servers UP 2025-12-25 04:42:54,847 - INFO - an-jenkins-deploy-platform-ansible-os-4927-1 connected to dns 10.20.0.209 2025-12-25 04:42:54,847 - INFO - an-jenkins-deploy-platform-ansible-os-4927-1:contrail-vrouter-agent module connection to ['10.20.0.14:53'] servers UP 2025-12-25 04:42:54,847 - INFO - an-jenkins-deploy-platform-ansible-os-4927-1 connected to dns 10.20.0.14 2025-12-25 04:42:54,847 - WARNING - an-jenkins-deploy-platform-ansible-os-4927-1:contrail-vrouter-agent module connection to ['10.20.0.18:53'] servers NOT UP 2025-12-25 04:42:54,847 - WARNING - an-jenkins-deploy-platform-ansible-os-4927-1:contrail-vrouter-agent module connection to ['10.0.0.27:8086'] servers NOT UP 2025-12-25 04:42:54,847 - WARNING - an-jenkins-deploy-platform-ansible-os-4927-1:contrail-vrouter-agent module connection to ['10.0.0.33:8086'] servers NOT UP 2025-12-25 04:42:54,847 - INFO - an-jenkins-deploy-platform-ansible-os-4927-1:contrail-vrouter-agent module connection to ['10.0.0.23:8086'] servers UP 2025-12-25 04:42:54,847 - INFO - an-jenkins-deploy-platform-ansible-os-4927-1 connected to collector 10.0.0.23 2025-12-25 04:42:54,848 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4927-1?flat 2025-12-25 04:42:55,325 - INFO - cn-jenkins-deploy-platform-ansible-os-4927-1:contrail-control module connection to 10.0.0.27:9041 servers UP 2025-12-25 04:42:55,325 - INFO - cn-jenkins-deploy-platform-ansible-os-4927-1:contrail-control module connection to 10.0.0.27:5673 servers UP 2025-12-25 04:42:55,325 - INFO - cn-jenkins-deploy-platform-ansible-os-4927-1:contrail-control module connection to ['10.0.0.27:8086'] servers UP 2025-12-25 04:42:55,325 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4927-2?flat 2025-12-25 04:42:55,420 - INFO - cn-jenkins-deploy-platform-ansible-os-4927-2:contrail-control module connection to 10.0.0.27:9041 servers UP 2025-12-25 04:42:55,421 - INFO - cn-jenkins-deploy-platform-ansible-os-4927-2:contrail-control module connection to 10.0.0.27:5673 servers UP 2025-12-25 04:42:55,421 - INFO - cn-jenkins-deploy-platform-ansible-os-4927-2:contrail-control module connection to ['10.0.0.27:8086'] servers UP 2025-12-25 04:42:55,421 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4927-3?flat 2025-12-25 04:42:55,445 - INFO - cn-jenkins-deploy-platform-ansible-os-4927-3:contrail-control module connection to 10.0.0.27:9041 servers UP 2025-12-25 04:42:55,445 - INFO - cn-jenkins-deploy-platform-ansible-os-4927-3:contrail-control module connection to 10.0.0.27:5673 servers UP 2025-12-25 04:42:55,445 - INFO - cn-jenkins-deploy-platform-ansible-os-4927-3:contrail-control module connection to ['10.0.0.27:8086'] servers UP 2025-12-25 04:42:55,446 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4927-1.?flat 2025-12-25 04:42:55,471 - DEBUG - contrail-api is running 2025-12-25 04:42:55,471 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4927-2.?flat 2025-12-25 04:42:55,501 - DEBUG - contrail-api is running 2025-12-25 04:42:55,501 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4927-3.?flat 2025-12-25 04:42:55,544 - DEBUG - contrail-api is running 2025-12-25 04:42:55,892 - INFO - Created VN ctest-vn3-14620459, UUID :7aefa9b9-772e-4690-ac8e-1bf0ff57747f 2025-12-25 04:42:56,388 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-12-25 04:42:56,463 - DEBUG - Requesting: http://10.0.0.27:8082/domain/d4f5d7ea-1779-482f-9e93-03873d431e99 2025-12-25 04:42:56,479 - DEBUG - Requesting: http://10.0.0.27:8082/project/edba148a-64cc-4e4a-8216-4c7dbc6a7d87 2025-12-25 04:42:56,614 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/d07b1963-2a5b-47ff-8f83-4d94e5eac0ea 2025-12-25 04:42:56,633 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/d07b1963-2a5b-47ff-8f83-4d94e5eac0ea 2025-12-25 04:42:56,651 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/226e8db5-3080-4ae5-944b-3627122b62a7 2025-12-25 04:42:56,666 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/226e8db5-3080-4ae5-944b-3627122b62a7 2025-12-25 04:42:56,677 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/ceef7a46-8d77-42c3-af18-68b90e1cda02 2025-12-25 04:42:56,691 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-25 04:42:56,691 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/d07b1963-2a5b-47ff-8f83-4d94e5eac0ea 2025-12-25 04:42:56,706 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/226e8db5-3080-4ae5-944b-3627122b62a7 2025-12-25 04:42:56,730 - INFO - Verified VN network id 15 for VN d07b1963-2a5b-47ff-8f83-4d94e5eac0ea 2025-12-25 04:42:56,732 - INFO - Verifications in API Server for VN ctest-vn1-91861474 passed 2025-12-25 04:42:56,732 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/d07b1963-2a5b-47ff-8f83-4d94e5eac0ea 2025-12-25 04:42:56,745 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/226e8db5-3080-4ae5-944b-3627122b62a7 2025-12-25 04:42:56,759 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/226e8db5-3080-4ae5-944b-3627122b62a7 2025-12-25 04:42:56,772 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/ceef7a46-8d77-42c3-af18-68b90e1cda02 2025-12-25 04:42:56,789 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474', 'interests': '1', 'advertised': '1', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-19229946', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '15022628895727503359', 'uuid-lslong': '10341194471280656618'}, 'enable': 'true', 'created': '2025-12-25T04:39:52', 'last-modified': '2025-12-25T04:39:53', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:07.080629'} 2025-12-25 04:42:56,801 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-25 04:42:56,809 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-19229946', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '15022628895727503359', 'uuid-lslong': '10341194471280656618'}, 'enable': 'true', 'created': '2025-12-25T04:39:52', 'last-modified': '2025-12-25T04:39:53', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:01.763389'} 2025-12-25 04:42:56,816 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-25 04:42:56,822 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-19229946', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '15022628895727503359', 'uuid-lslong': '10341194471280656618'}, 'enable': 'true', 'created': '2025-12-25T04:39:52', 'last-modified': '2025-12-25T04:39:53', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:56.119971'} 2025-12-25 04:42:56,830 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-25 04:42:56,831 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-91861474 passed 2025-12-25 04:42:56,831 - DEBUG - ====Verifying policy data for ctest-vn1-91861474 in API_Server ====== 2025-12-25 04:42:56,831 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-12-25 04:42:56,839 - DEBUG - Requesting: http://10.0.0.27:8082/domain/d4f5d7ea-1779-482f-9e93-03873d431e99 2025-12-25 04:42:56,858 - DEBUG - Requesting: http://10.0.0.27:8082/project/edba148a-64cc-4e4a-8216-4c7dbc6a7d87 2025-12-25 04:42:56,964 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/d07b1963-2a5b-47ff-8f83-4d94e5eac0ea 2025-12-25 04:42:56,986 - DEBUG - ==>Verifying data for policy with id: e903da13-8139-4fd8-864b-14eed02ac0cb, fqn: ['default-domain', 'ctest-TestBasicVMVN0-19229946', 'policy1'] 2025-12-25 04:42:56,986 - INFO - VN ctest-vn1-91861474 Policy verification: verify_vn_policy_in_api_server, status: True 2025-12-25 04:42:56,986 - DEBUG - Verifying the vn in opserver 2025-12-25 04:42:56,986 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 virtual network link through opserver 10.0.0.27 2025-12-25 04:42:56,987 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-12-25 04:42:57,005 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474?flat'} 2025-12-25 04:42:57,005 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 virtual network link through opserver 10.0.0.33 2025-12-25 04:42:57,005 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2025-12-25 04:42:57,093 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474?flat'} 2025-12-25 04:42:57,093 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 virtual network link through opserver 10.0.0.23 2025-12-25 04:42:57,093 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-12-25 04:42:57,202 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474?flat'} 2025-12-25 04:42:57,202 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 is found in opserver 2025-12-25 04:42:57,219 - DEBUG - VRF ids for VN ctest-vn1-91861474: {'10.0.0.19': '2'} 2025-12-25 04:42:58,913 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4927-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4927-1) 2025-12-25 04:42:58,985 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-12-25 04:43:04,101 - DEBUG - VM is in ACTIVE state now 2025-12-25 04:43:04,101 - INFO - VM name : ctest-vn1_vm2-40724543 2025-12-25 04:43:04,236 - DEBUG - VM ctest-vn1_vm2-40724543 ID is 3f47f918-a437-4e91-b959-7581c2cbd8e4 2025-12-25 04:43:04,361 - DEBUG - VM is in ACTIVE state now 2025-12-25 04:43:04,361 - DEBUG - VM ctest-vn1_vm2-40724543 launched on Node an-jenkins-deploy-platform-ansible-os-4927-1 2025-12-25 04:43:04,482 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/3f47f918-a437-4e91-b959-7581c2cbd8e4 2025-12-25 04:43:04,531 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/3f47f918-a437-4e91-b959-7581c2cbd8e4 2025-12-25 04:43:04,544 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/4902e508-5e53-427d-abd6-5decadce6098 2025-12-25 04:43:04,557 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/3f47f918-a437-4e91-b959-7581c2cbd8e4 2025-12-25 04:43:04,570 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/4902e508-5e53-427d-abd6-5decadce6098 2025-12-25 04:43:04,586 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/951690c5-0e15-4225-8e00-4542437ebb01 2025-12-25 04:43:04,603 - DEBUG - Verifying in api server 10.0.0.23 2025-12-25 04:43:04,603 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/3f47f918-a437-4e91-b959-7581c2cbd8e4 2025-12-25 04:43:04,615 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/4902e508-5e53-427d-abd6-5decadce6098 2025-12-25 04:43:04,631 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/951690c5-0e15-4225-8e00-4542437ebb01 2025-12-25 04:43:04,643 - INFO - VM ctest-vn1_vm2-40724543 verfication in all API Servers passed 2025-12-25 04:43:04,678 - DEBUG - VM ctest-vn1_vm2-40724543 Tap interface: {'index': '4', 'name': 'tap4902e508-5e', 'uuid': '4902e508-5e53-427d-abd6-5decadce6098', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474:ctest-vn1-91861474', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474', 'vm_uuid': '3f47f918-a437-4e91-b959-7581c2cbd8e4', 'vm_name': 'ctest-vn1_vm2-40724543', 'ip_addr': '192.168.1.251', 'mac_addr': '02:49:02:e5:08:5e', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '164', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-19229946:4902e508-5e53-427d-abd6-5decadce6098', 'sg_uuid_list': ['71e965cc-01d3-4f06-aa5d-b1e37addc289'], 'static_route_list': None, 'vm_project_uuid': 'edba148a-64cc-4e4a-8216-4c7dbc6a7d87', 'admin_state': 'Enabled', 'flow_key_idx': '31', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-25 04:43:04,678 - DEBUG - Agent 10.0.0.19 vrf name: default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474:ctest-vn1-91861474 2025-12-25 04:43:04,688 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474:ctest-vn1-91861474', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474', 'table_label': '-1', 'vxlan_id': '15', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.252:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-12-25 04:43:04,799 - DEBUG - VM ctest-vn1_vm2-40724543 : Tap interface tap4902e508-5e is set to Active 2025-12-25 04:43:04,800 - DEBUG - Tap interface tap4902e508-5e detail : {'index': '4', 'name': 'tap4902e508-5e', 'uuid': '4902e508-5e53-427d-abd6-5decadce6098', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474:ctest-vn1-91861474', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474', 'vm_uuid': '3f47f918-a437-4e91-b959-7581c2cbd8e4', 'vm_name': 'ctest-vn1_vm2-40724543', 'ip_addr': '192.168.1.251', 'mac_addr': '02:49:02:e5:08:5e', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '164', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-19229946:4902e508-5e53-427d-abd6-5decadce6098', 'sg_uuid_list': ['71e965cc-01d3-4f06-aa5d-b1e37addc289'], 'static_route_list': None, 'vm_project_uuid': 'edba148a-64cc-4e4a-8216-4c7dbc6a7d87', 'admin_state': 'Enabled', 'flow_key_idx': '31', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-25 04:43:04,918 - DEBUG - Starting Layer 2 verification in Agent 2025-12-25 04:43:04,933 - DEBUG - Layer 2 path is seen for VM MAC 02:49:02:e5:08:5e in agent 10.0.0.19 2025-12-25 04:43:04,933 - DEBUG - Active layer 2 route in agent is present for VMI tap4902e508-5e 2025-12-25 04:43:04,933 - DEBUG - L2 label(34) matches bw route table and interface table 2025-12-25 04:43:04,948 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 is found in Agent of node 10.0.0.19 2025-12-25 04:43:05,097 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 is consistent in agent 10.0.0.19 2025-12-25 04:43:05,097 - DEBUG - Route for VM IP ['192.168.1.251'] is consistent in agent 10.0.0.19 2025-12-25 04:43:05,097 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 verification for VM ctest-vn1_vm2-40724543 in Agent 10.0.0.19 passed 2025-12-25 04:43:05,222 - DEBUG - Starting all layer 2 verification in agent 10.0.0.19 2025-12-25 04:43:05,237 - DEBUG - Route for VM MAC 02:49:02:e5:08:5e is consistent in agent 10.0.0.19 2025-12-25 04:43:08,437 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms') 2025-12-25 04:43:08,437 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-40724543 failed! 2025-12-25 04:43:08,459 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 is 192.168.1.254 and allocation pool is NOT set 2025-12-25 04:43:12,532 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms') 2025-12-25 04:43:12,532 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-40724543 failed! 2025-12-25 04:43:12,548 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 is 192.168.1.254 and allocation pool is NOT set 2025-12-25 04:43:16,628 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-12-25 04:43:16,628 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-40724543 failed! 2025-12-25 04:43:16,647 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 is 192.168.1.254 and allocation pool is NOT set 2025-12-25 04:43:20,728 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2025-12-25 04:43:20,728 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-40724543 failed! 2025-12-25 04:43:20,749 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 is 192.168.1.254 and allocation pool is NOT set 2025-12-25 04:43:24,821 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1004ms') 2025-12-25 04:43:24,821 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-40724543 failed! 2025-12-25 04:43:24,842 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 is 192.168.1.254 and allocation pool is NOT set 2025-12-25 04:43:28,917 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1005ms') 2025-12-25 04:43:28,918 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-40724543 failed! 2025-12-25 04:43:28,934 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 is 192.168.1.254 and allocation pool is NOT set 2025-12-25 04:43:33,015 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-12-25 04:43:33,015 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-40724543 failed! 2025-12-25 04:43:33,032 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 is 192.168.1.254 and allocation pool is NOT set 2025-12-25 04:43:37,111 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1012ms') 2025-12-25 04:43:37,111 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-40724543 failed! 2025-12-25 04:43:37,129 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 is 192.168.1.254 and allocation pool is NOT set 2025-12-25 04:43:41,203 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1009ms') 2025-12-25 04:43:41,203 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-40724543 failed! 2025-12-25 04:43:41,220 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 is 192.168.1.254 and allocation pool is NOT set 2025-12-25 04:43:43,292 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=12.0 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=3.03 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 3.033/7.519/12.006/4.486 ms') 2025-12-25 04:43:43,292 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-40724543 passed 2025-12-25 04:43:43,292 - INFO - VM ctest-vn1_vm2-40724543 verifications in Compute nodes passed 2025-12-25 04:43:43,524 - DEBUG - Validated VM route 192.168.1.251 in vrouter of 10.0.0.19 2025-12-25 04:43:43,530 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-12-25 04:43:43,530 - INFO - Validated routes of VM ctest-vn1_vm2-40724543 in all vrouters 2025-12-25 04:43:43,969 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2025-12-25 04:43:44,105 - DEBUG - Layer2 route found for VM MAC 02:49:02:e5:08:5e in Control-node 10.0.0.27 2025-12-25 04:43:44,105 - DEBUG - L2 Label for VM ctest-vn1_vm2-40724543 same between Control-node 10.0.0.27 and Agent, Expected: 34, Seen: 34 2025-12-25 04:43:44,105 - DEBUG - Starting all layer2 verification in 10.0.0.33 Control Node 2025-12-25 04:43:44,242 - DEBUG - Layer2 route found for VM MAC 02:49:02:e5:08:5e in Control-node 10.0.0.33 2025-12-25 04:43:44,242 - DEBUG - L2 Label for VM ctest-vn1_vm2-40724543 same between Control-node 10.0.0.33 and Agent, Expected: 34, Seen: 34 2025-12-25 04:43:44,242 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-40724543 passed 2025-12-25 04:43:44,242 - DEBUG - Verifying the vm in opserver 2025-12-25 04:43:44,242 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-12-25 04:43:44,242 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/3f47f918-a437-4e91-b959-7581c2cbd8e4?flat 2025-12-25 04:43:44,324 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-19229946:4902e508-5e53-427d-abd6-5decadce6098?flat 2025-12-25 04:43:44,349 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-19229946:4902e508-5e53-427d-abd6-5decadce6098?flat 2025-12-25 04:43:44,370 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/3f47f918-a437-4e91-b959-7581c2cbd8e4?flat 2025-12-25 04:43:44,392 - DEBUG - Verifying vm in vn uve 2025-12-25 04:43:44,392 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-19229946:4902e508-5e53-427d-abd6-5decadce6098?flat 2025-12-25 04:43:44,413 - DEBUG - VM uve shows interface as {'active': True, 'rx_vlan': 65535, 'l2_active': True, 'vn_uuid': 'd07b1963-2a5b-47ff-8f83-4d94e5eac0ea', 'vm_uuid': '3f47f918-a437-4e91-b959-7581c2cbd8e4', 'vhostuser_mode': 0, 'ip_address': '192.168.1.251', 'label': 30, 'port_mirror_enabled': False, 'ip6_active': False, 'admin_state': True, 'ip6_address': '::', 'vm_name': 'ctest-vn1_vm2-40724543', 'mac_address': '02:49:02:e5:08:5e', 'uuid': '4902e508-5e53-427d-abd6-5decadce6098', 'is_health_check_active': True, 'tx_vlan': 65535, '__T': 1766637785367233, 'ip4_active': True, 'fixed_ip4_list': ['192.168.1.251'], 'gateway': '192.168.1.254', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474'} 2025-12-25 04:43:44,413 - DEBUG - VM uve shows ip address as ['192.168.1.251', '::'] 2025-12-25 04:43:44,413 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 2025-12-25 04:43:44,413 - INFO - Verifying through opserver in 10.0.0.27 2025-12-25 04:43:44,413 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474?flat 2025-12-25 04:43:44,441 - DEBUG - expected vm list 3f47f918-a437-4e91-b959-7581c2cbd8e4 2025-12-25 04:43:44,441 - DEBUG - Extracted vm list ['3f47f918-a437-4e91-b959-7581c2cbd8e4'] 2025-12-25 04:43:44,441 - INFO - VM 3f47f918-a437-4e91-b959-7581c2cbd8e4 is present in default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 2025-12-25 04:43:44,441 - DEBUG - Verifying vm in vrouter uve 2025-12-25 04:43:44,441 - DEBUG - Getting info from collector 10.0.0.23.. 2025-12-25 04:43:44,441 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/3f47f918-a437-4e91-b959-7581c2cbd8e4?flat 2025-12-25 04:43:44,528 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4927-1'] 2025-12-25 04:43:44,528 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-1:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:43:44,586 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-1?flat 2025-12-25 04:43:44,622 - DEBUG - VM 3f47f918-a437-4e91-b959-7581c2cbd8e4 is present in vrouter an-jenkins-deploy-platform-ansible-os-4927-1 uve 2025-12-25 04:43:44,622 - DEBUG - Validated that VM ctest-vn1_vm2-40724543 is in Vrouter an-jenkins-deploy-platform-ansible-os-4927-1 UVE 2025-12-25 04:43:44,622 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-12-25 04:43:44,635 - DEBUG - Expected tap interface of VM uuid 3f47f918-a437-4e91-b959-7581c2cbd8e4 is default-domain:ctest-TestBasicVMVN0-19229946:4902e508-5e53-427d-abd6-5decadce6098 2025-12-25 04:43:44,635 - DEBUG - Expected VN of VM uuid 3f47f918-a437-4e91-b959-7581c2cbd8e4 is default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 2025-12-25 04:43:44,636 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-1:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:43:44,691 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-1?flat 2025-12-25 04:43:44,728 - DEBUG - VM 3f47f918-a437-4e91-b959-7581c2cbd8e4 is present in vrouter an-jenkins-deploy-platform-ansible-os-4927-1 uve 2025-12-25 04:43:44,728 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-19229946:4902e508-5e53-427d-abd6-5decadce6098 of vm 3f47f918-a437-4e91-b959-7581c2cbd8e4 is present in vrouter an-jenkins-deploy-platform-ansible-os-4927-1 uve 2025-12-25 04:43:44,728 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 of vm 3f47f918-a437-4e91-b959-7581c2cbd8e4 is present in vrouter an-jenkins-deploy-platform-ansible-os-4927-1 uve 2025-12-25 04:43:44,728 - DEBUG - Verifying the 3f47f918-a437-4e91-b959-7581c2cbd8e4 virtual network link through opserver 10.0.0.27 2025-12-25 04:43:44,728 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-12-25 04:43:44,742 - DEBUG - VM link and name as {'name': '3f47f918-a437-4e91-b959-7581c2cbd8e4', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/3f47f918-a437-4e91-b959-7581c2cbd8e4?flat'} 2025-12-25 04:43:44,742 - DEBUG - Verifying the 3f47f918-a437-4e91-b959-7581c2cbd8e4 virtual network link through opserver 10.0.0.33 2025-12-25 04:43:44,742 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-12-25 04:43:44,822 - DEBUG - VM link and name as {'name': '3f47f918-a437-4e91-b959-7581c2cbd8e4', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-machine/3f47f918-a437-4e91-b959-7581c2cbd8e4?flat'} 2025-12-25 04:43:44,822 - DEBUG - Verifying the 3f47f918-a437-4e91-b959-7581c2cbd8e4 virtual network link through opserver 10.0.0.23 2025-12-25 04:43:44,822 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-12-25 04:43:44,836 - DEBUG - VM link and name as {'name': '3f47f918-a437-4e91-b959-7581c2cbd8e4', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/3f47f918-a437-4e91-b959-7581c2cbd8e4?flat'} 2025-12-25 04:43:44,836 - INFO - VM ctest-vn1_vm2-40724543 validations in Opserver passed 2025-12-25 04:43:46,265 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4927-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4927-2) 2025-12-25 04:43:46,330 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-12-25 04:43:51,449 - DEBUG - VM is in ACTIVE state now 2025-12-25 04:43:51,449 - INFO - VM name : ctest-vn2_vm2-33671587 2025-12-25 04:43:51,568 - DEBUG - VM ctest-vn2_vm2-33671587 ID is 760304cd-7856-4dd0-9878-3de3b0a5b574 2025-12-25 04:43:51,684 - DEBUG - VM is in ACTIVE state now 2025-12-25 04:43:51,684 - DEBUG - VM ctest-vn2_vm2-33671587 launched on Node an-jenkins-deploy-platform-ansible-os-4927-2 2025-12-25 04:43:51,790 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/760304cd-7856-4dd0-9878-3de3b0a5b574 2025-12-25 04:43:51,806 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/760304cd-7856-4dd0-9878-3de3b0a5b574 2025-12-25 04:43:51,816 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/6851c8bd-0cc3-4c6e-8068-a68c22f2e37f 2025-12-25 04:43:51,831 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/760304cd-7856-4dd0-9878-3de3b0a5b574 2025-12-25 04:43:51,840 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/6851c8bd-0cc3-4c6e-8068-a68c22f2e37f 2025-12-25 04:43:51,851 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/f2bc396f-77cf-46b0-9a74-d9020668d8b7 2025-12-25 04:43:51,863 - DEBUG - Verifying in api server 10.0.0.23 2025-12-25 04:43:51,863 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/760304cd-7856-4dd0-9878-3de3b0a5b574 2025-12-25 04:43:51,872 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/6851c8bd-0cc3-4c6e-8068-a68c22f2e37f 2025-12-25 04:43:51,883 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/f2bc396f-77cf-46b0-9a74-d9020668d8b7 2025-12-25 04:43:51,895 - INFO - VM ctest-vn2_vm2-33671587 verfication in all API Servers passed 2025-12-25 04:43:51,916 - DEBUG - VM ctest-vn2_vm2-33671587 Tap interface: {'index': '4', 'name': 'tap6851c8bd-0c', 'uuid': '6851c8bd-0cc3-4c6e-8068-a68c22f2e37f', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257:ctest-vn2-64314257', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '16', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257', 'vm_uuid': '760304cd-7856-4dd0-9878-3de3b0a5b574', 'vm_name': 'ctest-vn2_vm2-33671587', 'ip_addr': '192.168.2.251', 'mac_addr': '02:68:51:c8:bd:0c', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '131', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-19229946:6851c8bd-0cc3-4c6e-8068-a68c22f2e37f', 'sg_uuid_list': ['71e965cc-01d3-4f06-aa5d-b1e37addc289'], 'static_route_list': None, 'vm_project_uuid': 'edba148a-64cc-4e4a-8216-4c7dbc6a7d87', 'admin_state': 'Enabled', 'flow_key_idx': '26', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-25 04:43:51,916 - DEBUG - Agent 10.0.0.130 vrf name: default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257:ctest-vn2-64314257 2025-12-25 04:43:51,923 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257:ctest-vn2-64314257', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257', 'table_label': '-1', 'vxlan_id': '16', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.249:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-12-25 04:43:52,013 - DEBUG - VM ctest-vn2_vm2-33671587 : Tap interface tap6851c8bd-0c is set to Active 2025-12-25 04:43:52,014 - DEBUG - Tap interface tap6851c8bd-0c detail : {'index': '4', 'name': 'tap6851c8bd-0c', 'uuid': '6851c8bd-0cc3-4c6e-8068-a68c22f2e37f', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257:ctest-vn2-64314257', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '16', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257', 'vm_uuid': '760304cd-7856-4dd0-9878-3de3b0a5b574', 'vm_name': 'ctest-vn2_vm2-33671587', 'ip_addr': '192.168.2.251', 'mac_addr': '02:68:51:c8:bd:0c', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '131', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-19229946:6851c8bd-0cc3-4c6e-8068-a68c22f2e37f', 'sg_uuid_list': ['71e965cc-01d3-4f06-aa5d-b1e37addc289'], 'static_route_list': None, 'vm_project_uuid': 'edba148a-64cc-4e4a-8216-4c7dbc6a7d87', 'admin_state': 'Enabled', 'flow_key_idx': '26', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-25 04:43:52,135 - DEBUG - Starting Layer 2 verification in Agent 2025-12-25 04:43:52,152 - DEBUG - Layer 2 path is seen for VM MAC 02:68:51:c8:bd:0c in agent 10.0.0.130 2025-12-25 04:43:52,152 - DEBUG - Active layer 2 route in agent is present for VMI tap6851c8bd-0c 2025-12-25 04:43:52,152 - DEBUG - L2 label(34) matches bw route table and interface table 2025-12-25 04:43:52,170 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 is found in Agent of node 10.0.0.130 2025-12-25 04:43:52,271 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 is consistent in agent 10.0.0.130 2025-12-25 04:43:52,272 - DEBUG - Route for VM IP ['192.168.2.251'] is consistent in agent 10.0.0.130 2025-12-25 04:43:52,272 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 verification for VM ctest-vn2_vm2-33671587 in Agent 10.0.0.130 passed 2025-12-25 04:43:52,386 - DEBUG - Starting all layer 2 verification in agent 10.0.0.130 2025-12-25 04:43:52,401 - DEBUG - Route for VM MAC 02:68:51:c8:bd:0c is consistent in agent 10.0.0.130 2025-12-25 04:43:55,620 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1032ms') 2025-12-25 04:43:55,621 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-33671587 failed! 2025-12-25 04:43:55,639 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 is 192.168.2.254 and allocation pool is NOT set 2025-12-25 04:43:59,714 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms') 2025-12-25 04:43:59,714 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-33671587 failed! 2025-12-25 04:43:59,731 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 is 192.168.2.254 and allocation pool is NOT set 2025-12-25 04:44:03,804 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2025-12-25 04:44:03,805 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-33671587 failed! 2025-12-25 04:44:03,822 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 is 192.168.2.254 and allocation pool is NOT set 2025-12-25 04:44:07,905 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1021ms') 2025-12-25 04:44:07,905 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-33671587 failed! 2025-12-25 04:44:07,922 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 is 192.168.2.254 and allocation pool is NOT set 2025-12-25 04:44:12,004 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1016ms') 2025-12-25 04:44:12,004 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-33671587 failed! 2025-12-25 04:44:12,021 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 is 192.168.2.254 and allocation pool is NOT set 2025-12-25 04:44:16,101 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1012ms') 2025-12-25 04:44:16,101 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-33671587 failed! 2025-12-25 04:44:16,119 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 is 192.168.2.254 and allocation pool is NOT set 2025-12-25 04:44:20,191 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1012ms') 2025-12-25 04:44:20,191 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-33671587 failed! 2025-12-25 04:44:20,211 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 is 192.168.2.254 and allocation pool is NOT set 2025-12-25 04:44:24,291 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-12-25 04:44:24,291 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-33671587 failed! 2025-12-25 04:44:24,312 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 is 192.168.2.254 and allocation pool is NOT set 2025-12-25 04:44:28,389 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1009ms') 2025-12-25 04:44:28,389 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-33671587 failed! 2025-12-25 04:44:28,412 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 is 192.168.2.254 and allocation pool is NOT set 2025-12-25 04:44:32,480 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1005ms') 2025-12-25 04:44:32,480 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-33671587 failed! 2025-12-25 04:44:32,497 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 is 192.168.2.254 and allocation pool is NOT set 2025-12-25 04:44:36,581 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=7.02 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 1 received, 50% packet loss, time 1017ms\r\nrtt min/avg/max/mdev = 7.023/7.023/7.023/0.000 ms') 2025-12-25 04:44:36,581 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-33671587 passed 2025-12-25 04:44:36,582 - INFO - VM ctest-vn2_vm2-33671587 verifications in Compute nodes passed 2025-12-25 04:44:36,827 - DEBUG - Validated VM route 192.168.2.251 in vrouter of 10.0.0.130 2025-12-25 04:44:36,834 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-12-25 04:44:36,834 - INFO - Validated routes of VM ctest-vn2_vm2-33671587 in all vrouters 2025-12-25 04:44:37,280 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2025-12-25 04:44:37,418 - DEBUG - Layer2 route found for VM MAC 02:68:51:c8:bd:0c in Control-node 10.0.0.27 2025-12-25 04:44:37,419 - DEBUG - L2 Label for VM ctest-vn2_vm2-33671587 same between Control-node 10.0.0.27 and Agent, Expected: 34, Seen: 34 2025-12-25 04:44:37,419 - DEBUG - Starting all layer2 verification in 10.0.0.33 Control Node 2025-12-25 04:44:37,549 - DEBUG - Layer2 route found for VM MAC 02:68:51:c8:bd:0c in Control-node 10.0.0.33 2025-12-25 04:44:37,549 - DEBUG - L2 Label for VM ctest-vn2_vm2-33671587 same between Control-node 10.0.0.33 and Agent, Expected: 34, Seen: 34 2025-12-25 04:44:37,549 - INFO - Verification in Control-nodes for VM ctest-vn2_vm2-33671587 passed 2025-12-25 04:44:37,549 - DEBUG - Verifying the vm in opserver 2025-12-25 04:44:37,549 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-12-25 04:44:37,549 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/760304cd-7856-4dd0-9878-3de3b0a5b574?flat 2025-12-25 04:44:37,568 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-19229946:6851c8bd-0cc3-4c6e-8068-a68c22f2e37f?flat 2025-12-25 04:44:37,594 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-19229946:6851c8bd-0cc3-4c6e-8068-a68c22f2e37f?flat 2025-12-25 04:44:37,620 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/760304cd-7856-4dd0-9878-3de3b0a5b574?flat 2025-12-25 04:44:37,640 - DEBUG - Verifying vm in vn uve 2025-12-25 04:44:37,640 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-19229946:6851c8bd-0cc3-4c6e-8068-a68c22f2e37f?flat 2025-12-25 04:44:37,662 - DEBUG - VM uve shows interface as {'active': True, 'rx_vlan': 65535, 'l2_active': True, 'vn_uuid': '0a89949f-1f81-453a-b917-59c6379e0fba', 'vm_uuid': '760304cd-7856-4dd0-9878-3de3b0a5b574', 'vhostuser_mode': 0, 'ip_address': '192.168.2.251', 'label': 30, 'port_mirror_enabled': False, 'ip6_active': False, 'admin_state': True, 'ip6_address': '::', 'vm_name': 'ctest-vn2_vm2-33671587', 'mac_address': '02:68:51:c8:bd:0c', 'uuid': '6851c8bd-0cc3-4c6e-8068-a68c22f2e37f', 'is_health_check_active': True, 'tx_vlan': 65535, '__T': 1766637853898327, 'ip4_active': True, 'fixed_ip4_list': ['192.168.2.251'], 'gateway': '192.168.2.254', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257'} 2025-12-25 04:44:37,662 - DEBUG - VM uve shows ip address as ['192.168.2.251', '::'] 2025-12-25 04:44:37,662 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 2025-12-25 04:44:37,662 - INFO - Verifying through opserver in 10.0.0.27 2025-12-25 04:44:37,666 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257?flat 2025-12-25 04:44:37,706 - DEBUG - expected vm list 760304cd-7856-4dd0-9878-3de3b0a5b574 2025-12-25 04:44:37,706 - DEBUG - Extracted vm list ['760304cd-7856-4dd0-9878-3de3b0a5b574'] 2025-12-25 04:44:37,706 - INFO - VM 760304cd-7856-4dd0-9878-3de3b0a5b574 is present in default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 2025-12-25 04:44:37,707 - DEBUG - Verifying vm in vrouter uve 2025-12-25 04:44:37,707 - DEBUG - Getting info from collector 10.0.0.23.. 2025-12-25 04:44:37,707 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/760304cd-7856-4dd0-9878-3de3b0a5b574?flat 2025-12-25 04:44:37,723 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4927-2'] 2025-12-25 04:44:37,723 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-2:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:44:37,793 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-2?flat 2025-12-25 04:44:37,828 - DEBUG - VM 760304cd-7856-4dd0-9878-3de3b0a5b574 is present in vrouter an-jenkins-deploy-platform-ansible-os-4927-2 uve 2025-12-25 04:44:37,829 - DEBUG - Validated that VM ctest-vn2_vm2-33671587 is in Vrouter an-jenkins-deploy-platform-ansible-os-4927-2 UVE 2025-12-25 04:44:37,829 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-12-25 04:44:37,841 - DEBUG - Expected tap interface of VM uuid 760304cd-7856-4dd0-9878-3de3b0a5b574 is default-domain:ctest-TestBasicVMVN0-19229946:6851c8bd-0cc3-4c6e-8068-a68c22f2e37f 2025-12-25 04:44:37,841 - DEBUG - Expected VN of VM uuid 760304cd-7856-4dd0-9878-3de3b0a5b574 is default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 2025-12-25 04:44:37,841 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-2:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:44:37,899 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-2?flat 2025-12-25 04:44:37,932 - DEBUG - VM 760304cd-7856-4dd0-9878-3de3b0a5b574 is present in vrouter an-jenkins-deploy-platform-ansible-os-4927-2 uve 2025-12-25 04:44:37,932 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-19229946:6851c8bd-0cc3-4c6e-8068-a68c22f2e37f of vm 760304cd-7856-4dd0-9878-3de3b0a5b574 is present in vrouter an-jenkins-deploy-platform-ansible-os-4927-2 uve 2025-12-25 04:44:37,932 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 of vm 760304cd-7856-4dd0-9878-3de3b0a5b574 is present in vrouter an-jenkins-deploy-platform-ansible-os-4927-2 uve 2025-12-25 04:44:37,932 - DEBUG - Verifying the 760304cd-7856-4dd0-9878-3de3b0a5b574 virtual network link through opserver 10.0.0.27 2025-12-25 04:44:37,932 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-12-25 04:44:37,946 - DEBUG - VM link and name as {'name': '760304cd-7856-4dd0-9878-3de3b0a5b574', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/760304cd-7856-4dd0-9878-3de3b0a5b574?flat'} 2025-12-25 04:44:37,947 - DEBUG - Verifying the 760304cd-7856-4dd0-9878-3de3b0a5b574 virtual network link through opserver 10.0.0.33 2025-12-25 04:44:37,947 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-12-25 04:44:38,033 - DEBUG - VM link and name as {'name': '760304cd-7856-4dd0-9878-3de3b0a5b574', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-machine/760304cd-7856-4dd0-9878-3de3b0a5b574?flat'} 2025-12-25 04:44:38,033 - DEBUG - Verifying the 760304cd-7856-4dd0-9878-3de3b0a5b574 virtual network link through opserver 10.0.0.23 2025-12-25 04:44:38,033 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-12-25 04:44:38,048 - DEBUG - VM link and name as {'name': '760304cd-7856-4dd0-9878-3de3b0a5b574', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/760304cd-7856-4dd0-9878-3de3b0a5b574?flat'} 2025-12-25 04:44:38,048 - INFO - VM ctest-vn2_vm2-33671587 validations in Opserver passed 2025-12-25 04:44:38,048 - INFO - Waiting for VM ctest-vn1_vm2-40724543 to be up.. 2025-12-25 04:44:38,048 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:44:38,048 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/3f47f918-a437-4e91-b959-7581c2cbd8e4 2025-12-25 04:44:38,057 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/4902e508-5e53-427d-abd6-5decadce6098 2025-12-25 04:44:39,228 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=3.17 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.667 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.667/1.920/3.174/1.253 ms') 2025-12-25 04:44:39,228 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-40724543 passed 2025-12-25 04:44:39,310 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:44:39,310 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-40724543, IP 192.168.1.251, Port 22 2025-12-25 04:44:39,484 - DEBUG - VM ctest-vn1_vm2-40724543 is ready for SSH connections 2025-12-25 04:44:39,484 - INFO - Waiting for VM ctest-vn2_vm2-33671587 to be up.. 2025-12-25 04:44:39,484 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:44:39,484 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/760304cd-7856-4dd0-9878-3de3b0a5b574 2025-12-25 04:44:39,494 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/6851c8bd-0cc3-4c6e-8068-a68c22f2e37f 2025-12-25 04:44:40,669 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=3.32 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.960 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 0.960/2.142/3.324/1.182 ms') 2025-12-25 04:44:40,669 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-33671587 passed 2025-12-25 04:44:40,744 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:44:40,744 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-33671587, IP 192.168.2.251, Port 22 2025-12-25 04:44:40,811 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-25 04:44:40,916 - DEBUG - VM ctest-vn2_vm2-33671587 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-25 04:44:45,917 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:44:45,917 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-33671587, IP 192.168.2.251, Port 22 2025-12-25 04:44:45,985 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-25 04:44:46,087 - DEBUG - VM ctest-vn2_vm2-33671587 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-25 04:44:51,088 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:44:51,088 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-33671587, IP 192.168.2.251, Port 22 2025-12-25 04:44:51,159 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-25 04:44:51,270 - DEBUG - VM ctest-vn2_vm2-33671587 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-25 04:44:56,271 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:44:56,271 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-33671587, IP 192.168.2.251, Port 22 2025-12-25 04:44:56,337 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-25 04:44:56,434 - DEBUG - VM ctest-vn2_vm2-33671587 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-25 04:45:01,436 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:45:01,436 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-33671587, IP 192.168.2.251, Port 22 2025-12-25 04:45:01,610 - DEBUG - VM ctest-vn2_vm2-33671587 is ready for SSH connections 2025-12-25 04:45:01,610 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-25 04:45:01,610 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.251, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.19, gateway password: c0ntrail123 2025-12-25 04:45:01,611 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.251 2025-12-25 04:45:45,810 - DEBUG - PING 192.168.2.251 (192.168.2.251) 56(84) bytes of data. 64 bytes from 192.168.2.251: icmp_req=1 ttl=63 time=11.3 ms 64 bytes from 192.168.2.251: icmp_req=2 ttl=63 time=1.42 ms 64 bytes from 192.168.2.251: icmp_req=3 ttl=63 time=1.27 ms --- 192.168.2.251 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2005ms rtt min/avg/max/mdev = 1.273/4.689/11.371/4.725 ms 2025-12-25 04:45:45,810 - INFO - Ping to IP 192.168.2.251 from VM ctest-vn1_vm2-40724543 passed 2025-12-25 04:45:45,810 - INFO - Deleting VM ctest-vn2_vm2-33671587 2025-12-25 04:45:46,048 - DEBUG - Verifying in api server 10.0.0.23 2025-12-25 04:45:46,048 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/760304cd-7856-4dd0-9878-3de3b0a5b574 2025-12-25 04:45:46,057 - DEBUG - VM ID 760304cd-7856-4dd0-9878-3de3b0a5b574 of VM ctest-vn2_vm2-33671587 is still found in API Server 2025-12-25 04:45:48,058 - DEBUG - Verifying in api server 10.0.0.23 2025-12-25 04:45:48,059 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/760304cd-7856-4dd0-9878-3de3b0a5b574 2025-12-25 04:45:48,067 - DEBUG - Response Code: 404 2025-12-25 04:45:48,068 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/760304cd-7856-4dd0-9878-3de3b0a5b574 2025-12-25 04:45:48,076 - DEBUG - Response Code: 404 2025-12-25 04:45:48,076 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/760304cd-7856-4dd0-9878-3de3b0a5b574 2025-12-25 04:45:48,083 - DEBUG - Response Code: 404 2025-12-25 04:45:48,083 - INFO - VM ctest-vn2_vm2-33671587 is fully removed in API-Server 2025-12-25 04:45:48,250 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-12-25 04:45:48,295 - DEBUG - Validated that vrouter 10.0.0.19 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 2025-12-25 04:45:48,447 - DEBUG - VM route 192.168.2.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 of compute 10.0.0.130 2025-12-25 04:45:48,457 - DEBUG - Validated that vrouter 10.0.0.130 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 2025-12-25 04:45:48,457 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 2025-12-25 04:45:48,458 - INFO - VM ctest-vn2_vm2-33671587 is removed in Compute, and routes are removed in all compute nodes 2025-12-25 04:45:48,710 - INFO - Routes for VM ctest-vn2_vm2-33671587 is removed in all control-nodes 2025-12-25 04:45:48,711 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-2:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:45:48,764 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-2?flat 2025-12-25 04:45:48,807 - DEBUG - VM 760304cd-7856-4dd0-9878-3de3b0a5b574 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4927-2 uve 2025-12-25 04:45:48,808 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:45:48,808 - DEBUG - interface for vm 760304cd-7856-4dd0-9878-3de3b0a5b574 deleted from vrouter uve ... 2025-12-25 04:45:48,808 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:45:51,809 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-2:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:45:51,874 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-2?flat 2025-12-25 04:45:51,908 - DEBUG - VM 760304cd-7856-4dd0-9878-3de3b0a5b574 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4927-2 uve 2025-12-25 04:45:51,909 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:45:51,909 - DEBUG - interface for vm 760304cd-7856-4dd0-9878-3de3b0a5b574 deleted from vrouter uve ... 2025-12-25 04:45:51,909 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:45:54,909 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-2:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:45:54,969 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-2?flat 2025-12-25 04:45:55,006 - DEBUG - VM 760304cd-7856-4dd0-9878-3de3b0a5b574 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4927-2 uve 2025-12-25 04:45:55,006 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:45:55,006 - DEBUG - interface for vm 760304cd-7856-4dd0-9878-3de3b0a5b574 deleted from vrouter uve ... 2025-12-25 04:45:55,006 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:45:58,007 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-2:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:45:58,069 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-2?flat 2025-12-25 04:45:58,100 - DEBUG - VM 760304cd-7856-4dd0-9878-3de3b0a5b574 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4927-2 uve 2025-12-25 04:45:58,100 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:45:58,101 - DEBUG - interface for vm 760304cd-7856-4dd0-9878-3de3b0a5b574 deleted from vrouter uve ... 2025-12-25 04:45:58,101 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:46:01,101 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-2:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:46:01,173 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-2?flat 2025-12-25 04:46:01,297 - DEBUG - VM 760304cd-7856-4dd0-9878-3de3b0a5b574 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4927-2 uve 2025-12-25 04:46:01,297 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:46:01,297 - DEBUG - interface for vm 760304cd-7856-4dd0-9878-3de3b0a5b574 deleted from vrouter uve ... 2025-12-25 04:46:01,297 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:46:04,298 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-2:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:46:04,359 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-2?flat 2025-12-25 04:46:04,402 - DEBUG - VM 760304cd-7856-4dd0-9878-3de3b0a5b574 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4927-2 uve 2025-12-25 04:46:04,402 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:46:04,402 - DEBUG - interface for vm 760304cd-7856-4dd0-9878-3de3b0a5b574 deleted from vrouter uve ... 2025-12-25 04:46:04,402 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:46:07,403 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-2:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:46:07,459 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-2?flat 2025-12-25 04:46:07,493 - DEBUG - VM 760304cd-7856-4dd0-9878-3de3b0a5b574 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4927-2 uve 2025-12-25 04:46:07,493 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:46:07,493 - DEBUG - interface for vm 760304cd-7856-4dd0-9878-3de3b0a5b574 deleted from vrouter uve ... 2025-12-25 04:46:07,493 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:46:10,494 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-2:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:46:10,547 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-2?flat 2025-12-25 04:46:10,579 - DEBUG - VM 760304cd-7856-4dd0-9878-3de3b0a5b574 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4927-2 uve 2025-12-25 04:46:10,579 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:46:10,579 - DEBUG - interface for vm 760304cd-7856-4dd0-9878-3de3b0a5b574 deleted from vrouter uve ... 2025-12-25 04:46:10,579 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:46:13,580 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-2:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:46:13,634 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-2?flat 2025-12-25 04:46:13,669 - DEBUG - VM 760304cd-7856-4dd0-9878-3de3b0a5b574 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4927-2 uve 2025-12-25 04:46:13,669 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:46:13,669 - DEBUG - interface for vm 760304cd-7856-4dd0-9878-3de3b0a5b574 deleted from vrouter uve ... 2025-12-25 04:46:13,669 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:46:16,669 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-2:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:46:16,724 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-2?flat 2025-12-25 04:46:16,758 - DEBUG - VM 760304cd-7856-4dd0-9878-3de3b0a5b574 is not present in vrouter an-jenkins-deploy-platform-ansible-os-4927-2 uve 2025-12-25 04:46:16,758 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:46:16,758 - DEBUG - interface for vm 760304cd-7856-4dd0-9878-3de3b0a5b574 deleted from vrouter uve ... 2025-12-25 04:46:16,758 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:46:16,758 - DEBUG - Validated that VM 760304cd-7856-4dd0-9878-3de3b0a5b574 is removed in Vrouter UVE 2025-12-25 04:46:16,758 - DEBUG - Verifying through opserver in 10.0.0.27 2025-12-25 04:46:16,758 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257?flat 2025-12-25 04:46:16,786 - DEBUG - 760304cd-7856-4dd0-9878-3de3b0a5b574 BM not in default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn2-64314257 uve 2025-12-25 04:46:16,786 - DEBUG - Validated that VM 760304cd-7856-4dd0-9878-3de3b0a5b574 is not present in the VN UVE 2025-12-25 04:46:16,786 - DEBUG - Verifying the 760304cd-7856-4dd0-9878-3de3b0a5b574 virtual network link through opserver 10.0.0.27 2025-12-25 04:46:16,786 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-12-25 04:46:16,801 - INFO - 760304cd-7856-4dd0-9878-3de3b0a5b574 vm uve deleted from opserver 2025-12-25 04:46:16,801 - DEBUG - Verifying the 760304cd-7856-4dd0-9878-3de3b0a5b574 virtual network link through opserver 10.0.0.33 2025-12-25 04:46:16,801 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-12-25 04:46:16,816 - INFO - 760304cd-7856-4dd0-9878-3de3b0a5b574 vm uve deleted from opserver 2025-12-25 04:46:16,816 - DEBUG - Verifying the 760304cd-7856-4dd0-9878-3de3b0a5b574 virtual network link through opserver 10.0.0.23 2025-12-25 04:46:16,816 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-12-25 04:46:16,832 - INFO - 760304cd-7856-4dd0-9878-3de3b0a5b574 vm uve deleted from opserver 2025-12-25 04:46:16,832 - INFO - Validated that VM 760304cd-7856-4dd0-9878-3de3b0a5b574 is removed from Opserver 2025-12-25 04:46:16,832 - INFO - Deleting VM ctest-vn1_vm2-40724543 2025-12-25 04:46:16,929 - DEBUG - Verifying in api server 10.0.0.23 2025-12-25 04:46:16,929 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/3f47f918-a437-4e91-b959-7581c2cbd8e4 2025-12-25 04:46:16,940 - DEBUG - VM ID 3f47f918-a437-4e91-b959-7581c2cbd8e4 of VM ctest-vn1_vm2-40724543 is still found in API Server 2025-12-25 04:46:18,941 - DEBUG - Verifying in api server 10.0.0.23 2025-12-25 04:46:18,942 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/3f47f918-a437-4e91-b959-7581c2cbd8e4 2025-12-25 04:46:18,951 - DEBUG - Response Code: 404 2025-12-25 04:46:18,952 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/3f47f918-a437-4e91-b959-7581c2cbd8e4 2025-12-25 04:46:18,963 - DEBUG - Response Code: 404 2025-12-25 04:46:18,964 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/3f47f918-a437-4e91-b959-7581c2cbd8e4 2025-12-25 04:46:18,972 - DEBUG - Response Code: 404 2025-12-25 04:46:18,972 - INFO - VM ctest-vn1_vm2-40724543 is fully removed in API-Server 2025-12-25 04:46:19,134 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-12-25 04:46:19,349 - DEBUG - VM route 192.168.1.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 of compute 10.0.0.19 2025-12-25 04:46:19,356 - DEBUG - Validated that vrouter 10.0.0.19 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 2025-12-25 04:46:19,356 - DEBUG - Validated that vrouter 10.0.0.130 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 2025-12-25 04:46:19,357 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 2025-12-25 04:46:19,357 - INFO - VM ctest-vn1_vm2-40724543 is removed in Compute, and routes are removed in all compute nodes 2025-12-25 04:46:19,602 - INFO - Routes for VM ctest-vn1_vm2-40724543 is removed in all control-nodes 2025-12-25 04:46:19,602 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-1:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:46:19,662 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-1?flat 2025-12-25 04:46:19,701 - DEBUG - VM 3f47f918-a437-4e91-b959-7581c2cbd8e4 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4927-1 uve 2025-12-25 04:46:19,701 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:46:19,702 - DEBUG - interface for vm 3f47f918-a437-4e91-b959-7581c2cbd8e4 deleted from vrouter uve ... 2025-12-25 04:46:19,702 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:46:22,702 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-1:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:46:22,758 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-1?flat 2025-12-25 04:46:22,791 - DEBUG - VM 3f47f918-a437-4e91-b959-7581c2cbd8e4 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4927-1 uve 2025-12-25 04:46:22,791 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:46:22,792 - DEBUG - interface for vm 3f47f918-a437-4e91-b959-7581c2cbd8e4 deleted from vrouter uve ... 2025-12-25 04:46:22,792 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:46:25,792 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-1:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:46:25,855 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-1?flat 2025-12-25 04:46:25,890 - DEBUG - VM 3f47f918-a437-4e91-b959-7581c2cbd8e4 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4927-1 uve 2025-12-25 04:46:25,890 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:46:25,890 - DEBUG - interface for vm 3f47f918-a437-4e91-b959-7581c2cbd8e4 deleted from vrouter uve ... 2025-12-25 04:46:25,890 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:46:28,891 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-1:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:46:28,944 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-1?flat 2025-12-25 04:46:28,982 - DEBUG - VM 3f47f918-a437-4e91-b959-7581c2cbd8e4 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4927-1 uve 2025-12-25 04:46:28,982 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:46:28,982 - DEBUG - interface for vm 3f47f918-a437-4e91-b959-7581c2cbd8e4 deleted from vrouter uve ... 2025-12-25 04:46:28,982 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:46:31,983 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-1:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:46:32,040 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-1?flat 2025-12-25 04:46:32,079 - DEBUG - VM 3f47f918-a437-4e91-b959-7581c2cbd8e4 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4927-1 uve 2025-12-25 04:46:32,079 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:46:32,079 - DEBUG - interface for vm 3f47f918-a437-4e91-b959-7581c2cbd8e4 deleted from vrouter uve ... 2025-12-25 04:46:32,079 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:46:35,079 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-1:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:46:35,136 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-1?flat 2025-12-25 04:46:35,168 - DEBUG - VM 3f47f918-a437-4e91-b959-7581c2cbd8e4 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4927-1 uve 2025-12-25 04:46:35,168 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:46:35,168 - DEBUG - interface for vm 3f47f918-a437-4e91-b959-7581c2cbd8e4 deleted from vrouter uve ... 2025-12-25 04:46:35,168 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:46:38,169 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4927-1:Compute:contrail-vrouter-agent:0?flat 2025-12-25 04:46:38,226 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4927-1?flat 2025-12-25 04:46:38,260 - DEBUG - VM 3f47f918-a437-4e91-b959-7581c2cbd8e4 is not present in vrouter an-jenkins-deploy-platform-ansible-os-4927-1 uve 2025-12-25 04:46:38,260 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-25 04:46:38,260 - DEBUG - interface for vm 3f47f918-a437-4e91-b959-7581c2cbd8e4 deleted from vrouter uve ... 2025-12-25 04:46:38,260 - DEBUG - Deleted interface not in error interface list ... 2025-12-25 04:46:38,260 - DEBUG - Validated that VM 3f47f918-a437-4e91-b959-7581c2cbd8e4 is removed in Vrouter UVE 2025-12-25 04:46:38,260 - DEBUG - Verifying through opserver in 10.0.0.27 2025-12-25 04:46:38,261 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474?flat 2025-12-25 04:46:38,286 - DEBUG - 3f47f918-a437-4e91-b959-7581c2cbd8e4 BM not in default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 uve 2025-12-25 04:46:38,286 - DEBUG - Validated that VM 3f47f918-a437-4e91-b959-7581c2cbd8e4 is not present in the VN UVE 2025-12-25 04:46:38,286 - DEBUG - Verifying the 3f47f918-a437-4e91-b959-7581c2cbd8e4 virtual network link through opserver 10.0.0.27 2025-12-25 04:46:38,286 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-12-25 04:46:38,299 - INFO - 3f47f918-a437-4e91-b959-7581c2cbd8e4 vm uve deleted from opserver 2025-12-25 04:46:38,299 - DEBUG - Verifying the 3f47f918-a437-4e91-b959-7581c2cbd8e4 virtual network link through opserver 10.0.0.33 2025-12-25 04:46:38,300 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-12-25 04:46:38,315 - INFO - 3f47f918-a437-4e91-b959-7581c2cbd8e4 vm uve deleted from opserver 2025-12-25 04:46:38,315 - DEBUG - Verifying the 3f47f918-a437-4e91-b959-7581c2cbd8e4 virtual network link through opserver 10.0.0.23 2025-12-25 04:46:38,315 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-12-25 04:46:38,330 - INFO - 3f47f918-a437-4e91-b959-7581c2cbd8e4 vm uve deleted from opserver 2025-12-25 04:46:38,330 - INFO - Validated that VM 3f47f918-a437-4e91-b959-7581c2cbd8e4 is removed from Opserver 2025-12-25 04:46:38,330 - INFO - Deleting VN ctest-vn3-14620459 2025-12-25 04:46:38,472 - INFO - Deleting VM ctest-vn2_vm1-00543289 2025-12-25 04:46:38,593 - INFO - Deleting VM ctest-vn1_vm1-31633523 2025-12-25 04:46:38,874 - INFO - Deleting VN ctest-vn2-64314257 2025-12-25 04:46:38,936 - DEBUG - RefsExistError Delete when resource still referred: ['http://10.0.0.23:8082/virtual-machine-interface/4410cce1-743a-4da7-a2ba-fae322db9f39'] while deleting VN ctest-vn2-64314257..Will retry 2025-12-25 04:46:41,077 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/351ce0c9-7a04-4a7e-ac1b-b8600b343121 2025-12-25 04:46:41,086 - DEBUG - Response Code: 404 2025-12-25 04:46:41,087 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-12-25 04:46:41,094 - DEBUG - Requesting: http://10.0.0.27:8082/domain/d4f5d7ea-1779-482f-9e93-03873d431e99 2025-12-25 04:46:41,113 - DEBUG - Requesting: http://10.0.0.27:8082/project/edba148a-64cc-4e4a-8216-4c7dbc6a7d87 2025-12-25 04:46:41,216 - INFO - Validated that VN ctest-vn2-64314257 is not found in API Server 2025-12-25 04:46:41,233 - DEBUG - VN ctest-vn2-64314257 is not present in Agent 10.0.0.19 2025-12-25 04:46:41,244 - DEBUG - VN ctest-vn2-64314257 is not present in Agent 10.0.0.130 2025-12-25 04:46:41,244 - INFO - Validated that VN ctest-vn2-64314257 is not in any agent 2025-12-25 04:46:41,286 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn2-64314257 info 2025-12-25 04:46:41,464 - INFO - Deleting VN ctest-vn1-91861474 2025-12-25 04:46:41,606 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/226e8db5-3080-4ae5-944b-3627122b62a7 2025-12-25 04:46:41,615 - DEBUG - Response Code: 404 2025-12-25 04:46:41,615 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-12-25 04:46:41,629 - DEBUG - Requesting: http://10.0.0.27:8082/domain/d4f5d7ea-1779-482f-9e93-03873d431e99 2025-12-25 04:46:41,657 - DEBUG - Requesting: http://10.0.0.27:8082/project/edba148a-64cc-4e4a-8216-4c7dbc6a7d87 2025-12-25 04:46:41,772 - INFO - Validated that VN ctest-vn1-91861474 is not found in API Server 2025-12-25 04:46:41,786 - DEBUG - VN ctest-vn1-91861474 is not present in Agent 10.0.0.19 2025-12-25 04:46:41,798 - DEBUG - VN ctest-vn1-91861474 is not present in Agent 10.0.0.130 2025-12-25 04:46:41,798 - INFO - Validated that VN ctest-vn1-91861474 is not in any agent 2025-12-25 04:46:41,811 - DEBUG - VRF 2 is not seen in agent 10.0.0.19 2025-12-25 04:46:41,829 - DEBUG - Vrouter 10.0.0.19 does not have vrf 2 for VN ctest-vn1-91861474 2025-12-25 04:46:41,829 - INFO - Validated that all vrouters do not have the route table for VN default-domain:ctest-TestBasicVMVN0-19229946:ctest-vn1-91861474 2025-12-25 04:46:41,855 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-91861474 info 2025-12-25 04:46:41,904 - INFO - Deleted policy policy2 2025-12-25 04:46:41,952 - INFO - Deleted policy policy1 2025-12-25 04:46:43,243 - DEBUG - Skipping xmpp flap check

2025-12-25 04:46:43,243 - INFO - END TEST : test_process_restart_in_policy_between_vns : PASSED[0:06:53]

2025-12-25 04:46:43,244 - INFO - -------------------------------------------------------------------------------- 2025-12-25 04:46:43,251 - INFO - ================================================================================ 2025-12-25 04:46:43,251 - INFO - STARTING TEST : test_underlay_broadcast_traffic_handling 2025-12-25 04:46:43,251 - INFO - TEST DESCRIPTION : Test the underlay brocast traffic handling by vrouter. (Bug-1545229). 1. Send broadcast traffic from one compute node. 2. Other compute in same subnet should receive that traffic. 3. Receiving compute should treat this traffic as underlay. 4. Compute should not replicate the packet and send the copy back. Pass criteria: Step 3-4 should pass Maintainer : chhandak@juniper.net 2025-12-25 04:46:44,553 - DEBUG - Skipping xmpp flap check 2025-12-25 04:46:44,553 - INFO - Initial checks done. Running the testcase now 2025-12-25 04:46:44,554 - INFO - 2025-12-25 04:46:44,943 - INFO - Agent interface name: ens6 2025-12-25 04:46:45,068 - DEBUG - Executing command: sudo tcpdump -nni ens6 -U host 10.20.0.255 -w /tmp/ens6_ctest-random-03301867.pcap 2025-12-25 04:46:50,487 - INFO - Pinging broacast address 10.20.0.255 from compute 10.20.0.252 2025-12-25 04:47:04,753 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-12-25 04:47:06,893 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-03301867.pcap | grep -c length 2025-12-25 04:47:06,909 - DEBUG - STDOUT: 10 2025-12-25 04:47:06,910 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-03301867.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-12-25 04:47:06,910 - INFO - 10 packets are found in tcpdump output as expected 2025-12-25 04:47:06,910 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-12-25 04:47:08,958 - INFO - Packet count matched: Compute 10.0.0.130 has receive only 10 packet from source IP 10.20.0.255. No duplicate packet seen 2025-12-25 04:47:10,271 - DEBUG - Skipping xmpp flap check

2025-12-25 04:47:10,271 - INFO - END TEST : test_underlay_broadcast_traffic_handling : PASSED[0:00:27]

2025-12-25 04:47:10,271 - INFO - -------------------------------------------------------------------------------- 2025-12-25 04:47:11,192 - INFO - Deleted project: ctest-TestBasicVMVN0-19229946, ID : edba148a-64cc-4e4a-8216-4c7dbc6a7d87