2025-10-08 15:44:08,170 - INFO - Domain Default found not creating 2025-10-08 15:44:08,357 - INFO - Project ctest-TestBasicVMVN0-24481103 not found, creating it 2025-10-08 15:44:08,983 - INFO - Created Project:ctest-TestBasicVMVN0-24481103, ID : 2d0f9370-44fd-4f7b-99c3-73540ae4a690 2025-10-08 15:44:10,651 - INFO - ================================================================================ 2025-10-08 15:44:10,651 - INFO - STARTING TEST : test_control_node_switchover 2025-10-08 15:44:10,651 - INFO - TEST DESCRIPTION : Stop the control node and check peering with agent fallback to other control node. 1. Pick one VN from respource pool which has 2 VM's in it 2. Verify ping between VM's 3. Find active control node in cluster by agent inspect 4. Stop control service on active control node 5. Verify agents are connected to new active control-node using xmpp connections 6. Bring back control service on previous active node 7. Verify ping between VM's again after bringing up control serveice Pass criteria: Step 2,5 and 7 should pass 2025-10-08 15:44:11,936 - DEBUG - Skipping xmpp flap check 2025-10-08 15:44:11,937 - INFO - Initial checks done. Running the testcase now 2025-10-08 15:44:11,937 - INFO - 2025-10-08 15:44:13,080 - DEBUG - Response for create_network : {'network': {'id': 'acbfb9d7-3dae-4d32-b561-9b923c00fa90', 'name': 'ctest-vn1-59329388', 'tenant_id': '2d0f937044fd4f7b99c373540ae4a690', 'project_id': '2d0f937044fd4f7b99c373540ae4a690', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestBasicVMVN0-24481103', 'ctest-vn1-59329388'], 'port_security_enabled': True, 'description': ''}} 2025-10-08 15:44:13,381 - DEBUG - Response for create_subnet : {'subnet': {'id': '5a12c91e-2b23-4916-9597-69d6ff4fef48', 'name': '', 'tenant_id': '2d0f937044fd4f7b99c373540ae4a690', 'network_id': 'acbfb9d7-3dae-4d32-b561-9b923c00fa90', 'ip_version': 4, 'cidr': '192.168.1.0/24', 'allocation_pools': [{'start': '192.168.1.2', 'end': '192.168.1.254'}], 'gateway_ip': '192.168.1.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '192.168.1.2', 'tags': [], 'project_id': '2d0f937044fd4f7b99c373540ae4a690'}} 2025-10-08 15:44:13,443 - INFO - Created VN ctest-vn1-59329388 2025-10-08 15:44:13,463 - DEBUG - VN ctest-vn1-59329388 UUID is acbfb9d7-3dae-4d32-b561-9b923c00fa90 2025-10-08 15:44:13,562 - DEBUG - Requesting: http://10.0.0.133:8082/domains 2025-10-08 15:44:13,913 - DEBUG - Requesting: http://10.0.0.133:8082/domains 2025-10-08 15:44:13,948 - DEBUG - Requesting: http://10.0.0.133:8082/domain/adc52b80-2ed1-485c-90d4-83e78c8e1400 2025-10-08 15:44:13,967 - DEBUG - Requesting: http://10.0.0.133:8082/project/2d0f9370-44fd-4f7b-99c3-73540ae4a690 2025-10-08 15:44:14,081 - DEBUG - Requesting: http://10.0.0.133:8082/virtual-network/acbfb9d7-3dae-4d32-b561-9b923c00fa90 2025-10-08 15:44:14,097 - DEBUG - Requesting: http://10.0.0.133:8082/virtual-network/acbfb9d7-3dae-4d32-b561-9b923c00fa90 2025-10-08 15:44:14,112 - DEBUG - Requesting: http://10.0.0.133:8082/routing-instance/81b0778f-d29f-437a-90ec-6635035b81a3 2025-10-08 15:44:14,126 - DEBUG - Requesting: http://10.0.0.133:8082/routing-instance/81b0778f-d29f-437a-90ec-6635035b81a3 2025-10-08 15:44:14,138 - DEBUG - Requesting: http://10.0.0.133:8082/route-target/6f1312ce-e477-4a10-aecc-ed7220f36eb1 2025-10-08 15:44:14,147 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-08 15:44:14,147 - DEBUG - Requesting: http://10.0.0.133:8082/virtual-network/acbfb9d7-3dae-4d32-b561-9b923c00fa90 2025-10-08 15:44:14,160 - DEBUG - Requesting: http://10.0.0.133:8082/routing-instance/81b0778f-d29f-437a-90ec-6635035b81a3 2025-10-08 15:44:14,180 - INFO - Verified VN network id 14 for VN acbfb9d7-3dae-4d32-b561-9b923c00fa90 2025-10-08 15:44:14,180 - INFO - Verifications in API Server for VN ctest-vn1-59329388 passed 2025-10-08 15:44:14,180 - DEBUG - Requesting: http://10.0.0.133:8082/virtual-network/acbfb9d7-3dae-4d32-b561-9b923c00fa90 2025-10-08 15:44:14,192 - DEBUG - Requesting: http://10.0.0.133:8082/routing-instance/81b0778f-d29f-437a-90ec-6635035b81a3 2025-10-08 15:44:14,202 - DEBUG - Requesting: http://10.0.0.133:8082/routing-instance/81b0778f-d29f-437a-90ec-6635035b81a3 2025-10-08 15:44:14,212 - DEBUG - Requesting: http://10.0.0.133:8082/route-target/6f1312ce-e477-4a10-aecc-ed7220f36eb1 2025-10-08 15:44:14,230 - DEBUG - Control-node 10.0.0.133 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-24481103', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '12447872229179280690', 'uuid-lslong': '13069898645980117648'}, 'enable': 'true', 'created': '2025-10-08T15:44:13', 'last-modified': '2025-10-08T15:44:13', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.932904'} 2025-10-08 15:44:14,237 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-08 15:44:14,244 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-24481103', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '12447872229179280690', 'uuid-lslong': '13069898645980117648'}, 'enable': 'true', 'created': '2025-10-08T15:44:13', 'last-modified': '2025-10-08T15:44:13', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.953644'} 2025-10-08 15:44:14,252 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-08 15:44:14,258 - DEBUG - Control-node 10.0.0.242 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-24481103', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '12447872229179280690', 'uuid-lslong': '13069898645980117648'}, 'enable': 'true', 'created': '2025-10-08T15:44:13', 'last-modified': '2025-10-08T15:44:13', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.968140'} 2025-10-08 15:44:14,265 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-08 15:44:14,265 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-59329388 passed 2025-10-08 15:44:14,266 - DEBUG - ====Verifying policy data for ctest-vn1-59329388 in API_Server ====== 2025-10-08 15:44:14,266 - DEBUG - Requesting: http://10.0.0.133:8082/domains 2025-10-08 15:44:14,276 - DEBUG - Requesting: http://10.0.0.133:8082/domain/adc52b80-2ed1-485c-90d4-83e78c8e1400 2025-10-08 15:44:14,290 - DEBUG - Requesting: http://10.0.0.133:8082/project/2d0f9370-44fd-4f7b-99c3-73540ae4a690 2025-10-08 15:44:14,398 - DEBUG - Requesting: http://10.0.0.133:8082/virtual-network/acbfb9d7-3dae-4d32-b561-9b923c00fa90 2025-10-08 15:44:14,409 - DEBUG - =>VN ctest-vn1-59329388 has no policy to be verified 2025-10-08 15:44:14,410 - DEBUG - Verifying the vn in opserver 2025-10-08 15:44:14,410 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 virtual network link through opserver 10.0.0.133 2025-10-08 15:44:14,410 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-networks 2025-10-08 15:44:14,714 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-networks 2025-10-08 15:44:14,791 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388', 'href': 'http://10.0.0.133:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388?flat'} 2025-10-08 15:44:14,791 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 virtual network link through opserver 10.0.0.50 2025-10-08 15:44:14,792 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2025-10-08 15:44:15,121 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2025-10-08 15:44:15,199 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388?flat'} 2025-10-08 15:44:15,199 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 virtual network link through opserver 10.0.0.242 2025-10-08 15:44:15,200 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-networks 2025-10-08 15:44:15,513 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-networks 2025-10-08 15:44:15,586 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388', 'href': 'http://10.0.0.242:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388?flat'} 2025-10-08 15:44:15,586 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 is found in opserver 2025-10-08 15:44:15,610 - DEBUG - Do not have enough data to verify VN in agent 2025-10-08 15:44:15,623 - DEBUG - VRF ids for VN ctest-vn1-59329388: {} 2025-10-08 15:44:15,777 - DEBUG - Services list from nova: [, , , ] 2025-10-08 15:44:17,009 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4236-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4236-2) 2025-10-08 15:44:18,777 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4236-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4236-1) 2025-10-08 15:44:18,777 - INFO - Waiting for VM ctest-vn1_vm1-67939375 to be up.. 2025-10-08 15:44:18,882 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-10-08 15:44:23,978 - DEBUG - VM is in ACTIVE state now 2025-10-08 15:44:23,978 - INFO - VM name : ctest-vn1_vm1-67939375 2025-10-08 15:44:24,081 - DEBUG - VM ctest-vn1_vm1-67939375 ID is 0fac7c9b-dfd7-4659-aeb3-169db789ca1b 2025-10-08 15:44:24,113 - DEBUG - VM ctest-vn1_vm1-67939375 launched on Node an-jenkins-deploy-platform-ansible-os-4236-2 2025-10-08 15:44:24,209 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/0fac7c9b-dfd7-4659-aeb3-169db789ca1b 2025-10-08 15:44:24,530 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/0fac7c9b-dfd7-4659-aeb3-169db789ca1b 2025-10-08 15:44:24,576 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine-interface/a82a845f-e516-4530-9e58-8aad0cff8e7e 2025-10-08 15:44:27,783 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1032ms') 2025-10-08 15:44:27,783 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67939375 failed! 2025-10-08 15:44:27,802 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 is 192.168.1.1 and allocation pool is NOT set 2025-10-08 15:44:31,885 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-10-08 15:44:31,885 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67939375 failed! 2025-10-08 15:44:31,904 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 is 192.168.1.1 and allocation pool is NOT set 2025-10-08 15:44:35,974 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2025-10-08 15:44:35,974 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67939375 failed! 2025-10-08 15:44:35,991 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 is 192.168.1.1 and allocation pool is NOT set 2025-10-08 15:44:40,074 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-10-08 15:44:40,075 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67939375 failed! 2025-10-08 15:44:40,091 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 is 192.168.1.1 and allocation pool is NOT set 2025-10-08 15:44:44,166 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2025-10-08 15:44:44,166 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67939375 failed! 2025-10-08 15:44:44,180 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 is 192.168.1.1 and allocation pool is NOT set 2025-10-08 15:44:48,266 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-10-08 15:44:48,266 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67939375 failed! 2025-10-08 15:44:48,293 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 is 192.168.1.1 and allocation pool is NOT set 2025-10-08 15:44:52,358 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1003ms') 2025-10-08 15:44:52,358 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67939375 failed! 2025-10-08 15:44:52,374 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 is 192.168.1.1 and allocation pool is NOT set 2025-10-08 15:44:56,451 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-10-08 15:44:56,451 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67939375 failed! 2025-10-08 15:44:56,471 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 is 192.168.1.1 and allocation pool is NOT set 2025-10-08 15:44:58,543 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=13.3 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=3.16 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 3.155/8.249/13.343/5.094 ms') 2025-10-08 15:44:58,543 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-67939375 passed 2025-10-08 15:44:58,623 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:44:58,624 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-67939375, IP 192.168.1.3, Port 22 2025-10-08 15:44:58,691 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-08 15:44:58,818 - DEBUG - VM ctest-vn1_vm1-67939375 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-08 15:45:03,819 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:45:03,819 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-67939375, IP 192.168.1.3, Port 22 2025-10-08 15:45:03,876 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-08 15:45:03,973 - DEBUG - VM ctest-vn1_vm1-67939375 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-08 15:45:08,974 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:45:08,975 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-67939375, IP 192.168.1.3, Port 22 2025-10-08 15:45:09,043 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-08 15:45:09,159 - DEBUG - VM ctest-vn1_vm1-67939375 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-08 15:45:14,160 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:45:14,160 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-67939375, IP 192.168.1.3, Port 22 2025-10-08 15:45:14,227 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-08 15:45:14,319 - DEBUG - VM ctest-vn1_vm1-67939375 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-08 15:45:19,320 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:45:19,321 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-67939375, IP 192.168.1.3, Port 22 2025-10-08 15:45:19,487 - DEBUG - VM ctest-vn1_vm1-67939375 is ready for SSH connections 2025-10-08 15:45:19,487 - INFO - Waiting for VM ctest-vn1_vm2-82798679 to be up.. 2025-10-08 15:45:19,612 - DEBUG - VM is in ACTIVE state now 2025-10-08 15:45:19,612 - INFO - VM name : ctest-vn1_vm2-82798679 2025-10-08 15:45:19,728 - DEBUG - VM ctest-vn1_vm2-82798679 ID is f87b4189-800c-46d0-9010-31791364729c 2025-10-08 15:45:19,728 - DEBUG - VM ctest-vn1_vm2-82798679 launched on Node an-jenkins-deploy-platform-ansible-os-4236-1 2025-10-08 15:45:19,841 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/f87b4189-800c-46d0-9010-31791364729c 2025-10-08 15:45:19,852 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine-interface/4f98a5d3-b393-4060-95c6-d836fc4371af 2025-10-08 15:45:21,034 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=3.72 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=1.42 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 1.423/2.573/3.723/1.150 ms') 2025-10-08 15:45:21,035 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm2-82798679 passed 2025-10-08 15:45:21,129 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:45:21,130 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-82798679, IP 192.168.1.4, Port 22 2025-10-08 15:45:21,287 - DEBUG - VM ctest-vn1_vm2-82798679 is ready for SSH connections 2025-10-08 15:45:21,287 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:45:21,288 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.241, gateway password: c0ntrail123 2025-10-08 15:45:21,288 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4 2025-10-08 15:45:43,110 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data. 64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=13.9 ms 64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.50 ms 64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.01 ms --- 192.168.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 1.016/5.486/13.935/5.977 ms 2025-10-08 15:45:43,110 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-67939375 passed 2025-10-08 15:45:43,111 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:45:43,111 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.61, gateway password: c0ntrail123 2025-10-08 15:45:43,111 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3 2025-10-08 15:46:21,837 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. 64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=7.90 ms 64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.18 ms 64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.30 ms --- 192.168.1.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.184/3.464/7.905/3.140 ms 2025-10-08 15:46:21,838 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-82798679 passed 2025-10-08 15:46:21,845 - INFO - Active control node from the Agent 10.0.0.241 is 10.0.0.242 2025-10-08 15:46:21,845 - INFO - Stoping the Control service in 10.0.0.242 2025-10-08 15:46:22,281 - INFO - 10.0.0.242 2025-10-08 15:46:22,282 - INFO - control:inactive 2025-10-08 15:46:22,282 - INFO - Contrail services control are down on nodes 10.0.0.242 2025-10-08 15:46:27,291 - INFO - Active control node from the Agent 10.0.0.241 is 10.0.0.133 2025-10-08 15:46:27,292 - INFO - Starting the Control service in 10.0.0.242 2025-10-08 15:46:27,670 - INFO - 10.0.0.242 2025-10-08 15:46:27,676 - INFO - control:initializing 2025-10-08 15:46:27,676 - DEBUG - defaultdict(, {'10.0.0.242': {'control': {'status': 'initializing', 'description': None}}}) 2025-10-08 15:46:27,676 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-08 15:46:32,775 - INFO - 10.0.0.242 2025-10-08 15:46:32,781 - INFO - control:active 2025-10-08 15:46:32,782 - INFO - Contrail services ['control'] are up on nodes 10.0.0.242 2025-10-08 15:46:37,903 - DEBUG - VM is in ACTIVE state now 2025-10-08 15:46:37,903 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:46:37,903 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/0fac7c9b-dfd7-4659-aeb3-169db789ca1b 2025-10-08 15:46:37,915 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine-interface/a82a845f-e516-4530-9e58-8aad0cff8e7e 2025-10-08 15:46:37,934 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/0fac7c9b-dfd7-4659-aeb3-169db789ca1b 2025-10-08 15:46:37,946 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine-interface/a82a845f-e516-4530-9e58-8aad0cff8e7e 2025-10-08 15:46:37,961 - DEBUG - Requesting: http://10.0.0.242:8082/instance-ip/68d2717a-b8e6-4c8a-9a7d-ebed2490484d 2025-10-08 15:46:37,973 - DEBUG - Verifying in api server 10.0.0.242 2025-10-08 15:46:37,974 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/0fac7c9b-dfd7-4659-aeb3-169db789ca1b 2025-10-08 15:46:37,982 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine-interface/a82a845f-e516-4530-9e58-8aad0cff8e7e 2025-10-08 15:46:37,993 - DEBUG - Requesting: http://10.0.0.242:8082/instance-ip/68d2717a-b8e6-4c8a-9a7d-ebed2490484d 2025-10-08 15:46:38,005 - INFO - VM ctest-vn1_vm1-67939375 verfication in all API Servers passed 2025-10-08 15:46:38,027 - DEBUG - VM ctest-vn1_vm1-67939375 Tap interface: {'index': '3', 'name': 'tapa82a845f-e5', 'uuid': 'a82a845f-e516-4530-9e58-8aad0cff8e7e', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388:ctest-vn1-59329388', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388', 'vm_uuid': '0fac7c9b-dfd7-4659-aeb3-169db789ca1b', 'vm_name': 'ctest-vn1_vm1-67939375', 'ip_addr': '192.168.1.3', 'mac_addr': '02:a8:2a:84:5f:e5', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '161', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-24481103:a82a845f-e516-4530-9e58-8aad0cff8e7e', 'sg_uuid_list': ['cbf6bc58-853e-4673-ab9d-9aaf1cab7a76'], 'static_route_list': None, 'vm_project_uuid': '2d0f9370-44fd-4f7b-99c3-73540ae4a690', 'admin_state': 'Enabled', 'flow_key_idx': '26', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-08 15:46:38,027 - DEBUG - Agent 10.0.0.241 vrf name: default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388:ctest-vn1-59329388 2025-10-08 15:46:38,033 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388:ctest-vn1-59329388', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388', 'table_label': '-1', 'vxlan_id': '14', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.242:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-10-08 15:46:38,114 - DEBUG - VM ctest-vn1_vm1-67939375 : Tap interface tapa82a845f-e5 is set to Active 2025-10-08 15:46:38,115 - DEBUG - Tap interface tapa82a845f-e5 detail : {'index': '3', 'name': 'tapa82a845f-e5', 'uuid': 'a82a845f-e516-4530-9e58-8aad0cff8e7e', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388:ctest-vn1-59329388', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388', 'vm_uuid': '0fac7c9b-dfd7-4659-aeb3-169db789ca1b', 'vm_name': 'ctest-vn1_vm1-67939375', 'ip_addr': '192.168.1.3', 'mac_addr': '02:a8:2a:84:5f:e5', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '161', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-24481103:a82a845f-e516-4530-9e58-8aad0cff8e7e', 'sg_uuid_list': ['cbf6bc58-853e-4673-ab9d-9aaf1cab7a76'], 'static_route_list': None, 'vm_project_uuid': '2d0f9370-44fd-4f7b-99c3-73540ae4a690', 'admin_state': 'Enabled', 'flow_key_idx': '26', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-08 15:46:38,229 - DEBUG - Starting Layer 2 verification in Agent 2025-10-08 15:46:38,242 - DEBUG - Layer 2 path is seen for VM MAC 02:a8:2a:84:5f:e5 in agent 10.0.0.241 2025-10-08 15:46:38,242 - DEBUG - Active layer 2 route in agent is present for VMI tapa82a845f-e5 2025-10-08 15:46:38,242 - DEBUG - L2 label(29) matches bw route table and interface table 2025-10-08 15:46:38,250 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 is found in Agent of node 10.0.0.61 2025-10-08 15:46:38,382 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 is consistent in agent 10.0.0.61 2025-10-08 15:46:38,382 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.61 2025-10-08 15:46:38,382 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 verification for VM ctest-vn1_vm1-67939375 in Agent 10.0.0.61 passed 2025-10-08 15:46:38,504 - DEBUG - Starting all layer 2 verification in agent 10.0.0.61 2025-10-08 15:46:38,516 - DEBUG - Route for VM MAC 02:a8:2a:84:5f:e5 is consistent in agent 10.0.0.61 2025-10-08 15:46:38,524 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 is found in Agent of node 10.0.0.241 2025-10-08 15:46:38,658 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 is consistent in agent 10.0.0.241 2025-10-08 15:46:38,658 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.241 2025-10-08 15:46:38,658 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 verification for VM ctest-vn1_vm1-67939375 in Agent 10.0.0.241 passed 2025-10-08 15:46:38,775 - DEBUG - Starting all layer 2 verification in agent 10.0.0.241 2025-10-08 15:46:38,792 - DEBUG - Route for VM MAC 02:a8:2a:84:5f:e5 is consistent in agent 10.0.0.241 2025-10-08 15:46:38,792 - INFO - VM ctest-vn1_vm1-67939375 verifications in Compute nodes passed 2025-10-08 15:46:39,043 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.61 2025-10-08 15:46:39,051 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.242, 25; Got 10.20.0.242,25 2025-10-08 15:46:39,051 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.61 2025-10-08 15:46:39,224 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.241 2025-10-08 15:46:39,231 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-10-08 15:46:39,232 - INFO - Validated routes of VM ctest-vn1_vm1-67939375 in all vrouters 2025-10-08 15:46:39,787 - DEBUG - Starting all layer2 verification in 10.0.0.133 Control Node 2025-10-08 15:46:39,931 - DEBUG - Layer2 route found for VM MAC 02:a8:2a:84:5f:e5 in Control-node 10.0.0.133 2025-10-08 15:46:39,931 - DEBUG - L2 Label for VM ctest-vn1_vm1-67939375 same between Control-node 10.0.0.133 and Agent, Expected: 29, Seen: 29 2025-10-08 15:46:39,931 - DEBUG - Starting all layer2 verification in 10.0.0.50 Control Node 2025-10-08 15:46:40,068 - DEBUG - Layer2 route found for VM MAC 02:a8:2a:84:5f:e5 in Control-node 10.0.0.50 2025-10-08 15:46:40,068 - DEBUG - L2 Label for VM ctest-vn1_vm1-67939375 same between Control-node 10.0.0.50 and Agent, Expected: 29, Seen: 29 2025-10-08 15:46:40,068 - DEBUG - Starting all layer2 verification in 10.0.0.242 Control Node 2025-10-08 15:46:40,205 - DEBUG - Layer2 route found for VM MAC 02:a8:2a:84:5f:e5 in Control-node 10.0.0.242 2025-10-08 15:46:40,206 - DEBUG - L2 Label for VM ctest-vn1_vm1-67939375 same between Control-node 10.0.0.242 and Agent, Expected: 29, Seen: 29 2025-10-08 15:46:40,206 - INFO - Verification in Control-nodes for VM ctest-vn1_vm1-67939375 passed 2025-10-08 15:46:40,206 - DEBUG - Verifying the vm in opserver 2025-10-08 15:46:40,206 - DEBUG - Verifying in collector 10.0.0.242 ... 2025-10-08 15:46:40,206 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-machine/0fac7c9b-dfd7-4659-aeb3-169db789ca1b?flat 2025-10-08 15:46:40,226 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-24481103:a82a845f-e516-4530-9e58-8aad0cff8e7e?flat 2025-10-08 15:46:40,252 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-24481103:a82a845f-e516-4530-9e58-8aad0cff8e7e?flat 2025-10-08 15:46:40,348 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-machine/0fac7c9b-dfd7-4659-aeb3-169db789ca1b?flat 2025-10-08 15:46:40,430 - DEBUG - Verifying vm in vn uve 2025-10-08 15:46:40,431 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-24481103:a82a845f-e516-4530-9e58-8aad0cff8e7e?flat 2025-10-08 15:46:40,509 - DEBUG - VM uve shows interface as {'fixed_ip4_list': ['192.168.1.3'], 'mac_address': '02:a8:2a:84:5f:e5', 'vm_name': 'ctest-vn1_vm1-67939375', 'uuid': 'a82a845f-e516-4530-9e58-8aad0cff8e7e', 'port_mirror_enabled': False, 'vm_uuid': '0fac7c9b-dfd7-4659-aeb3-169db789ca1b', 'active': True, 'ip6_active': False, 'is_health_check_active': True, 'tx_vlan': 65535, 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388', 'vhostuser_mode': 0, 'ip6_address': '::', 'admin_state': True, 'gateway': '192.168.1.1', 'label': 25, 'rx_vlan': 65535, 'l2_active': True, 'ip_address': '192.168.1.3', 'vn_uuid': 'acbfb9d7-3dae-4d32-b561-9b923c00fa90', '__T': 1759938396395141, 'ip4_active': True} 2025-10-08 15:46:40,509 - DEBUG - VM uve shows ip address as ['192.168.1.3', '::'] 2025-10-08 15:46:40,509 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 2025-10-08 15:46:40,509 - INFO - Verifying through opserver in 10.0.0.133 2025-10-08 15:46:40,509 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388?flat 2025-10-08 15:46:40,538 - DEBUG - expected vm list 0fac7c9b-dfd7-4659-aeb3-169db789ca1b 2025-10-08 15:46:40,538 - DEBUG - Extracted vm list ['0fac7c9b-dfd7-4659-aeb3-169db789ca1b'] 2025-10-08 15:46:40,538 - INFO - VM 0fac7c9b-dfd7-4659-aeb3-169db789ca1b is present in default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 2025-10-08 15:46:40,538 - DEBUG - Verifying vm in vrouter uve 2025-10-08 15:46:40,538 - DEBUG - Getting info from collector 10.0.0.242.. 2025-10-08 15:46:40,538 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-machine/0fac7c9b-dfd7-4659-aeb3-169db789ca1b?flat 2025-10-08 15:46:40,619 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4236-2'] 2025-10-08 15:46:40,619 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4236-2:Compute:contrail-vrouter-agent:0?flat 2025-10-08 15:46:40,677 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4236-2?flat 2025-10-08 15:46:40,717 - DEBUG - VM 0fac7c9b-dfd7-4659-aeb3-169db789ca1b is present in vrouter an-jenkins-deploy-platform-ansible-os-4236-2 uve 2025-10-08 15:46:40,717 - DEBUG - Validated that VM ctest-vn1_vm1-67939375 is in Vrouter an-jenkins-deploy-platform-ansible-os-4236-2 UVE 2025-10-08 15:46:40,717 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-10-08 15:46:40,729 - DEBUG - Expected tap interface of VM uuid 0fac7c9b-dfd7-4659-aeb3-169db789ca1b is default-domain:ctest-TestBasicVMVN0-24481103:a82a845f-e516-4530-9e58-8aad0cff8e7e 2025-10-08 15:46:40,729 - DEBUG - Expected VN of VM uuid 0fac7c9b-dfd7-4659-aeb3-169db789ca1b is default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 2025-10-08 15:46:40,729 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4236-2:Compute:contrail-vrouter-agent:0?flat 2025-10-08 15:46:40,786 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4236-2?flat 2025-10-08 15:46:40,826 - DEBUG - VM 0fac7c9b-dfd7-4659-aeb3-169db789ca1b is present in vrouter an-jenkins-deploy-platform-ansible-os-4236-2 uve 2025-10-08 15:46:40,826 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-24481103:a82a845f-e516-4530-9e58-8aad0cff8e7e of vm 0fac7c9b-dfd7-4659-aeb3-169db789ca1b is present in vrouter an-jenkins-deploy-platform-ansible-os-4236-2 uve 2025-10-08 15:46:40,826 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 of vm 0fac7c9b-dfd7-4659-aeb3-169db789ca1b is present in vrouter an-jenkins-deploy-platform-ansible-os-4236-2 uve 2025-10-08 15:46:40,826 - DEBUG - Verifying the 0fac7c9b-dfd7-4659-aeb3-169db789ca1b virtual network link through opserver 10.0.0.133 2025-10-08 15:46:40,826 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-machines 2025-10-08 15:46:40,843 - DEBUG - VM link and name as {'name': '0fac7c9b-dfd7-4659-aeb3-169db789ca1b', 'href': 'http://10.0.0.133:8081/analytics/uves/virtual-machine/0fac7c9b-dfd7-4659-aeb3-169db789ca1b?flat'} 2025-10-08 15:46:40,843 - DEBUG - Verifying the 0fac7c9b-dfd7-4659-aeb3-169db789ca1b virtual network link through opserver 10.0.0.50 2025-10-08 15:46:40,843 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2025-10-08 15:46:40,858 - DEBUG - VM link and name as {'name': '0fac7c9b-dfd7-4659-aeb3-169db789ca1b', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-machine/0fac7c9b-dfd7-4659-aeb3-169db789ca1b?flat'} 2025-10-08 15:46:40,858 - DEBUG - Verifying the 0fac7c9b-dfd7-4659-aeb3-169db789ca1b virtual network link through opserver 10.0.0.242 2025-10-08 15:46:40,858 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-machines 2025-10-08 15:46:40,874 - DEBUG - VM link and name as {'name': '0fac7c9b-dfd7-4659-aeb3-169db789ca1b', 'href': 'http://10.0.0.242:8081/analytics/uves/virtual-machine/0fac7c9b-dfd7-4659-aeb3-169db789ca1b?flat'} 2025-10-08 15:46:40,874 - INFO - VM ctest-vn1_vm1-67939375 validations in Opserver passed 2025-10-08 15:46:40,976 - DEBUG - VM is in ACTIVE state now 2025-10-08 15:46:40,976 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:46:40,977 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/f87b4189-800c-46d0-9010-31791364729c 2025-10-08 15:46:40,985 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine-interface/4f98a5d3-b393-4060-95c6-d836fc4371af 2025-10-08 15:46:40,997 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/f87b4189-800c-46d0-9010-31791364729c 2025-10-08 15:46:41,008 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine-interface/4f98a5d3-b393-4060-95c6-d836fc4371af 2025-10-08 15:46:41,019 - DEBUG - Requesting: http://10.0.0.242:8082/instance-ip/9488c973-3a41-4d06-8971-66d8af5717de 2025-10-08 15:46:41,032 - DEBUG - Verifying in api server 10.0.0.242 2025-10-08 15:46:41,032 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/f87b4189-800c-46d0-9010-31791364729c 2025-10-08 15:46:41,041 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine-interface/4f98a5d3-b393-4060-95c6-d836fc4371af 2025-10-08 15:46:41,055 - DEBUG - Requesting: http://10.0.0.242:8082/instance-ip/9488c973-3a41-4d06-8971-66d8af5717de 2025-10-08 15:46:41,066 - INFO - VM ctest-vn1_vm2-82798679 verfication in all API Servers passed 2025-10-08 15:46:41,088 - DEBUG - VM ctest-vn1_vm2-82798679 Tap interface: {'index': '3', 'name': 'tap4f98a5d3-b3', 'uuid': '4f98a5d3-b393-4060-95c6-d836fc4371af', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388:ctest-vn1-59329388', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388', 'vm_uuid': 'f87b4189-800c-46d0-9010-31791364729c', 'vm_name': 'ctest-vn1_vm2-82798679', 'ip_addr': '192.168.1.4', 'mac_addr': '02:4f:98:a5:d3:b3', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '127', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-24481103:4f98a5d3-b393-4060-95c6-d836fc4371af', 'sg_uuid_list': ['cbf6bc58-853e-4673-ab9d-9aaf1cab7a76'], 'static_route_list': None, 'vm_project_uuid': '2d0f9370-44fd-4f7b-99c3-73540ae4a690', 'admin_state': 'Enabled', 'flow_key_idx': '25', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-08 15:46:41,088 - DEBUG - Agent 10.0.0.61 vrf name: default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388:ctest-vn1-59329388 2025-10-08 15:46:41,095 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388:ctest-vn1-59329388', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388', 'table_label': '-1', 'vxlan_id': '14', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.241:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-10-08 15:46:41,184 - DEBUG - VM ctest-vn1_vm2-82798679 : Tap interface tap4f98a5d3-b3 is set to Active 2025-10-08 15:46:41,184 - DEBUG - Tap interface tap4f98a5d3-b3 detail : {'index': '3', 'name': 'tap4f98a5d3-b3', 'uuid': '4f98a5d3-b393-4060-95c6-d836fc4371af', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388:ctest-vn1-59329388', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388', 'vm_uuid': 'f87b4189-800c-46d0-9010-31791364729c', 'vm_name': 'ctest-vn1_vm2-82798679', 'ip_addr': '192.168.1.4', 'mac_addr': '02:4f:98:a5:d3:b3', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '127', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-24481103:4f98a5d3-b393-4060-95c6-d836fc4371af', 'sg_uuid_list': ['cbf6bc58-853e-4673-ab9d-9aaf1cab7a76'], 'static_route_list': None, 'vm_project_uuid': '2d0f9370-44fd-4f7b-99c3-73540ae4a690', 'admin_state': 'Enabled', 'flow_key_idx': '25', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-08 15:46:41,306 - DEBUG - Starting Layer 2 verification in Agent 2025-10-08 15:46:41,318 - DEBUG - Layer 2 path is seen for VM MAC 02:4f:98:a5:d3:b3 in agent 10.0.0.61 2025-10-08 15:46:41,318 - DEBUG - Active layer 2 route in agent is present for VMI tap4f98a5d3-b3 2025-10-08 15:46:41,318 - DEBUG - L2 label(29) matches bw route table and interface table 2025-10-08 15:46:41,325 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 is found in Agent of node 10.0.0.61 2025-10-08 15:46:41,463 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 is consistent in agent 10.0.0.61 2025-10-08 15:46:41,464 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.61 2025-10-08 15:46:41,464 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 verification for VM ctest-vn1_vm2-82798679 in Agent 10.0.0.61 passed 2025-10-08 15:46:41,595 - DEBUG - Starting all layer 2 verification in agent 10.0.0.61 2025-10-08 15:46:41,607 - DEBUG - Route for VM MAC 02:4f:98:a5:d3:b3 is consistent in agent 10.0.0.61 2025-10-08 15:46:41,614 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 is found in Agent of node 10.0.0.241 2025-10-08 15:46:41,757 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 is consistent in agent 10.0.0.241 2025-10-08 15:46:41,757 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.241 2025-10-08 15:46:41,757 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 verification for VM ctest-vn1_vm2-82798679 in Agent 10.0.0.241 passed 2025-10-08 15:46:41,878 - DEBUG - Starting all layer 2 verification in agent 10.0.0.241 2025-10-08 15:46:41,890 - DEBUG - Route for VM MAC 02:4f:98:a5:d3:b3 is consistent in agent 10.0.0.241 2025-10-08 15:46:41,891 - INFO - VM ctest-vn1_vm2-82798679 verifications in Compute nodes passed 2025-10-08 15:46:42,137 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.61 2025-10-08 15:46:42,143 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-10-08 15:46:42,254 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.241 2025-10-08 15:46:42,260 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.241, 24; Got 10.20.0.241,24 2025-10-08 15:46:42,260 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.241 2025-10-08 15:46:42,260 - INFO - Validated routes of VM ctest-vn1_vm2-82798679 in all vrouters 2025-10-08 15:46:42,810 - DEBUG - Starting all layer2 verification in 10.0.0.133 Control Node 2025-10-08 15:46:42,942 - DEBUG - Layer2 route found for VM MAC 02:4f:98:a5:d3:b3 in Control-node 10.0.0.133 2025-10-08 15:46:42,943 - DEBUG - L2 Label for VM ctest-vn1_vm2-82798679 same between Control-node 10.0.0.133 and Agent, Expected: 29, Seen: 29 2025-10-08 15:46:42,943 - DEBUG - Starting all layer2 verification in 10.0.0.50 Control Node 2025-10-08 15:46:43,086 - DEBUG - Layer2 route found for VM MAC 02:4f:98:a5:d3:b3 in Control-node 10.0.0.50 2025-10-08 15:46:43,087 - DEBUG - L2 Label for VM ctest-vn1_vm2-82798679 same between Control-node 10.0.0.50 and Agent, Expected: 29, Seen: 29 2025-10-08 15:46:43,087 - DEBUG - Starting all layer2 verification in 10.0.0.242 Control Node 2025-10-08 15:46:43,226 - DEBUG - Layer2 route found for VM MAC 02:4f:98:a5:d3:b3 in Control-node 10.0.0.242 2025-10-08 15:46:43,226 - DEBUG - L2 Label for VM ctest-vn1_vm2-82798679 same between Control-node 10.0.0.242 and Agent, Expected: 29, Seen: 29 2025-10-08 15:46:43,226 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-82798679 passed 2025-10-08 15:46:43,226 - DEBUG - Verifying the vm in opserver 2025-10-08 15:46:43,226 - DEBUG - Verifying in collector 10.0.0.242 ... 2025-10-08 15:46:43,226 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-machine/f87b4189-800c-46d0-9010-31791364729c?flat 2025-10-08 15:46:43,247 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-24481103:4f98a5d3-b393-4060-95c6-d836fc4371af?flat 2025-10-08 15:46:43,270 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-24481103:4f98a5d3-b393-4060-95c6-d836fc4371af?flat 2025-10-08 15:46:43,295 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-machine/f87b4189-800c-46d0-9010-31791364729c?flat 2025-10-08 15:46:43,314 - DEBUG - Verifying vm in vn uve 2025-10-08 15:46:43,314 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-24481103:4f98a5d3-b393-4060-95c6-d836fc4371af?flat 2025-10-08 15:46:43,342 - DEBUG - VM uve shows interface as {'vm_uuid': 'f87b4189-800c-46d0-9010-31791364729c', 'fixed_ip4_list': ['192.168.1.4'], 'mac_address': '02:4f:98:a5:d3:b3', 'admin_state': True, 'l2_active': True, 'rx_vlan': 65535, 'tx_vlan': 65535, 'gateway': '192.168.1.1', 'port_mirror_enabled': False, 'ip4_active': True, 'ip_address': '192.168.1.4', 'vn_uuid': 'acbfb9d7-3dae-4d32-b561-9b923c00fa90', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388', 'vhostuser_mode': 0, 'uuid': '4f98a5d3-b393-4060-95c6-d836fc4371af', 'active': True, 'label': 24, 'ip6_address': '::', 'vm_name': 'ctest-vn1_vm2-82798679', '__T': 1759938387923684, 'ip6_active': False, 'is_health_check_active': True} 2025-10-08 15:46:43,342 - DEBUG - VM uve shows ip address as ['192.168.1.4', '::'] 2025-10-08 15:46:43,342 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 2025-10-08 15:46:43,342 - INFO - Verifying through opserver in 10.0.0.133 2025-10-08 15:46:43,342 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388?flat 2025-10-08 15:46:43,372 - DEBUG - expected vm list f87b4189-800c-46d0-9010-31791364729c 2025-10-08 15:46:43,372 - DEBUG - Extracted vm list ['f87b4189-800c-46d0-9010-31791364729c'] 2025-10-08 15:46:43,373 - INFO - VM f87b4189-800c-46d0-9010-31791364729c is present in default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 2025-10-08 15:46:43,373 - DEBUG - Verifying vm in vrouter uve 2025-10-08 15:46:43,373 - DEBUG - Getting info from collector 10.0.0.242.. 2025-10-08 15:46:43,373 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-machine/f87b4189-800c-46d0-9010-31791364729c?flat 2025-10-08 15:46:43,395 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4236-1'] 2025-10-08 15:46:43,396 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4236-1:Compute:contrail-vrouter-agent:0?flat 2025-10-08 15:46:43,449 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4236-1?flat 2025-10-08 15:46:43,530 - DEBUG - VM f87b4189-800c-46d0-9010-31791364729c is present in vrouter an-jenkins-deploy-platform-ansible-os-4236-1 uve 2025-10-08 15:46:43,530 - DEBUG - Validated that VM ctest-vn1_vm2-82798679 is in Vrouter an-jenkins-deploy-platform-ansible-os-4236-1 UVE 2025-10-08 15:46:43,530 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-10-08 15:46:43,539 - DEBUG - Expected tap interface of VM uuid f87b4189-800c-46d0-9010-31791364729c is default-domain:ctest-TestBasicVMVN0-24481103:4f98a5d3-b393-4060-95c6-d836fc4371af 2025-10-08 15:46:43,539 - DEBUG - Expected VN of VM uuid f87b4189-800c-46d0-9010-31791364729c is default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 2025-10-08 15:46:43,539 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4236-1:Compute:contrail-vrouter-agent:0?flat 2025-10-08 15:46:43,597 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4236-1?flat 2025-10-08 15:46:43,693 - DEBUG - VM f87b4189-800c-46d0-9010-31791364729c is present in vrouter an-jenkins-deploy-platform-ansible-os-4236-1 uve 2025-10-08 15:46:43,693 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-24481103:4f98a5d3-b393-4060-95c6-d836fc4371af of vm f87b4189-800c-46d0-9010-31791364729c is present in vrouter an-jenkins-deploy-platform-ansible-os-4236-1 uve 2025-10-08 15:46:43,694 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 of vm f87b4189-800c-46d0-9010-31791364729c is present in vrouter an-jenkins-deploy-platform-ansible-os-4236-1 uve 2025-10-08 15:46:43,694 - DEBUG - Verifying the f87b4189-800c-46d0-9010-31791364729c virtual network link through opserver 10.0.0.133 2025-10-08 15:46:43,694 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-machines 2025-10-08 15:46:43,709 - DEBUG - VM link and name as {'name': 'f87b4189-800c-46d0-9010-31791364729c', 'href': 'http://10.0.0.133:8081/analytics/uves/virtual-machine/f87b4189-800c-46d0-9010-31791364729c?flat'} 2025-10-08 15:46:43,709 - DEBUG - Verifying the f87b4189-800c-46d0-9010-31791364729c virtual network link through opserver 10.0.0.50 2025-10-08 15:46:43,709 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2025-10-08 15:46:43,725 - DEBUG - VM link and name as {'name': 'f87b4189-800c-46d0-9010-31791364729c', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-machine/f87b4189-800c-46d0-9010-31791364729c?flat'} 2025-10-08 15:46:43,725 - DEBUG - Verifying the f87b4189-800c-46d0-9010-31791364729c virtual network link through opserver 10.0.0.242 2025-10-08 15:46:43,725 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-machines 2025-10-08 15:46:43,746 - DEBUG - VM link and name as {'name': 'f87b4189-800c-46d0-9010-31791364729c', 'href': 'http://10.0.0.242:8081/analytics/uves/virtual-machine/f87b4189-800c-46d0-9010-31791364729c?flat'} 2025-10-08 15:46:43,746 - INFO - VM ctest-vn1_vm2-82798679 validations in Opserver passed 2025-10-08 15:46:43,747 - INFO - Checking the ping between the VM again 2025-10-08 15:46:43,748 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:46:43,748 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.241, gateway password: c0ntrail123 2025-10-08 15:46:43,748 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4 2025-10-08 15:46:46,203 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data. 64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=3.41 ms 64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.22 ms 64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.25 ms --- 192.168.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.223/1.965/3.419/1.028 ms 2025-10-08 15:46:46,203 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-67939375 passed 2025-10-08 15:46:46,203 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:46:46,203 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.61, gateway password: c0ntrail123 2025-10-08 15:46:46,204 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3 2025-10-08 15:46:48,693 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. 64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=2.70 ms 64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=2.50 ms 64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.63 ms --- 192.168.1.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.639/2.282/2.705/0.462 ms 2025-10-08 15:46:48,693 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-82798679 passed 2025-10-08 15:46:48,694 - INFO - Deleting VM ctest-vn1_vm2-82798679 2025-10-08 15:46:48,778 - DEBUG - Verifying in api server 10.0.0.242 2025-10-08 15:46:48,778 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/f87b4189-800c-46d0-9010-31791364729c 2025-10-08 15:46:48,789 - DEBUG - VM ID f87b4189-800c-46d0-9010-31791364729c of VM ctest-vn1_vm2-82798679 is still found in API Server 2025-10-08 15:46:50,790 - DEBUG - Verifying in api server 10.0.0.242 2025-10-08 15:46:50,790 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/f87b4189-800c-46d0-9010-31791364729c 2025-10-08 15:46:50,798 - DEBUG - Response Code: 404 2025-10-08 15:46:50,798 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/f87b4189-800c-46d0-9010-31791364729c 2025-10-08 15:46:50,805 - DEBUG - Response Code: 404 2025-10-08 15:46:50,805 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/f87b4189-800c-46d0-9010-31791364729c 2025-10-08 15:46:50,814 - DEBUG - Response Code: 404 2025-10-08 15:46:50,815 - INFO - VM ctest-vn1_vm2-82798679 is fully removed in API-Server 2025-10-08 15:46:50,973 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-10-08 15:46:51,021 - DEBUG - Validated that vrouter 10.0.0.61 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 2025-10-08 15:46:51,156 - DEBUG - VM route 192.168.1.4 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 of compute 10.0.0.241 2025-10-08 15:46:51,163 - DEBUG - Validated that vrouter 10.0.0.241 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 2025-10-08 15:46:51,163 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 2025-10-08 15:46:51,163 - INFO - VM ctest-vn1_vm2-82798679 is removed in Compute, and routes are removed in all compute nodes 2025-10-08 15:46:51,436 - INFO - Routes for VM ctest-vn1_vm2-82798679 is removed in all control-nodes 2025-10-08 15:46:51,437 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4236-1:Compute:contrail-vrouter-agent:0?flat 2025-10-08 15:46:51,489 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4236-1?flat 2025-10-08 15:46:51,523 - DEBUG - VM f87b4189-800c-46d0-9010-31791364729c is still present in vrouter an-jenkins-deploy-platform-ansible-os-4236-1 uve 2025-10-08 15:46:51,523 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-08 15:46:51,524 - DEBUG - interface for vm f87b4189-800c-46d0-9010-31791364729c deleted from vrouter uve ... 2025-10-08 15:46:51,524 - DEBUG - Deleted interface not in error interface list ... 2025-10-08 15:46:54,525 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4236-1:Compute:contrail-vrouter-agent:0?flat 2025-10-08 15:46:54,600 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4236-1?flat 2025-10-08 15:46:54,633 - DEBUG - VM f87b4189-800c-46d0-9010-31791364729c is still present in vrouter an-jenkins-deploy-platform-ansible-os-4236-1 uve 2025-10-08 15:46:54,633 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-08 15:46:54,633 - DEBUG - interface for vm f87b4189-800c-46d0-9010-31791364729c deleted from vrouter uve ... 2025-10-08 15:46:54,633 - DEBUG - Deleted interface not in error interface list ... 2025-10-08 15:46:57,634 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4236-1:Compute:contrail-vrouter-agent:0?flat 2025-10-08 15:46:57,687 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4236-1?flat 2025-10-08 15:46:57,721 - DEBUG - VM f87b4189-800c-46d0-9010-31791364729c is still present in vrouter an-jenkins-deploy-platform-ansible-os-4236-1 uve 2025-10-08 15:46:57,721 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-08 15:46:57,721 - DEBUG - interface for vm f87b4189-800c-46d0-9010-31791364729c deleted from vrouter uve ... 2025-10-08 15:46:57,722 - DEBUG - Deleted interface not in error interface list ... 2025-10-08 15:47:00,723 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4236-1:Compute:contrail-vrouter-agent:0?flat 2025-10-08 15:47:00,779 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4236-1?flat 2025-10-08 15:47:00,811 - DEBUG - VM f87b4189-800c-46d0-9010-31791364729c is not present in vrouter an-jenkins-deploy-platform-ansible-os-4236-1 uve 2025-10-08 15:47:00,811 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-08 15:47:00,811 - DEBUG - interface for vm f87b4189-800c-46d0-9010-31791364729c deleted from vrouter uve ... 2025-10-08 15:47:00,811 - DEBUG - Deleted interface not in error interface list ... 2025-10-08 15:47:00,811 - DEBUG - Validated that VM f87b4189-800c-46d0-9010-31791364729c is removed in Vrouter UVE 2025-10-08 15:47:00,811 - DEBUG - Verifying through opserver in 10.0.0.133 2025-10-08 15:47:00,811 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388?flat 2025-10-08 15:47:00,835 - DEBUG - f87b4189-800c-46d0-9010-31791364729c BM not in default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 uve 2025-10-08 15:47:00,835 - DEBUG - Validated that VM f87b4189-800c-46d0-9010-31791364729c is not present in the VN UVE 2025-10-08 15:47:00,835 - DEBUG - Verifying the f87b4189-800c-46d0-9010-31791364729c virtual network link through opserver 10.0.0.133 2025-10-08 15:47:00,835 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-machines 2025-10-08 15:47:00,852 - INFO - f87b4189-800c-46d0-9010-31791364729c vm uve deleted from opserver 2025-10-08 15:47:00,852 - DEBUG - Verifying the f87b4189-800c-46d0-9010-31791364729c virtual network link through opserver 10.0.0.50 2025-10-08 15:47:00,852 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2025-10-08 15:47:00,871 - INFO - f87b4189-800c-46d0-9010-31791364729c vm uve deleted from opserver 2025-10-08 15:47:00,871 - DEBUG - Verifying the f87b4189-800c-46d0-9010-31791364729c virtual network link through opserver 10.0.0.242 2025-10-08 15:47:00,872 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-machines 2025-10-08 15:47:00,887 - INFO - f87b4189-800c-46d0-9010-31791364729c vm uve deleted from opserver 2025-10-08 15:47:00,887 - INFO - Validated that VM f87b4189-800c-46d0-9010-31791364729c is removed from Opserver 2025-10-08 15:47:00,887 - INFO - Deleting VM ctest-vn1_vm1-67939375 2025-10-08 15:47:01,013 - DEBUG - Verifying in api server 10.0.0.242 2025-10-08 15:47:01,013 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/0fac7c9b-dfd7-4659-aeb3-169db789ca1b 2025-10-08 15:47:01,022 - DEBUG - VM ID 0fac7c9b-dfd7-4659-aeb3-169db789ca1b of VM ctest-vn1_vm1-67939375 is still found in API Server 2025-10-08 15:47:03,023 - DEBUG - Verifying in api server 10.0.0.242 2025-10-08 15:47:03,023 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/0fac7c9b-dfd7-4659-aeb3-169db789ca1b 2025-10-08 15:47:03,033 - DEBUG - Response Code: 404 2025-10-08 15:47:03,034 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/0fac7c9b-dfd7-4659-aeb3-169db789ca1b 2025-10-08 15:47:03,042 - DEBUG - Response Code: 404 2025-10-08 15:47:03,042 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/0fac7c9b-dfd7-4659-aeb3-169db789ca1b 2025-10-08 15:47:03,053 - DEBUG - Response Code: 404 2025-10-08 15:47:03,053 - INFO - VM ctest-vn1_vm1-67939375 is fully removed in API-Server 2025-10-08 15:47:03,227 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-10-08 15:47:03,287 - DEBUG - Validated that vrouter 10.0.0.61 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 2025-10-08 15:47:03,301 - DEBUG - Validated that vrouter 10.0.0.241 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 2025-10-08 15:47:03,302 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 2025-10-08 15:47:03,302 - INFO - VM ctest-vn1_vm1-67939375 is removed in Compute, and routes are removed in all compute nodes 2025-10-08 15:47:03,550 - INFO - Routes for VM ctest-vn1_vm1-67939375 is removed in all control-nodes 2025-10-08 15:47:03,550 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4236-2:Compute:contrail-vrouter-agent:0?flat 2025-10-08 15:47:03,589 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4236-2?flat 2025-10-08 15:47:03,625 - DEBUG - VM 0fac7c9b-dfd7-4659-aeb3-169db789ca1b is still present in vrouter an-jenkins-deploy-platform-ansible-os-4236-2 uve 2025-10-08 15:47:03,626 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-08 15:47:03,626 - DEBUG - interface for vm 0fac7c9b-dfd7-4659-aeb3-169db789ca1b deleted from vrouter uve ... 2025-10-08 15:47:03,626 - DEBUG - Deleted interface not in error interface list ... 2025-10-08 15:47:06,627 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4236-2:Compute:contrail-vrouter-agent:0?flat 2025-10-08 15:47:06,674 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4236-2?flat 2025-10-08 15:47:06,711 - DEBUG - VM 0fac7c9b-dfd7-4659-aeb3-169db789ca1b is not present in vrouter an-jenkins-deploy-platform-ansible-os-4236-2 uve 2025-10-08 15:47:06,711 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-08 15:47:06,711 - DEBUG - interface for vm 0fac7c9b-dfd7-4659-aeb3-169db789ca1b deleted from vrouter uve ... 2025-10-08 15:47:06,711 - DEBUG - Deleted interface not in error interface list ... 2025-10-08 15:47:06,711 - DEBUG - Validated that VM 0fac7c9b-dfd7-4659-aeb3-169db789ca1b is removed in Vrouter UVE 2025-10-08 15:47:06,711 - DEBUG - Verifying through opserver in 10.0.0.133 2025-10-08 15:47:06,711 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388?flat 2025-10-08 15:47:06,737 - DEBUG - 0fac7c9b-dfd7-4659-aeb3-169db789ca1b BM not in default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-59329388 uve 2025-10-08 15:47:06,737 - DEBUG - Validated that VM 0fac7c9b-dfd7-4659-aeb3-169db789ca1b is not present in the VN UVE 2025-10-08 15:47:06,737 - DEBUG - Verifying the 0fac7c9b-dfd7-4659-aeb3-169db789ca1b virtual network link through opserver 10.0.0.133 2025-10-08 15:47:06,738 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-machines 2025-10-08 15:47:06,753 - INFO - 0fac7c9b-dfd7-4659-aeb3-169db789ca1b vm uve deleted from opserver 2025-10-08 15:47:06,753 - DEBUG - Verifying the 0fac7c9b-dfd7-4659-aeb3-169db789ca1b virtual network link through opserver 10.0.0.50 2025-10-08 15:47:06,753 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2025-10-08 15:47:06,771 - INFO - 0fac7c9b-dfd7-4659-aeb3-169db789ca1b vm uve deleted from opserver 2025-10-08 15:47:06,771 - DEBUG - Verifying the 0fac7c9b-dfd7-4659-aeb3-169db789ca1b virtual network link through opserver 10.0.0.242 2025-10-08 15:47:06,771 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-machines 2025-10-08 15:47:06,788 - INFO - 0fac7c9b-dfd7-4659-aeb3-169db789ca1b vm uve deleted from opserver 2025-10-08 15:47:06,788 - INFO - Validated that VM 0fac7c9b-dfd7-4659-aeb3-169db789ca1b is removed from Opserver 2025-10-08 15:47:06,788 - INFO - Deleting VN ctest-vn1-59329388 2025-10-08 15:47:06,974 - DEBUG - Response for deleting network () 2025-10-08 15:47:06,974 - DEBUG - Requesting: http://10.0.0.133:8082/routing-instance/81b0778f-d29f-437a-90ec-6635035b81a3 2025-10-08 15:47:06,982 - DEBUG - Response Code: 404 2025-10-08 15:47:06,982 - DEBUG - Requesting: http://10.0.0.133:8082/domains 2025-10-08 15:47:06,990 - DEBUG - Requesting: http://10.0.0.133:8082/domain/adc52b80-2ed1-485c-90d4-83e78c8e1400 2025-10-08 15:47:07,017 - DEBUG - Requesting: http://10.0.0.133:8082/project/2d0f9370-44fd-4f7b-99c3-73540ae4a690 2025-10-08 15:47:07,112 - INFO - Validated that VN ctest-vn1-59329388 is not found in API Server 2025-10-08 15:47:07,124 - DEBUG - VN ctest-vn1-59329388 is not present in Agent 10.0.0.61 2025-10-08 15:47:07,134 - DEBUG - VN ctest-vn1-59329388 is not present in Agent 10.0.0.241 2025-10-08 15:47:07,135 - INFO - Validated that VN ctest-vn1-59329388 is not in any agent 2025-10-08 15:47:07,172 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-59329388 info 2025-10-08 15:47:08,437 - DEBUG - Skipping xmpp flap check 2025-10-08 15:47:08,437 - INFO - END TEST : test_control_node_switchover : PASSED[0:02:58] 2025-10-08 15:47:08,437 - INFO - -------------------------------------------------------------------------------- 2025-10-08 15:47:08,440 - INFO - ================================================================================ 2025-10-08 15:47:08,440 - INFO - STARTING TEST : test_process_restart_in_policy_between_vns 2025-10-08 15:47:08,440 - INFO - TEST DESCRIPTION : Test to validate that with policy having rule to check icmp fwding between VMs on different VNs , ping between VMs should pass with process restarts 1. Pick 2 VN's from resource pool which has one VM each 2. Create policy with icmp allow rule between those VN's and bind it networks 3. Ping from one VM to another VM 4. Restart process 'vrouter' and 'control' on setup 5. Ping again between VM's after process restart Pass criteria: Step 2,3,4 and 5 should pass 2025-10-08 15:47:09,700 - DEBUG - Skipping xmpp flap check 2025-10-08 15:47:09,700 - INFO - Initial checks done. Running the testcase now 2025-10-08 15:47:09,700 - INFO - 2025-10-08 15:47:09,709 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-10-08 15:47:10,000 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-10-08 15:47:10,572 - INFO - Created VN ctest-vn1-31040416, UUID :6105f376-caef-48e9-9074-6e6879ce8ae4 2025-10-08 15:47:10,929 - DEBUG - Requesting: http://10.0.0.133:8082/domains 2025-10-08 15:47:10,938 - DEBUG - Requesting: http://10.0.0.133:8082/domain/adc52b80-2ed1-485c-90d4-83e78c8e1400 2025-10-08 15:47:10,953 - DEBUG - Requesting: http://10.0.0.133:8082/project/2d0f9370-44fd-4f7b-99c3-73540ae4a690 2025-10-08 15:47:11,072 - DEBUG - Requesting: http://10.0.0.133:8082/virtual-network/6105f376-caef-48e9-9074-6e6879ce8ae4 2025-10-08 15:47:11,090 - DEBUG - Requesting: http://10.0.0.133:8082/virtual-network/6105f376-caef-48e9-9074-6e6879ce8ae4 2025-10-08 15:47:11,104 - DEBUG - Requesting: http://10.0.0.133:8082/routing-instance/d733f0ef-873d-40f1-ba17-4e5de4976db9 2025-10-08 15:47:11,113 - DEBUG - Requesting: http://10.0.0.133:8082/routing-instance/d733f0ef-873d-40f1-ba17-4e5de4976db9 2025-10-08 15:47:11,123 - DEBUG - Requesting: http://10.0.0.133:8082/route-target/472aa6bb-0a53-4234-a8fe-768b270ce29f 2025-10-08 15:47:11,135 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-08 15:47:11,135 - DEBUG - Requesting: http://10.0.0.133:8082/virtual-network/6105f376-caef-48e9-9074-6e6879ce8ae4 2025-10-08 15:47:11,150 - DEBUG - Requesting: http://10.0.0.133:8082/routing-instance/d733f0ef-873d-40f1-ba17-4e5de4976db9 2025-10-08 15:47:11,175 - INFO - Verified VN network id 14 for VN 6105f376-caef-48e9-9074-6e6879ce8ae4 2025-10-08 15:47:11,176 - INFO - Verifications in API Server for VN ctest-vn1-31040416 passed 2025-10-08 15:47:11,176 - DEBUG - Requesting: http://10.0.0.133:8082/virtual-network/6105f376-caef-48e9-9074-6e6879ce8ae4 2025-10-08 15:47:11,188 - DEBUG - Requesting: http://10.0.0.133:8082/routing-instance/d733f0ef-873d-40f1-ba17-4e5de4976db9 2025-10-08 15:47:11,197 - DEBUG - Requesting: http://10.0.0.133:8082/routing-instance/d733f0ef-873d-40f1-ba17-4e5de4976db9 2025-10-08 15:47:11,208 - DEBUG - Requesting: http://10.0.0.133:8082/route-target/472aa6bb-0a53-4234-a8fe-768b270ce29f 2025-10-08 15:47:11,224 - DEBUG - Control-node 10.0.0.133 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-24481103', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '6991261688098932969', 'uuid-lslong': '10409066033759292132'}, 'enable': 'true', 'created': '2025-10-08T15:47:10', 'last-modified': '2025-10-08T15:47:10', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.500751'} 2025-10-08 15:47:11,234 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-08 15:47:11,240 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-24481103', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '6991261688098932969', 'uuid-lslong': '10409066033759292132'}, 'enable': 'true', 'created': '2025-10-08T15:47:10', 'last-modified': '2025-10-08T15:47:10', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.514935'} 2025-10-08 15:47:11,248 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-08 15:47:11,257 - DEBUG - Control-node 10.0.0.242 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-24481103', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '6991261688098932969', 'uuid-lslong': '10409066033759292132'}, 'enable': 'true', 'created': '2025-10-08T15:47:10', 'last-modified': '2025-10-08T15:47:10', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.526601'} 2025-10-08 15:47:11,267 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-08 15:47:11,268 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-31040416 passed 2025-10-08 15:47:11,268 - DEBUG - ====Verifying policy data for ctest-vn1-31040416 in API_Server ====== 2025-10-08 15:47:11,268 - DEBUG - Requesting: http://10.0.0.133:8082/domains 2025-10-08 15:47:11,275 - DEBUG - Requesting: http://10.0.0.133:8082/domain/adc52b80-2ed1-485c-90d4-83e78c8e1400 2025-10-08 15:47:11,290 - DEBUG - Requesting: http://10.0.0.133:8082/project/2d0f9370-44fd-4f7b-99c3-73540ae4a690 2025-10-08 15:47:11,398 - DEBUG - Requesting: http://10.0.0.133:8082/virtual-network/6105f376-caef-48e9-9074-6e6879ce8ae4 2025-10-08 15:47:11,412 - DEBUG - =>VN ctest-vn1-31040416 has no policy to be verified 2025-10-08 15:47:11,412 - DEBUG - Verifying the vn in opserver 2025-10-08 15:47:11,412 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 virtual network link through opserver 10.0.0.133 2025-10-08 15:47:11,412 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-networks 2025-10-08 15:47:11,428 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416', 'href': 'http://10.0.0.133:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416?flat'} 2025-10-08 15:47:11,428 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 virtual network link through opserver 10.0.0.50 2025-10-08 15:47:11,428 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2025-10-08 15:47:11,445 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416?flat'} 2025-10-08 15:47:11,445 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 virtual network link through opserver 10.0.0.242 2025-10-08 15:47:11,446 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-networks 2025-10-08 15:47:11,465 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416', 'href': 'http://10.0.0.242:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416?flat'} 2025-10-08 15:47:11,465 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 is found in opserver 2025-10-08 15:47:11,490 - DEBUG - Do not have enough data to verify VN in agent 2025-10-08 15:47:11,501 - DEBUG - VRF ids for VN ctest-vn1-31040416: {} 2025-10-08 15:47:12,296 - INFO - Created VN ctest-vn2-11998607, UUID :d92c3b74-c648-4bef-8568-e08ca301bc00 2025-10-08 15:47:12,688 - DEBUG - Requesting: http://10.0.0.133:8082/domains 2025-10-08 15:47:12,698 - DEBUG - Requesting: http://10.0.0.133:8082/domain/adc52b80-2ed1-485c-90d4-83e78c8e1400 2025-10-08 15:47:12,716 - DEBUG - Requesting: http://10.0.0.133:8082/project/2d0f9370-44fd-4f7b-99c3-73540ae4a690 2025-10-08 15:47:12,838 - DEBUG - Requesting: http://10.0.0.133:8082/virtual-network/d92c3b74-c648-4bef-8568-e08ca301bc00 2025-10-08 15:47:12,856 - DEBUG - Requesting: http://10.0.0.133:8082/virtual-network/d92c3b74-c648-4bef-8568-e08ca301bc00 2025-10-08 15:47:12,870 - DEBUG - Requesting: http://10.0.0.133:8082/routing-instance/95310814-43d1-496c-96be-cda460a203d9 2025-10-08 15:47:12,886 - DEBUG - Requesting: http://10.0.0.133:8082/routing-instance/95310814-43d1-496c-96be-cda460a203d9 2025-10-08 15:47:12,899 - DEBUG - Requesting: http://10.0.0.133:8082/route-target/082655f4-cc6f-44a4-b7bb-e6ae75e4615b 2025-10-08 15:47:12,908 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-10-08 15:47:12,908 - DEBUG - Requesting: http://10.0.0.133:8082/virtual-network/d92c3b74-c648-4bef-8568-e08ca301bc00 2025-10-08 15:47:12,920 - DEBUG - Requesting: http://10.0.0.133:8082/routing-instance/95310814-43d1-496c-96be-cda460a203d9 2025-10-08 15:47:12,939 - INFO - Verified VN network id 15 for VN d92c3b74-c648-4bef-8568-e08ca301bc00 2025-10-08 15:47:12,939 - INFO - Verifications in API Server for VN ctest-vn2-11998607 passed 2025-10-08 15:47:12,939 - DEBUG - Requesting: http://10.0.0.133:8082/virtual-network/d92c3b74-c648-4bef-8568-e08ca301bc00 2025-10-08 15:47:12,950 - DEBUG - Requesting: http://10.0.0.133:8082/routing-instance/95310814-43d1-496c-96be-cda460a203d9 2025-10-08 15:47:12,959 - DEBUG - Requesting: http://10.0.0.133:8082/routing-instance/95310814-43d1-496c-96be-cda460a203d9 2025-10-08 15:47:12,968 - DEBUG - Requesting: http://10.0.0.133:8082/route-target/082655f4-cc6f-44a4-b7bb-e6ae75e4615b 2025-10-08 15:47:12,984 - DEBUG - Control-node 10.0.0.133 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-24481103', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '15648948177934502895', 'uuid-lslong': '9613180299257166848'}, 'enable': 'true', 'created': '2025-10-08T15:47:12', 'last-modified': '2025-10-08T15:47:12', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.530474'} 2025-10-08 15:47:12,990 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-10-08 15:47:12,997 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-24481103', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '15648948177934502895', 'uuid-lslong': '9613180299257166848'}, 'enable': 'true', 'created': '2025-10-08T15:47:12', 'last-modified': '2025-10-08T15:47:12', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.543622'} 2025-10-08 15:47:13,005 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-10-08 15:47:13,012 - DEBUG - Control-node 10.0.0.242 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-24481103', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '15648948177934502895', 'uuid-lslong': '9613180299257166848'}, 'enable': 'true', 'created': '2025-10-08T15:47:12', 'last-modified': '2025-10-08T15:47:12', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.551738'} 2025-10-08 15:47:13,019 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-10-08 15:47:13,019 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn2-11998607 passed 2025-10-08 15:47:13,019 - DEBUG - ====Verifying policy data for ctest-vn2-11998607 in API_Server ====== 2025-10-08 15:47:13,019 - DEBUG - Requesting: http://10.0.0.133:8082/domains 2025-10-08 15:47:13,029 - DEBUG - Requesting: http://10.0.0.133:8082/domain/adc52b80-2ed1-485c-90d4-83e78c8e1400 2025-10-08 15:47:13,043 - DEBUG - Requesting: http://10.0.0.133:8082/project/2d0f9370-44fd-4f7b-99c3-73540ae4a690 2025-10-08 15:47:13,165 - DEBUG - Requesting: http://10.0.0.133:8082/virtual-network/d92c3b74-c648-4bef-8568-e08ca301bc00 2025-10-08 15:47:13,181 - DEBUG - =>VN ctest-vn2-11998607 has no policy to be verified 2025-10-08 15:47:13,181 - DEBUG - Verifying the vn in opserver 2025-10-08 15:47:13,181 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 virtual network link through opserver 10.0.0.133 2025-10-08 15:47:13,182 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-networks 2025-10-08 15:47:13,197 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607', 'href': 'http://10.0.0.133:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607?flat'} 2025-10-08 15:47:13,197 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 virtual network link through opserver 10.0.0.50 2025-10-08 15:47:13,197 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2025-10-08 15:47:13,216 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607?flat'} 2025-10-08 15:47:13,216 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 virtual network link through opserver 10.0.0.242 2025-10-08 15:47:13,216 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-networks 2025-10-08 15:47:13,233 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607', 'href': 'http://10.0.0.242:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607?flat'} 2025-10-08 15:47:13,233 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 is found in opserver 2025-10-08 15:47:13,258 - DEBUG - Do not have enough data to verify VN in agent 2025-10-08 15:47:13,271 - DEBUG - VRF ids for VN ctest-vn2-11998607: {} 2025-10-08 15:47:15,057 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4236-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4236-2) 2025-10-08 15:47:16,675 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4236-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4236-1) 2025-10-08 15:47:16,675 - INFO - Waiting for VM ctest-vn1_vm1-37998096 to be up.. 2025-10-08 15:47:16,784 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-10-08 15:47:21,894 - DEBUG - VM is in ACTIVE state now 2025-10-08 15:47:21,894 - INFO - VM name : ctest-vn1_vm1-37998096 2025-10-08 15:47:21,992 - DEBUG - VM ctest-vn1_vm1-37998096 ID is 28f3e351-9a4d-49ad-a202-d99b531f81c9 2025-10-08 15:47:21,992 - DEBUG - VM ctest-vn1_vm1-37998096 launched on Node an-jenkins-deploy-platform-ansible-os-4236-2 2025-10-08 15:47:22,105 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/28f3e351-9a4d-49ad-a202-d99b531f81c9 2025-10-08 15:47:22,119 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine-interface/8f2718ed-d877-4eed-8deb-25e83d6c6223 2025-10-08 15:47:25,314 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1016ms') 2025-10-08 15:47:25,314 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-37998096 failed! 2025-10-08 15:47:25,330 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 is 192.168.1.254 and allocation pool is NOT set 2025-10-08 15:47:29,418 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2025-10-08 15:47:29,418 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-37998096 failed! 2025-10-08 15:47:29,438 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 is 192.168.1.254 and allocation pool is NOT set 2025-10-08 15:47:33,513 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms') 2025-10-08 15:47:33,513 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-37998096 failed! 2025-10-08 15:47:33,529 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 is 192.168.1.254 and allocation pool is NOT set 2025-10-08 15:47:37,612 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-10-08 15:47:37,612 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-37998096 failed! 2025-10-08 15:47:37,632 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 is 192.168.1.254 and allocation pool is NOT set 2025-10-08 15:47:41,699 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1006ms') 2025-10-08 15:47:41,699 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-37998096 failed! 2025-10-08 15:47:41,715 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 is 192.168.1.254 and allocation pool is NOT set 2025-10-08 15:47:45,794 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1024ms') 2025-10-08 15:47:45,794 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-37998096 failed! 2025-10-08 15:47:45,809 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 is 192.168.1.254 and allocation pool is NOT set 2025-10-08 15:47:49,899 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2025-10-08 15:47:49,899 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-37998096 failed! 2025-10-08 15:47:49,916 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 is 192.168.1.254 and allocation pool is NOT set 2025-10-08 15:47:53,986 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms') 2025-10-08 15:47:53,986 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-37998096 failed! 2025-10-08 15:47:54,002 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 is 192.168.1.254 and allocation pool is NOT set 2025-10-08 15:47:56,071 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=5.66 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.836 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 0.836/3.246/5.656/2.410 ms') 2025-10-08 15:47:56,072 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-37998096 passed 2025-10-08 15:47:56,153 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:47:56,153 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-37998096, IP 192.168.1.252, Port 22 2025-10-08 15:47:56,223 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-08 15:47:56,329 - DEBUG - VM ctest-vn1_vm1-37998096 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-08 15:48:01,330 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:48:01,330 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-37998096, IP 192.168.1.252, Port 22 2025-10-08 15:48:01,400 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-08 15:48:01,517 - DEBUG - VM ctest-vn1_vm1-37998096 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-08 15:48:06,518 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:48:06,518 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-37998096, IP 192.168.1.252, Port 22 2025-10-08 15:48:06,587 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-08 15:48:06,701 - DEBUG - VM ctest-vn1_vm1-37998096 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-08 15:48:11,702 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:48:11,702 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-37998096, IP 192.168.1.252, Port 22 2025-10-08 15:48:11,772 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-08 15:48:11,876 - DEBUG - VM ctest-vn1_vm1-37998096 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-08 15:48:16,877 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:48:16,878 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-37998096, IP 192.168.1.252, Port 22 2025-10-08 15:48:17,050 - DEBUG - VM ctest-vn1_vm1-37998096 is ready for SSH connections 2025-10-08 15:48:17,051 - INFO - Waiting for VM ctest-vn2_vm1-83080481 to be up.. 2025-10-08 15:48:17,188 - DEBUG - VM is in ACTIVE state now 2025-10-08 15:48:17,188 - INFO - VM name : ctest-vn2_vm1-83080481 2025-10-08 15:48:17,299 - DEBUG - VM ctest-vn2_vm1-83080481 ID is c2660dc8-7c52-4946-ba2c-e2be16af9cde 2025-10-08 15:48:17,299 - DEBUG - VM ctest-vn2_vm1-83080481 launched on Node an-jenkins-deploy-platform-ansible-os-4236-1 2025-10-08 15:48:17,440 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/c2660dc8-7c52-4946-ba2c-e2be16af9cde 2025-10-08 15:48:17,457 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine-interface/4ce3468b-6957-40f7-a17e-03af90bab406 2025-10-08 15:48:18,658 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=2.08 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.467 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.467/1.273/2.079/0.806 ms') 2025-10-08 15:48:18,658 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn2_vm1-83080481 passed 2025-10-08 15:48:18,739 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:48:18,739 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-83080481, IP 192.168.2.252, Port 22 2025-10-08 15:48:18,895 - DEBUG - VM ctest-vn2_vm1-83080481 is ready for SSH connections 2025-10-08 15:48:18,896 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:48:18,896 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.252, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.241, gateway password: c0ntrail123 2025-10-08 15:48:18,896 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.252 2025-10-08 15:48:41,504 - DEBUG - PING 192.168.2.252 (192.168.2.252) 56(84) bytes of data. 64 bytes from 192.168.2.252: icmp_req=1 ttl=63 time=8.13 ms 64 bytes from 192.168.2.252: icmp_req=2 ttl=63 time=1.36 ms 64 bytes from 192.168.2.252: icmp_req=3 ttl=63 time=1.30 ms --- 192.168.2.252 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 1.305/3.600/8.131/3.203 ms 2025-10-08 15:48:41,504 - INFO - Ping to IP 192.168.2.252 from VM ctest-vn1_vm1-37998096 passed 2025-10-08 15:48:44,882 - INFO - 10.0.0.61 2025-10-08 15:48:44,888 - INFO - agent:initializing 2025-10-08 15:48:44,888 - DEBUG - defaultdict(, {'10.0.0.61': {'agent': {'status': 'initializing', 'description': None}}}) 2025-10-08 15:48:44,888 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-08 15:48:49,980 - INFO - 10.0.0.61 2025-10-08 15:48:49,988 - INFO - agent:active 2025-10-08 15:48:49,989 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.61 2025-10-08 15:48:53,360 - INFO - 10.0.0.241 2025-10-08 15:48:53,365 - INFO - agent:initializing 2025-10-08 15:48:53,365 - DEBUG - defaultdict(, {'10.0.0.241': {'agent': {'status': 'initializing', 'description': None}}}) 2025-10-08 15:48:53,365 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-08 15:48:58,456 - INFO - 10.0.0.241 2025-10-08 15:48:58,462 - INFO - agent:active 2025-10-08 15:48:58,463 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.241 2025-10-08 15:48:59,129 - INFO - 10.0.0.133 2025-10-08 15:48:59,133 - INFO - control:initializing 2025-10-08 15:48:59,133 - DEBUG - defaultdict(, {'10.0.0.133': {'control': {'status': 'initializing', 'description': None}}}) 2025-10-08 15:48:59,133 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-08 15:49:04,264 - INFO - 10.0.0.133 2025-10-08 15:49:04,275 - INFO - control:active 2025-10-08 15:49:04,276 - INFO - Contrail services ['control'] are up on nodes 10.0.0.133 2025-10-08 15:49:04,663 - INFO - 10.0.0.50 2025-10-08 15:49:04,667 - INFO - control:initializing 2025-10-08 15:49:04,668 - DEBUG - defaultdict(, {'10.0.0.50': {'control': {'status': 'initializing', 'description': None}}}) 2025-10-08 15:49:04,668 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-08 15:49:09,769 - INFO - 10.0.0.50 2025-10-08 15:49:09,776 - INFO - control:active 2025-10-08 15:49:09,776 - INFO - Contrail services ['control'] are up on nodes 10.0.0.50 2025-10-08 15:49:10,229 - INFO - 10.0.0.242 2025-10-08 15:49:10,234 - INFO - control:initializing 2025-10-08 15:49:10,234 - DEBUG - defaultdict(, {'10.0.0.242': {'control': {'status': 'initializing', 'description': None}}}) 2025-10-08 15:49:10,234 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-08 15:49:15,338 - INFO - 10.0.0.242 2025-10-08 15:49:15,346 - INFO - control:active 2025-10-08 15:49:15,346 - INFO - Contrail services ['control'] are up on nodes 10.0.0.242 2025-10-08 15:49:16,277 - INFO - 10.0.0.133 2025-10-08 15:49:16,281 - INFO - api-server:initializing 2025-10-08 15:49:16,281 - DEBUG - defaultdict(, {'10.0.0.133': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-08 15:49:16,281 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-08 15:49:21,449 - INFO - 10.0.0.133 2025-10-08 15:49:21,458 - INFO - api-server:initializing 2025-10-08 15:49:21,459 - DEBUG - defaultdict(, {'10.0.0.133': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-08 15:49:21,459 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-10-08 15:49:26,582 - INFO - 10.0.0.133 2025-10-08 15:49:26,588 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2025-10-08 15:49:26,588 - DEBUG - defaultdict(, {'10.0.0.133': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2025-10-08 15:49:26,588 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-10-08 15:49:31,703 - INFO - 10.0.0.133 2025-10-08 15:49:31,708 - INFO - api-server:active 2025-10-08 15:49:31,709 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.133 2025-10-08 15:49:32,354 - INFO - 10.0.0.50 2025-10-08 15:49:32,360 - INFO - api-server:initializing 2025-10-08 15:49:32,360 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-08 15:49:32,360 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-08 15:49:37,449 - INFO - 10.0.0.50 2025-10-08 15:49:37,457 - INFO - api-server:initializing 2025-10-08 15:49:37,457 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-08 15:49:37,457 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-10-08 15:49:42,546 - INFO - 10.0.0.50 2025-10-08 15:49:42,552 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2025-10-08 15:49:42,552 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2025-10-08 15:49:42,552 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-10-08 15:49:47,642 - INFO - 10.0.0.50 2025-10-08 15:49:47,648 - INFO - api-server:active 2025-10-08 15:49:47,648 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.50 2025-10-08 15:49:48,422 - INFO - 10.0.0.242 2025-10-08 15:49:48,427 - INFO - api-server:initializing 2025-10-08 15:49:48,427 - DEBUG - defaultdict(, {'10.0.0.242': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-08 15:49:48,427 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-08 15:49:53,528 - INFO - 10.0.0.242 2025-10-08 15:49:53,535 - INFO - api-server:initializing 2025-10-08 15:49:53,535 - DEBUG - defaultdict(, {'10.0.0.242': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-08 15:49:53,535 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-10-08 15:49:58,635 - INFO - 10.0.0.242 2025-10-08 15:49:58,641 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2025-10-08 15:49:58,642 - DEBUG - defaultdict(, {'10.0.0.242': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2025-10-08 15:49:58,642 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-10-08 15:50:03,732 - INFO - 10.0.0.242 2025-10-08 15:50:03,738 - INFO - api-server:active 2025-10-08 15:50:03,738 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.242 2025-10-08 15:50:03,862 - INFO - 10.0.0.133 2025-10-08 15:50:03,862 - INFO - config 2025-10-08 15:50:03,868 - INFO - config-nodemgr:active 2025-10-08 15:50:03,872 - INFO - api-server:active 2025-10-08 15:50:03,876 - INFO - schema:backup 2025-10-08 15:50:03,879 - INFO - svc-monitor:backup 2025-10-08 15:50:03,882 - INFO - device-manager:backup 2025-10-08 15:50:03,882 - INFO - config-database 2025-10-08 15:50:03,882 - INFO - config-cassandra:active 2025-10-08 15:50:03,882 - INFO - config-zookeeper:active 2025-10-08 15:50:03,882 - INFO - config-rabbitmq:active 2025-10-08 15:50:03,883 - INFO - control 2025-10-08 15:50:03,887 - INFO - control-nodemgr:active 2025-10-08 15:50:03,893 - INFO - control:active 2025-10-08 15:50:03,894 - INFO - named:active 2025-10-08 15:50:03,900 - INFO - dns:active 2025-10-08 15:50:03,900 - INFO - analytics 2025-10-08 15:50:03,906 - INFO - analytics-nodemgr:active 2025-10-08 15:50:03,911 - INFO - analytics-api:active 2025-10-08 15:50:03,916 - INFO - collector:active 2025-10-08 15:50:03,916 - INFO - analytics-database 2025-10-08 15:50:03,916 - INFO - analytics-cassandra:active 2025-10-08 15:50:03,921 - INFO - analyticsdb-nodemgr:active 2025-10-08 15:50:03,926 - INFO - query-engine:active 2025-10-08 15:50:03,927 - INFO - webui 2025-10-08 15:50:03,927 - INFO - webui:active 2025-10-08 15:50:03,927 - INFO - webui-middleware:active 2025-10-08 15:50:03,927 - INFO - redis:active 2025-10-08 15:50:03,927 - INFO - analytics_snmp 2025-10-08 15:50:03,932 - INFO - snmp-collector:active 2025-10-08 15:50:03,932 - INFO - snmp-topology:active 2025-10-08 15:50:03,932 - INFO - analytics_alarm 2025-10-08 15:50:03,932 - INFO - alarmgen:active 2025-10-08 15:50:04,026 - INFO - 10.0.0.50 2025-10-08 15:50:04,026 - INFO - config 2025-10-08 15:50:04,033 - INFO - config-nodemgr:active 2025-10-08 15:50:04,039 - INFO - api-server:active 2025-10-08 15:50:04,044 - INFO - schema:active 2025-10-08 15:50:04,049 - INFO - svc-monitor:active 2025-10-08 15:50:04,053 - INFO - device-manager:active 2025-10-08 15:50:04,054 - INFO - config-database 2025-10-08 15:50:04,054 - INFO - config-cassandra:active 2025-10-08 15:50:04,054 - INFO - config-zookeeper:active 2025-10-08 15:50:04,054 - INFO - config-rabbitmq:active 2025-10-08 15:50:04,054 - INFO - control 2025-10-08 15:50:04,060 - INFO - control-nodemgr:active 2025-10-08 15:50:04,066 - INFO - control:active 2025-10-08 15:50:04,066 - INFO - named:active 2025-10-08 15:50:04,072 - INFO - dns:active 2025-10-08 15:50:04,072 - INFO - analytics 2025-10-08 15:50:04,078 - INFO - analytics-nodemgr:active 2025-10-08 15:50:04,083 - INFO - analytics-api:active 2025-10-08 15:50:04,088 - INFO - collector:active 2025-10-08 15:50:04,089 - INFO - analytics-database 2025-10-08 15:50:04,089 - INFO - analytics-cassandra:active 2025-10-08 15:50:04,094 - INFO - analyticsdb-nodemgr:active 2025-10-08 15:50:04,099 - INFO - query-engine:active 2025-10-08 15:50:04,100 - INFO - webui 2025-10-08 15:50:04,100 - INFO - webui:active 2025-10-08 15:50:04,100 - INFO - webui-middleware:active 2025-10-08 15:50:04,100 - INFO - redis:active 2025-10-08 15:50:04,100 - INFO - analytics_snmp 2025-10-08 15:50:04,104 - INFO - snmp-collector:active 2025-10-08 15:50:04,104 - INFO - snmp-topology:active 2025-10-08 15:50:04,104 - INFO - analytics_alarm 2025-10-08 15:50:04,104 - INFO - alarmgen:active 2025-10-08 15:50:04,204 - INFO - 10.0.0.242 2025-10-08 15:50:04,204 - INFO - config 2025-10-08 15:50:04,211 - INFO - config-nodemgr:active 2025-10-08 15:50:04,218 - INFO - api-server:active 2025-10-08 15:50:04,223 - INFO - schema:backup 2025-10-08 15:50:04,227 - INFO - svc-monitor:backup 2025-10-08 15:50:04,231 - INFO - device-manager:backup 2025-10-08 15:50:04,231 - INFO - config-database 2025-10-08 15:50:04,232 - INFO - config-cassandra:active 2025-10-08 15:50:04,232 - INFO - config-zookeeper:active 2025-10-08 15:50:04,232 - INFO - config-rabbitmq:active 2025-10-08 15:50:04,232 - INFO - control 2025-10-08 15:50:04,238 - INFO - control-nodemgr:active 2025-10-08 15:50:04,244 - INFO - control:active 2025-10-08 15:50:04,245 - INFO - named:active 2025-10-08 15:50:04,252 - INFO - dns:active 2025-10-08 15:50:04,253 - INFO - analytics 2025-10-08 15:50:04,263 - INFO - analytics-nodemgr:active 2025-10-08 15:50:04,272 - INFO - analytics-api:active 2025-10-08 15:50:04,280 - INFO - collector:active 2025-10-08 15:50:04,280 - INFO - analytics-database 2025-10-08 15:50:04,280 - INFO - analytics-cassandra:active 2025-10-08 15:50:04,287 - INFO - analyticsdb-nodemgr:active 2025-10-08 15:50:04,297 - INFO - query-engine:active 2025-10-08 15:50:04,297 - INFO - webui 2025-10-08 15:50:04,297 - INFO - webui:active 2025-10-08 15:50:04,297 - INFO - webui-middleware:active 2025-10-08 15:50:04,297 - INFO - redis:active 2025-10-08 15:50:04,297 - INFO - analytics_snmp 2025-10-08 15:50:04,303 - INFO - snmp-collector:active 2025-10-08 15:50:04,303 - INFO - snmp-topology:active 2025-10-08 15:50:04,303 - INFO - analytics_alarm 2025-10-08 15:50:04,303 - INFO - alarmgen:active 2025-10-08 15:50:04,396 - INFO - 10.0.0.61 2025-10-08 15:50:04,397 - INFO - vrouter 2025-10-08 15:50:04,403 - INFO - vrouter-nodemgr:active 2025-10-08 15:50:04,411 - INFO - agent:active 2025-10-08 15:50:04,502 - INFO - 10.0.0.241 2025-10-08 15:50:04,502 - INFO - vrouter 2025-10-08 15:50:04,509 - INFO - vrouter-nodemgr:active 2025-10-08 15:50:04,515 - INFO - agent:active 2025-10-08 15:50:04,515 - INFO - All the contrail services are up on all nodes 2025-10-08 15:50:04,516 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4236-1?flat 2025-10-08 15:50:04,956 - INFO - an-jenkins-deploy-platform-ansible-os-4236-1:contrail-vrouter-agent module connection to ['10.20.0.200:5269'] servers UP 2025-10-08 15:50:04,957 - INFO - an-jenkins-deploy-platform-ansible-os-4236-1 connected to xmpp 10.20.0.200 2025-10-08 15:50:04,957 - WARNING - an-jenkins-deploy-platform-ansible-os-4236-1:contrail-vrouter-agent module connection to ['10.20.0.252:5269'] servers NOT UP 2025-10-08 15:50:04,957 - INFO - an-jenkins-deploy-platform-ansible-os-4236-1:contrail-vrouter-agent module connection to ['10.20.0.249:5269'] servers UP 2025-10-08 15:50:04,957 - INFO - an-jenkins-deploy-platform-ansible-os-4236-1 connected to xmpp 10.20.0.249 2025-10-08 15:50:04,957 - INFO - an-jenkins-deploy-platform-ansible-os-4236-1:contrail-vrouter-agent module connection to ['10.20.0.200:53'] servers UP 2025-10-08 15:50:04,957 - INFO - an-jenkins-deploy-platform-ansible-os-4236-1 connected to dns 10.20.0.200 2025-10-08 15:50:04,957 - INFO - an-jenkins-deploy-platform-ansible-os-4236-1:contrail-vrouter-agent module connection to ['10.20.0.252:53'] servers UP 2025-10-08 15:50:04,957 - INFO - an-jenkins-deploy-platform-ansible-os-4236-1 connected to dns 10.20.0.252 2025-10-08 15:50:04,957 - WARNING - an-jenkins-deploy-platform-ansible-os-4236-1:contrail-vrouter-agent module connection to ['10.20.0.249:53'] servers NOT UP 2025-10-08 15:50:04,957 - WARNING - an-jenkins-deploy-platform-ansible-os-4236-1:contrail-vrouter-agent module connection to ['10.0.0.133:8086'] servers NOT UP 2025-10-08 15:50:04,957 - INFO - an-jenkins-deploy-platform-ansible-os-4236-1:contrail-vrouter-agent module connection to ['10.0.0.50:8086'] servers UP 2025-10-08 15:50:04,957 - INFO - an-jenkins-deploy-platform-ansible-os-4236-1 connected to collector 10.0.0.50 2025-10-08 15:50:04,957 - WARNING - an-jenkins-deploy-platform-ansible-os-4236-1:contrail-vrouter-agent module connection to ['10.0.0.242:8086'] servers NOT UP 2025-10-08 15:50:04,957 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4236-1?flat 2025-10-08 15:50:05,409 - INFO - cn-jenkins-deploy-platform-ansible-os-4236-1:contrail-control module connection to 10.0.0.133:9041 servers UP 2025-10-08 15:50:05,410 - INFO - cn-jenkins-deploy-platform-ansible-os-4236-1:contrail-control module connection to 10.0.0.133:5673 servers UP 2025-10-08 15:50:05,410 - INFO - cn-jenkins-deploy-platform-ansible-os-4236-1:contrail-control module connection to ['10.0.0.133:8086'] servers UP 2025-10-08 15:50:05,411 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4236-2?flat 2025-10-08 15:50:05,873 - INFO - cn-jenkins-deploy-platform-ansible-os-4236-2:contrail-control module connection to 10.0.0.133:9041 servers UP 2025-10-08 15:50:05,873 - INFO - cn-jenkins-deploy-platform-ansible-os-4236-2:contrail-control module connection to 10.0.0.133:5673 servers UP 2025-10-08 15:50:05,873 - INFO - cn-jenkins-deploy-platform-ansible-os-4236-2:contrail-control module connection to ['10.0.0.133:8086'] servers UP 2025-10-08 15:50:05,873 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4236-3?flat 2025-10-08 15:50:05,896 - INFO - cn-jenkins-deploy-platform-ansible-os-4236-3:contrail-control module connection to 10.0.0.133:9041 servers UP 2025-10-08 15:50:05,896 - INFO - cn-jenkins-deploy-platform-ansible-os-4236-3:contrail-control module connection to 10.0.0.133:5673 servers UP 2025-10-08 15:50:05,896 - INFO - cn-jenkins-deploy-platform-ansible-os-4236-3:contrail-control module connection to ['10.0.0.133:8086'] servers UP 2025-10-08 15:50:05,896 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4236-1.?flat 2025-10-08 15:50:05,923 - DEBUG - contrail-api is running 2025-10-08 15:50:05,923 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4236-2.?flat 2025-10-08 15:50:05,950 - DEBUG - contrail-api is running 2025-10-08 15:50:05,950 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4236-3.?flat 2025-10-08 15:50:05,976 - DEBUG - contrail-api is running 2025-10-08 15:50:06,280 - INFO - Created VN ctest-vn3-57172718, UUID :333d2cdb-1a71-488c-9642-c5d21dfbf60f 2025-10-08 15:50:06,830 - DEBUG - Requesting: http://10.0.0.133:8082/domains 2025-10-08 15:50:06,893 - DEBUG - Requesting: http://10.0.0.133:8082/domain/adc52b80-2ed1-485c-90d4-83e78c8e1400 2025-10-08 15:50:06,928 - DEBUG - Requesting: http://10.0.0.133:8082/project/2d0f9370-44fd-4f7b-99c3-73540ae4a690 2025-10-08 15:50:07,030 - DEBUG - Requesting: http://10.0.0.133:8082/virtual-network/6105f376-caef-48e9-9074-6e6879ce8ae4 2025-10-08 15:50:07,044 - DEBUG - Requesting: http://10.0.0.133:8082/virtual-network/6105f376-caef-48e9-9074-6e6879ce8ae4 2025-10-08 15:50:07,059 - DEBUG - Requesting: http://10.0.0.133:8082/routing-instance/d733f0ef-873d-40f1-ba17-4e5de4976db9 2025-10-08 15:50:07,069 - DEBUG - Requesting: http://10.0.0.133:8082/routing-instance/d733f0ef-873d-40f1-ba17-4e5de4976db9 2025-10-08 15:50:07,078 - DEBUG - Requesting: http://10.0.0.133:8082/route-target/472aa6bb-0a53-4234-a8fe-768b270ce29f 2025-10-08 15:50:07,088 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-08 15:50:07,088 - DEBUG - Requesting: http://10.0.0.133:8082/virtual-network/6105f376-caef-48e9-9074-6e6879ce8ae4 2025-10-08 15:50:07,100 - DEBUG - Requesting: http://10.0.0.133:8082/routing-instance/d733f0ef-873d-40f1-ba17-4e5de4976db9 2025-10-08 15:50:07,121 - INFO - Verified VN network id 14 for VN 6105f376-caef-48e9-9074-6e6879ce8ae4 2025-10-08 15:50:07,121 - INFO - Verifications in API Server for VN ctest-vn1-31040416 passed 2025-10-08 15:50:07,121 - DEBUG - Requesting: http://10.0.0.133:8082/virtual-network/6105f376-caef-48e9-9074-6e6879ce8ae4 2025-10-08 15:50:07,133 - DEBUG - Requesting: http://10.0.0.133:8082/routing-instance/d733f0ef-873d-40f1-ba17-4e5de4976db9 2025-10-08 15:50:07,141 - DEBUG - Requesting: http://10.0.0.133:8082/routing-instance/d733f0ef-873d-40f1-ba17-4e5de4976db9 2025-10-08 15:50:07,150 - DEBUG - Requesting: http://10.0.0.133:8082/route-target/472aa6bb-0a53-4234-a8fe-768b270ce29f 2025-10-08 15:50:07,164 - DEBUG - Control-node 10.0.0.133 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-24481103', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '6991261688098932969', 'uuid-lslong': '10409066033759292132'}, 'enable': 'true', 'created': '2025-10-08T15:47:10', 'last-modified': '2025-10-08T15:47:11', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:07.463318'} 2025-10-08 15:50:07,171 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-08 15:50:07,178 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-24481103', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '6991261688098932969', 'uuid-lslong': '10409066033759292132'}, 'enable': 'true', 'created': '2025-10-08T15:47:10', 'last-modified': '2025-10-08T15:47:11', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:02.104509'} 2025-10-08 15:50:07,185 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-08 15:50:07,190 - DEBUG - Control-node 10.0.0.242 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-24481103', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '6991261688098932969', 'uuid-lslong': '10409066033759292132'}, 'enable': 'true', 'created': '2025-10-08T15:47:10', 'last-modified': '2025-10-08T15:47:11', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:56.543097'} 2025-10-08 15:50:07,197 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-08 15:50:07,197 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-31040416 passed 2025-10-08 15:50:07,198 - DEBUG - ====Verifying policy data for ctest-vn1-31040416 in API_Server ====== 2025-10-08 15:50:07,198 - DEBUG - Requesting: http://10.0.0.133:8082/domains 2025-10-08 15:50:07,205 - DEBUG - Requesting: http://10.0.0.133:8082/domain/adc52b80-2ed1-485c-90d4-83e78c8e1400 2025-10-08 15:50:07,221 - DEBUG - Requesting: http://10.0.0.133:8082/project/2d0f9370-44fd-4f7b-99c3-73540ae4a690 2025-10-08 15:50:07,328 - DEBUG - Requesting: http://10.0.0.133:8082/virtual-network/6105f376-caef-48e9-9074-6e6879ce8ae4 2025-10-08 15:50:07,340 - DEBUG - ==>Verifying data for policy with id: 332a8ea7-b397-49df-bf42-7291ebf7c4e8, fqn: ['default-domain', 'ctest-TestBasicVMVN0-24481103', 'policy1'] 2025-10-08 15:50:07,340 - INFO - VN ctest-vn1-31040416 Policy verification: verify_vn_policy_in_api_server, status: True 2025-10-08 15:50:07,340 - DEBUG - Verifying the vn in opserver 2025-10-08 15:50:07,340 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 virtual network link through opserver 10.0.0.133 2025-10-08 15:50:07,340 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-networks 2025-10-08 15:50:07,355 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416', 'href': 'http://10.0.0.133:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416?flat'} 2025-10-08 15:50:07,355 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 virtual network link through opserver 10.0.0.50 2025-10-08 15:50:07,355 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2025-10-08 15:50:07,427 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416?flat'} 2025-10-08 15:50:07,427 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 virtual network link through opserver 10.0.0.242 2025-10-08 15:50:07,427 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-networks 2025-10-08 15:50:07,497 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416', 'href': 'http://10.0.0.242:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416?flat'} 2025-10-08 15:50:07,497 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 is found in opserver 2025-10-08 15:50:07,510 - DEBUG - VRF ids for VN ctest-vn1-31040416: {'10.0.0.241': '2'} 2025-10-08 15:50:09,177 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4236-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4236-2) 2025-10-08 15:50:09,238 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-10-08 15:50:14,351 - DEBUG - VM is in ACTIVE state now 2025-10-08 15:50:14,351 - INFO - VM name : ctest-vn1_vm2-02553484 2025-10-08 15:50:14,453 - DEBUG - VM ctest-vn1_vm2-02553484 ID is a4f84b86-49e3-405a-9012-9136204d94f4 2025-10-08 15:50:14,560 - DEBUG - VM is in ACTIVE state now 2025-10-08 15:50:14,560 - DEBUG - VM ctest-vn1_vm2-02553484 launched on Node an-jenkins-deploy-platform-ansible-os-4236-2 2025-10-08 15:50:14,667 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/a4f84b86-49e3-405a-9012-9136204d94f4 2025-10-08 15:50:14,721 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/a4f84b86-49e3-405a-9012-9136204d94f4 2025-10-08 15:50:14,732 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine-interface/d7365475-5158-40e7-a585-6f1d89711290 2025-10-08 15:50:14,748 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/a4f84b86-49e3-405a-9012-9136204d94f4 2025-10-08 15:50:14,768 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine-interface/d7365475-5158-40e7-a585-6f1d89711290 2025-10-08 15:50:14,784 - DEBUG - Requesting: http://10.0.0.242:8082/instance-ip/28756d01-7a45-4105-baf1-006205b760ab 2025-10-08 15:50:14,798 - DEBUG - Verifying in api server 10.0.0.242 2025-10-08 15:50:14,798 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/a4f84b86-49e3-405a-9012-9136204d94f4 2025-10-08 15:50:14,808 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine-interface/d7365475-5158-40e7-a585-6f1d89711290 2025-10-08 15:50:14,822 - DEBUG - Requesting: http://10.0.0.242:8082/instance-ip/28756d01-7a45-4105-baf1-006205b760ab 2025-10-08 15:50:14,834 - INFO - VM ctest-vn1_vm2-02553484 verfication in all API Servers passed 2025-10-08 15:50:14,855 - DEBUG - VM ctest-vn1_vm2-02553484 Tap interface: {'index': '4', 'name': 'tapd7365475-51', 'uuid': 'd7365475-5158-40e7-a585-6f1d89711290', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416:ctest-vn1-31040416', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416', 'vm_uuid': 'a4f84b86-49e3-405a-9012-9136204d94f4', 'vm_name': 'ctest-vn1_vm2-02553484', 'ip_addr': '192.168.1.251', 'mac_addr': '02:d7:36:54:75:51', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '165', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-24481103:d7365475-5158-40e7-a585-6f1d89711290', 'sg_uuid_list': ['cbf6bc58-853e-4673-ab9d-9aaf1cab7a76'], 'static_route_list': None, 'vm_project_uuid': '2d0f9370-44fd-4f7b-99c3-73540ae4a690', 'admin_state': 'Enabled', 'flow_key_idx': '35', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-08 15:50:14,855 - DEBUG - Agent 10.0.0.241 vrf name: default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416:ctest-vn1-31040416 2025-10-08 15:50:14,860 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416:ctest-vn1-31040416', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416', 'table_label': '-1', 'vxlan_id': '14', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.242:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-10-08 15:50:14,956 - DEBUG - VM ctest-vn1_vm2-02553484 : Tap interface tapd7365475-51 is set to Active 2025-10-08 15:50:14,956 - DEBUG - Tap interface tapd7365475-51 detail : {'index': '4', 'name': 'tapd7365475-51', 'uuid': 'd7365475-5158-40e7-a585-6f1d89711290', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416:ctest-vn1-31040416', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416', 'vm_uuid': 'a4f84b86-49e3-405a-9012-9136204d94f4', 'vm_name': 'ctest-vn1_vm2-02553484', 'ip_addr': '192.168.1.251', 'mac_addr': '02:d7:36:54:75:51', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '165', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-24481103:d7365475-5158-40e7-a585-6f1d89711290', 'sg_uuid_list': ['cbf6bc58-853e-4673-ab9d-9aaf1cab7a76'], 'static_route_list': None, 'vm_project_uuid': '2d0f9370-44fd-4f7b-99c3-73540ae4a690', 'admin_state': 'Enabled', 'flow_key_idx': '35', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-08 15:50:15,070 - DEBUG - Starting Layer 2 verification in Agent 2025-10-08 15:50:15,091 - DEBUG - Layer 2 path is seen for VM MAC 02:d7:36:54:75:51 in agent 10.0.0.241 2025-10-08 15:50:15,092 - DEBUG - Active layer 2 route in agent is present for VMI tapd7365475-51 2025-10-08 15:50:15,092 - DEBUG - L2 label(34) matches bw route table and interface table 2025-10-08 15:50:15,110 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 is found in Agent of node 10.0.0.241 2025-10-08 15:50:15,213 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 is consistent in agent 10.0.0.241 2025-10-08 15:50:15,213 - DEBUG - Route for VM IP ['192.168.1.251'] is consistent in agent 10.0.0.241 2025-10-08 15:50:15,213 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 verification for VM ctest-vn1_vm2-02553484 in Agent 10.0.0.241 passed 2025-10-08 15:50:15,334 - DEBUG - Starting all layer 2 verification in agent 10.0.0.241 2025-10-08 15:50:15,354 - DEBUG - Route for VM MAC 02:d7:36:54:75:51 is consistent in agent 10.0.0.241 2025-10-08 15:50:18,570 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1016ms') 2025-10-08 15:50:18,570 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-02553484 failed! 2025-10-08 15:50:18,594 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 is 192.168.1.254 and allocation pool is NOT set 2025-10-08 15:50:22,665 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1007ms') 2025-10-08 15:50:22,665 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-02553484 failed! 2025-10-08 15:50:22,683 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 is 192.168.1.254 and allocation pool is NOT set 2025-10-08 15:50:26,762 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-10-08 15:50:26,762 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-02553484 failed! 2025-10-08 15:50:26,781 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 is 192.168.1.254 and allocation pool is NOT set 2025-10-08 15:50:30,857 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2025-10-08 15:50:30,857 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-02553484 failed! 2025-10-08 15:50:30,874 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 is 192.168.1.254 and allocation pool is NOT set 2025-10-08 15:50:34,955 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms') 2025-10-08 15:50:34,955 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-02553484 failed! 2025-10-08 15:50:34,976 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 is 192.168.1.254 and allocation pool is NOT set 2025-10-08 15:50:39,050 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1006ms') 2025-10-08 15:50:39,050 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-02553484 failed! 2025-10-08 15:50:39,071 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 is 192.168.1.254 and allocation pool is NOT set 2025-10-08 15:50:43,146 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms') 2025-10-08 15:50:43,146 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-02553484 failed! 2025-10-08 15:50:43,162 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 is 192.168.1.254 and allocation pool is NOT set 2025-10-08 15:50:47,238 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2025-10-08 15:50:47,239 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-02553484 failed! 2025-10-08 15:50:47,256 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 is 192.168.1.254 and allocation pool is NOT set 2025-10-08 15:50:51,332 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=5.05 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 1 received, 50% packet loss, time 1014ms\r\nrtt min/avg/max/mdev = 5.048/5.048/5.048/0.000 ms') 2025-10-08 15:50:51,332 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-02553484 passed 2025-10-08 15:50:51,332 - INFO - VM ctest-vn1_vm2-02553484 verifications in Compute nodes passed 2025-10-08 15:50:51,568 - DEBUG - Validated VM route 192.168.1.251 in vrouter of 10.0.0.241 2025-10-08 15:50:51,576 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-10-08 15:50:51,576 - INFO - Validated routes of VM ctest-vn1_vm2-02553484 in all vrouters 2025-10-08 15:50:52,010 - DEBUG - Starting all layer2 verification in 10.0.0.133 Control Node 2025-10-08 15:50:52,147 - DEBUG - Layer2 route found for VM MAC 02:d7:36:54:75:51 in Control-node 10.0.0.133 2025-10-08 15:50:52,147 - DEBUG - L2 Label for VM ctest-vn1_vm2-02553484 same between Control-node 10.0.0.133 and Agent, Expected: 34, Seen: 34 2025-10-08 15:50:52,147 - DEBUG - Starting all layer2 verification in 10.0.0.242 Control Node 2025-10-08 15:50:52,277 - DEBUG - Layer2 route found for VM MAC 02:d7:36:54:75:51 in Control-node 10.0.0.242 2025-10-08 15:50:52,277 - DEBUG - L2 Label for VM ctest-vn1_vm2-02553484 same between Control-node 10.0.0.242 and Agent, Expected: 34, Seen: 34 2025-10-08 15:50:52,277 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-02553484 passed 2025-10-08 15:50:52,278 - DEBUG - Verifying the vm in opserver 2025-10-08 15:50:52,278 - DEBUG - Verifying in collector 10.0.0.242 ... 2025-10-08 15:50:52,278 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-machine/a4f84b86-49e3-405a-9012-9136204d94f4?flat 2025-10-08 15:50:52,364 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-24481103:d7365475-5158-40e7-a585-6f1d89711290?flat 2025-10-08 15:50:52,387 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-24481103:d7365475-5158-40e7-a585-6f1d89711290?flat 2025-10-08 15:50:52,412 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-machine/a4f84b86-49e3-405a-9012-9136204d94f4?flat 2025-10-08 15:50:52,428 - DEBUG - Verifying vm in vn uve 2025-10-08 15:50:52,428 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-24481103:d7365475-5158-40e7-a585-6f1d89711290?flat 2025-10-08 15:50:52,452 - DEBUG - VM uve shows interface as {'ip6_address': '::', 'admin_state': True, 'rx_vlan': 65535, 'port_mirror_enabled': False, 'gateway': '192.168.1.254', 'active': True, 'vn_uuid': '6105f376-caef-48e9-9074-6e6879ce8ae4', 'l2_active': True, 'vm_name': 'ctest-vn1_vm2-02553484', 'vm_uuid': 'a4f84b86-49e3-405a-9012-9136204d94f4', 'is_health_check_active': True, 'mac_address': '02:d7:36:54:75:51', 'label': 30, 'vhostuser_mode': 0, 'tx_vlan': 65535, 'fixed_ip4_list': ['192.168.1.251'], 'ip_address': '192.168.1.251', 'uuid': 'd7365475-5158-40e7-a585-6f1d89711290', 'ip6_active': False, '__T': 1759938623963554, 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416', 'ip4_active': True} 2025-10-08 15:50:52,453 - DEBUG - VM uve shows ip address as ['192.168.1.251', '::'] 2025-10-08 15:50:52,453 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 2025-10-08 15:50:52,453 - INFO - Verifying through opserver in 10.0.0.133 2025-10-08 15:50:52,453 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416?flat 2025-10-08 15:50:52,480 - DEBUG - expected vm list a4f84b86-49e3-405a-9012-9136204d94f4 2025-10-08 15:50:52,480 - DEBUG - Extracted vm list ['a4f84b86-49e3-405a-9012-9136204d94f4'] 2025-10-08 15:50:52,480 - INFO - VM a4f84b86-49e3-405a-9012-9136204d94f4 is present in default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 2025-10-08 15:50:52,480 - DEBUG - Verifying vm in vrouter uve 2025-10-08 15:50:52,480 - DEBUG - Getting info from collector 10.0.0.242.. 2025-10-08 15:50:52,480 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-machine/a4f84b86-49e3-405a-9012-9136204d94f4?flat 2025-10-08 15:50:52,559 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4236-2'] 2025-10-08 15:50:52,559 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4236-2:Compute:contrail-vrouter-agent:0?flat 2025-10-08 15:50:52,616 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4236-2?flat 2025-10-08 15:50:52,653 - DEBUG - VM a4f84b86-49e3-405a-9012-9136204d94f4 is present in vrouter an-jenkins-deploy-platform-ansible-os-4236-2 uve 2025-10-08 15:50:52,653 - DEBUG - Validated that VM ctest-vn1_vm2-02553484 is in Vrouter an-jenkins-deploy-platform-ansible-os-4236-2 UVE 2025-10-08 15:50:52,653 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-10-08 15:50:52,663 - DEBUG - Expected tap interface of VM uuid a4f84b86-49e3-405a-9012-9136204d94f4 is default-domain:ctest-TestBasicVMVN0-24481103:d7365475-5158-40e7-a585-6f1d89711290 2025-10-08 15:50:52,663 - DEBUG - Expected VN of VM uuid a4f84b86-49e3-405a-9012-9136204d94f4 is default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 2025-10-08 15:50:52,663 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4236-2:Compute:contrail-vrouter-agent:0?flat 2025-10-08 15:50:52,725 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4236-2?flat 2025-10-08 15:50:52,763 - DEBUG - VM a4f84b86-49e3-405a-9012-9136204d94f4 is present in vrouter an-jenkins-deploy-platform-ansible-os-4236-2 uve 2025-10-08 15:50:52,763 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-24481103:d7365475-5158-40e7-a585-6f1d89711290 of vm a4f84b86-49e3-405a-9012-9136204d94f4 is present in vrouter an-jenkins-deploy-platform-ansible-os-4236-2 uve 2025-10-08 15:50:52,763 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 of vm a4f84b86-49e3-405a-9012-9136204d94f4 is present in vrouter an-jenkins-deploy-platform-ansible-os-4236-2 uve 2025-10-08 15:50:52,763 - DEBUG - Verifying the a4f84b86-49e3-405a-9012-9136204d94f4 virtual network link through opserver 10.0.0.133 2025-10-08 15:50:52,764 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-machines 2025-10-08 15:50:52,780 - DEBUG - VM link and name as {'name': 'a4f84b86-49e3-405a-9012-9136204d94f4', 'href': 'http://10.0.0.133:8081/analytics/uves/virtual-machine/a4f84b86-49e3-405a-9012-9136204d94f4?flat'} 2025-10-08 15:50:52,781 - DEBUG - Verifying the a4f84b86-49e3-405a-9012-9136204d94f4 virtual network link through opserver 10.0.0.50 2025-10-08 15:50:52,781 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2025-10-08 15:50:52,859 - DEBUG - VM link and name as {'name': 'a4f84b86-49e3-405a-9012-9136204d94f4', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-machine/a4f84b86-49e3-405a-9012-9136204d94f4?flat'} 2025-10-08 15:50:52,859 - DEBUG - Verifying the a4f84b86-49e3-405a-9012-9136204d94f4 virtual network link through opserver 10.0.0.242 2025-10-08 15:50:52,859 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-machines 2025-10-08 15:50:52,876 - DEBUG - VM link and name as {'name': 'a4f84b86-49e3-405a-9012-9136204d94f4', 'href': 'http://10.0.0.242:8081/analytics/uves/virtual-machine/a4f84b86-49e3-405a-9012-9136204d94f4?flat'} 2025-10-08 15:50:52,876 - INFO - VM ctest-vn1_vm2-02553484 validations in Opserver passed 2025-10-08 15:50:54,308 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4236-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4236-1) 2025-10-08 15:50:54,376 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-10-08 15:50:59,523 - DEBUG - VM is in ACTIVE state now 2025-10-08 15:50:59,523 - INFO - VM name : ctest-vn2_vm2-55589463 2025-10-08 15:50:59,633 - DEBUG - VM ctest-vn2_vm2-55589463 ID is e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c 2025-10-08 15:50:59,728 - DEBUG - VM is in ACTIVE state now 2025-10-08 15:50:59,729 - DEBUG - VM ctest-vn2_vm2-55589463 launched on Node an-jenkins-deploy-platform-ansible-os-4236-1 2025-10-08 15:50:59,827 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c 2025-10-08 15:50:59,841 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c 2025-10-08 15:50:59,850 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine-interface/0a4236a8-669e-4397-ba5d-68ed5b40da96 2025-10-08 15:50:59,861 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c 2025-10-08 15:50:59,870 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine-interface/0a4236a8-669e-4397-ba5d-68ed5b40da96 2025-10-08 15:50:59,883 - DEBUG - Requesting: http://10.0.0.242:8082/instance-ip/b5e46e00-e6b2-4d32-8962-88fa85492b54 2025-10-08 15:50:59,895 - DEBUG - Verifying in api server 10.0.0.242 2025-10-08 15:50:59,895 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c 2025-10-08 15:50:59,903 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine-interface/0a4236a8-669e-4397-ba5d-68ed5b40da96 2025-10-08 15:50:59,914 - DEBUG - Requesting: http://10.0.0.242:8082/instance-ip/b5e46e00-e6b2-4d32-8962-88fa85492b54 2025-10-08 15:50:59,924 - INFO - VM ctest-vn2_vm2-55589463 verfication in all API Servers passed 2025-10-08 15:50:59,948 - DEBUG - VM ctest-vn2_vm2-55589463 Tap interface: {'index': '4', 'name': 'tap0a4236a8-66', 'uuid': '0a4236a8-669e-4397-ba5d-68ed5b40da96', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607:ctest-vn2-11998607', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607', 'vm_uuid': 'e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c', 'vm_name': 'ctest-vn2_vm2-55589463', 'ip_addr': '192.168.2.251', 'mac_addr': '02:0a:42:36:a8:66', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '131', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-24481103:0a4236a8-669e-4397-ba5d-68ed5b40da96', 'sg_uuid_list': ['cbf6bc58-853e-4673-ab9d-9aaf1cab7a76'], 'static_route_list': None, 'vm_project_uuid': '2d0f9370-44fd-4f7b-99c3-73540ae4a690', 'admin_state': 'Enabled', 'flow_key_idx': '37', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-08 15:50:59,948 - DEBUG - Agent 10.0.0.61 vrf name: default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607:ctest-vn2-11998607 2025-10-08 15:50:59,954 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607:ctest-vn2-11998607', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607', 'table_label': '-1', 'vxlan_id': '15', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.241:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-10-08 15:51:00,046 - DEBUG - VM ctest-vn2_vm2-55589463 : Tap interface tap0a4236a8-66 is set to Active 2025-10-08 15:51:00,047 - DEBUG - Tap interface tap0a4236a8-66 detail : {'index': '4', 'name': 'tap0a4236a8-66', 'uuid': '0a4236a8-669e-4397-ba5d-68ed5b40da96', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607:ctest-vn2-11998607', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607', 'vm_uuid': 'e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c', 'vm_name': 'ctest-vn2_vm2-55589463', 'ip_addr': '192.168.2.251', 'mac_addr': '02:0a:42:36:a8:66', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '131', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-24481103:0a4236a8-669e-4397-ba5d-68ed5b40da96', 'sg_uuid_list': ['cbf6bc58-853e-4673-ab9d-9aaf1cab7a76'], 'static_route_list': None, 'vm_project_uuid': '2d0f9370-44fd-4f7b-99c3-73540ae4a690', 'admin_state': 'Enabled', 'flow_key_idx': '37', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-08 15:51:00,172 - DEBUG - Starting Layer 2 verification in Agent 2025-10-08 15:51:00,188 - DEBUG - Layer 2 path is seen for VM MAC 02:0a:42:36:a8:66 in agent 10.0.0.61 2025-10-08 15:51:00,188 - DEBUG - Active layer 2 route in agent is present for VMI tap0a4236a8-66 2025-10-08 15:51:00,188 - DEBUG - L2 label(34) matches bw route table and interface table 2025-10-08 15:51:00,196 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 is found in Agent of node 10.0.0.61 2025-10-08 15:51:00,340 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 is consistent in agent 10.0.0.61 2025-10-08 15:51:00,340 - DEBUG - Route for VM IP ['192.168.2.251'] is consistent in agent 10.0.0.61 2025-10-08 15:51:00,341 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 verification for VM ctest-vn2_vm2-55589463 in Agent 10.0.0.61 passed 2025-10-08 15:51:00,454 - DEBUG - Starting all layer 2 verification in agent 10.0.0.61 2025-10-08 15:51:00,469 - DEBUG - Route for VM MAC 02:0a:42:36:a8:66 is consistent in agent 10.0.0.61 2025-10-08 15:51:03,681 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1026ms') 2025-10-08 15:51:03,681 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-55589463 failed! 2025-10-08 15:51:03,703 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 is 192.168.2.254 and allocation pool is NOT set 2025-10-08 15:51:07,776 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1006ms') 2025-10-08 15:51:07,777 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-55589463 failed! 2025-10-08 15:51:07,799 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 is 192.168.2.254 and allocation pool is NOT set 2025-10-08 15:51:11,897 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1035ms') 2025-10-08 15:51:11,898 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-55589463 failed! 2025-10-08 15:51:11,914 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 is 192.168.2.254 and allocation pool is NOT set 2025-10-08 15:51:15,999 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-10-08 15:51:16,000 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-55589463 failed! 2025-10-08 15:51:16,019 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 is 192.168.2.254 and allocation pool is NOT set 2025-10-08 15:51:20,092 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-10-08 15:51:20,092 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-55589463 failed! 2025-10-08 15:51:20,113 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 is 192.168.2.254 and allocation pool is NOT set 2025-10-08 15:51:24,190 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2025-10-08 15:51:24,190 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-55589463 failed! 2025-10-08 15:51:24,214 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 is 192.168.2.254 and allocation pool is NOT set 2025-10-08 15:51:28,290 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2025-10-08 15:51:28,290 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-55589463 failed! 2025-10-08 15:51:28,310 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 is 192.168.2.254 and allocation pool is NOT set 2025-10-08 15:51:32,383 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2025-10-08 15:51:32,383 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-55589463 failed! 2025-10-08 15:51:32,399 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 is 192.168.2.254 and allocation pool is NOT set 2025-10-08 15:51:34,460 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=5.26 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=3.10 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 3.101/4.179/5.257/1.078 ms') 2025-10-08 15:51:34,460 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-55589463 passed 2025-10-08 15:51:34,461 - INFO - VM ctest-vn2_vm2-55589463 verifications in Compute nodes passed 2025-10-08 15:51:34,695 - DEBUG - Validated VM route 192.168.2.251 in vrouter of 10.0.0.61 2025-10-08 15:51:34,703 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-10-08 15:51:34,703 - INFO - Validated routes of VM ctest-vn2_vm2-55589463 in all vrouters 2025-10-08 15:51:35,150 - DEBUG - Starting all layer2 verification in 10.0.0.133 Control Node 2025-10-08 15:51:35,293 - DEBUG - Layer2 route found for VM MAC 02:0a:42:36:a8:66 in Control-node 10.0.0.133 2025-10-08 15:51:35,293 - DEBUG - L2 Label for VM ctest-vn2_vm2-55589463 same between Control-node 10.0.0.133 and Agent, Expected: 34, Seen: 34 2025-10-08 15:51:35,293 - DEBUG - Starting all layer2 verification in 10.0.0.242 Control Node 2025-10-08 15:51:35,425 - DEBUG - Layer2 route found for VM MAC 02:0a:42:36:a8:66 in Control-node 10.0.0.242 2025-10-08 15:51:35,425 - DEBUG - L2 Label for VM ctest-vn2_vm2-55589463 same between Control-node 10.0.0.242 and Agent, Expected: 34, Seen: 34 2025-10-08 15:51:35,425 - INFO - Verification in Control-nodes for VM ctest-vn2_vm2-55589463 passed 2025-10-08 15:51:35,425 - DEBUG - Verifying the vm in opserver 2025-10-08 15:51:35,425 - DEBUG - Verifying in collector 10.0.0.242 ... 2025-10-08 15:51:35,425 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-machine/e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c?flat 2025-10-08 15:51:35,445 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-24481103:0a4236a8-669e-4397-ba5d-68ed5b40da96?flat 2025-10-08 15:51:35,468 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-24481103:0a4236a8-669e-4397-ba5d-68ed5b40da96?flat 2025-10-08 15:51:35,489 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-machine/e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c?flat 2025-10-08 15:51:35,508 - DEBUG - Verifying vm in vn uve 2025-10-08 15:51:35,508 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-24481103:0a4236a8-669e-4397-ba5d-68ed5b40da96?flat 2025-10-08 15:51:35,534 - DEBUG - VM uve shows interface as {'vm_uuid': 'e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c', 'fixed_ip4_list': ['192.168.2.251'], 'mac_address': '02:0a:42:36:a8:66', 'admin_state': True, 'l2_active': True, 'rx_vlan': 65535, 'tx_vlan': 65535, 'gateway': '192.168.2.254', 'port_mirror_enabled': False, 'ip4_active': True, 'ip_address': '192.168.2.251', 'vn_uuid': 'd92c3b74-c648-4bef-8568-e08ca301bc00', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607', 'vhostuser_mode': 0, 'uuid': '0a4236a8-669e-4397-ba5d-68ed5b40da96', 'active': True, 'label': 30, 'ip6_address': '::', 'vm_name': 'ctest-vn2_vm2-55589463', '__T': 1759938675417398, 'ip6_active': False, 'is_health_check_active': True} 2025-10-08 15:51:35,534 - DEBUG - VM uve shows ip address as ['192.168.2.251', '::'] 2025-10-08 15:51:35,534 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 2025-10-08 15:51:35,534 - INFO - Verifying through opserver in 10.0.0.133 2025-10-08 15:51:35,534 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607?flat 2025-10-08 15:51:35,559 - DEBUG - expected vm list e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c 2025-10-08 15:51:35,559 - DEBUG - Extracted vm list ['e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c'] 2025-10-08 15:51:35,560 - INFO - VM e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c is present in default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 2025-10-08 15:51:35,560 - DEBUG - Verifying vm in vrouter uve 2025-10-08 15:51:35,560 - DEBUG - Getting info from collector 10.0.0.242.. 2025-10-08 15:51:35,560 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-machine/e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c?flat 2025-10-08 15:51:35,577 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4236-1'] 2025-10-08 15:51:35,577 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4236-1:Compute:contrail-vrouter-agent:0?flat 2025-10-08 15:51:35,639 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4236-1?flat 2025-10-08 15:51:35,680 - DEBUG - VM e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c is present in vrouter an-jenkins-deploy-platform-ansible-os-4236-1 uve 2025-10-08 15:51:35,680 - DEBUG - Validated that VM ctest-vn2_vm2-55589463 is in Vrouter an-jenkins-deploy-platform-ansible-os-4236-1 UVE 2025-10-08 15:51:35,680 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-10-08 15:51:35,692 - DEBUG - Expected tap interface of VM uuid e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c is default-domain:ctest-TestBasicVMVN0-24481103:0a4236a8-669e-4397-ba5d-68ed5b40da96 2025-10-08 15:51:35,693 - DEBUG - Expected VN of VM uuid e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c is default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 2025-10-08 15:51:35,693 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4236-1:Compute:contrail-vrouter-agent:0?flat 2025-10-08 15:51:35,757 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4236-1?flat 2025-10-08 15:51:35,846 - DEBUG - VM e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c is present in vrouter an-jenkins-deploy-platform-ansible-os-4236-1 uve 2025-10-08 15:51:35,846 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-24481103:0a4236a8-669e-4397-ba5d-68ed5b40da96 of vm e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c is present in vrouter an-jenkins-deploy-platform-ansible-os-4236-1 uve 2025-10-08 15:51:35,846 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 of vm e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c is present in vrouter an-jenkins-deploy-platform-ansible-os-4236-1 uve 2025-10-08 15:51:35,846 - DEBUG - Verifying the e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c virtual network link through opserver 10.0.0.133 2025-10-08 15:51:35,846 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-machines 2025-10-08 15:51:35,861 - DEBUG - VM link and name as {'name': 'e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c', 'href': 'http://10.0.0.133:8081/analytics/uves/virtual-machine/e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c?flat'} 2025-10-08 15:51:35,861 - DEBUG - Verifying the e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c virtual network link through opserver 10.0.0.50 2025-10-08 15:51:35,861 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2025-10-08 15:51:35,874 - DEBUG - VM link and name as {'name': 'e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-machine/e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c?flat'} 2025-10-08 15:51:35,874 - DEBUG - Verifying the e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c virtual network link through opserver 10.0.0.242 2025-10-08 15:51:35,874 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-machines 2025-10-08 15:51:35,890 - DEBUG - VM link and name as {'name': 'e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c', 'href': 'http://10.0.0.242:8081/analytics/uves/virtual-machine/e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c?flat'} 2025-10-08 15:51:35,890 - INFO - VM ctest-vn2_vm2-55589463 validations in Opserver passed 2025-10-08 15:51:35,890 - INFO - Waiting for VM ctest-vn1_vm2-02553484 to be up.. 2025-10-08 15:51:35,890 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:51:35,891 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/a4f84b86-49e3-405a-9012-9136204d94f4 2025-10-08 15:51:35,902 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine-interface/d7365475-5158-40e7-a585-6f1d89711290 2025-10-08 15:51:37,069 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=3.35 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.492 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.492/1.919/3.347/1.427 ms') 2025-10-08 15:51:37,070 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-02553484 passed 2025-10-08 15:51:37,150 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:51:37,151 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-02553484, IP 192.168.1.251, Port 22 2025-10-08 15:51:37,318 - DEBUG - VM ctest-vn1_vm2-02553484 is ready for SSH connections 2025-10-08 15:51:37,318 - INFO - Waiting for VM ctest-vn2_vm2-55589463 to be up.. 2025-10-08 15:51:37,318 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:51:37,319 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c 2025-10-08 15:51:37,332 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine-interface/0a4236a8-669e-4397-ba5d-68ed5b40da96 2025-10-08 15:51:38,513 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=1.77 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=2.52 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 1.767/2.145/2.524/0.378 ms') 2025-10-08 15:51:38,513 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-55589463 passed 2025-10-08 15:51:38,598 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:51:38,598 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-55589463, IP 192.168.2.251, Port 22 2025-10-08 15:51:38,656 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-08 15:51:38,763 - DEBUG - VM ctest-vn2_vm2-55589463 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-08 15:51:43,763 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:51:43,763 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-55589463, IP 192.168.2.251, Port 22 2025-10-08 15:51:43,826 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-08 15:51:43,934 - DEBUG - VM ctest-vn2_vm2-55589463 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-08 15:51:48,935 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:51:48,936 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-55589463, IP 192.168.2.251, Port 22 2025-10-08 15:51:48,992 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-08 15:51:49,092 - DEBUG - VM ctest-vn2_vm2-55589463 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-08 15:51:54,093 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:51:54,093 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-55589463, IP 192.168.2.251, Port 22 2025-10-08 15:51:54,262 - DEBUG - VM ctest-vn2_vm2-55589463 is ready for SSH connections 2025-10-08 15:51:54,263 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-08 15:51:54,263 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.251, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.241, gateway password: c0ntrail123 2025-10-08 15:51:54,263 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.251 2025-10-08 15:52:34,476 - DEBUG - PING 192.168.2.251 (192.168.2.251) 56(84) bytes of data. 64 bytes from 192.168.2.251: icmp_req=1 ttl=63 time=12.0 ms 64 bytes from 192.168.2.251: icmp_req=2 ttl=63 time=1.37 ms 64 bytes from 192.168.2.251: icmp_req=3 ttl=63 time=1.06 ms --- 192.168.2.251 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.061/4.834/12.065/5.114 ms 2025-10-08 15:52:34,477 - INFO - Ping to IP 192.168.2.251 from VM ctest-vn1_vm2-02553484 passed 2025-10-08 15:52:34,477 - INFO - Deleting VM ctest-vn2_vm2-55589463 2025-10-08 15:52:34,578 - DEBUG - Verifying in api server 10.0.0.242 2025-10-08 15:52:34,578 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c 2025-10-08 15:52:34,588 - DEBUG - VM ID e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c of VM ctest-vn2_vm2-55589463 is still found in API Server 2025-10-08 15:52:36,588 - DEBUG - Verifying in api server 10.0.0.242 2025-10-08 15:52:36,589 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c 2025-10-08 15:52:36,599 - DEBUG - Response Code: 404 2025-10-08 15:52:36,600 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c 2025-10-08 15:52:36,606 - DEBUG - Response Code: 404 2025-10-08 15:52:36,607 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c 2025-10-08 15:52:36,615 - DEBUG - Response Code: 404 2025-10-08 15:52:36,615 - INFO - VM ctest-vn2_vm2-55589463 is fully removed in API-Server 2025-10-08 15:52:36,783 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-10-08 15:52:36,960 - DEBUG - VM route 192.168.2.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 of compute 10.0.0.61 2025-10-08 15:52:36,968 - DEBUG - Validated that vrouter 10.0.0.61 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 2025-10-08 15:52:36,969 - DEBUG - Validated that vrouter 10.0.0.241 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 2025-10-08 15:52:36,969 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 2025-10-08 15:52:36,969 - INFO - VM ctest-vn2_vm2-55589463 is removed in Compute, and routes are removed in all compute nodes 2025-10-08 15:52:37,205 - INFO - Routes for VM ctest-vn2_vm2-55589463 is removed in all control-nodes 2025-10-08 15:52:37,205 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4236-1:Compute:contrail-vrouter-agent:0?flat 2025-10-08 15:52:37,260 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4236-1?flat 2025-10-08 15:52:37,295 - DEBUG - VM e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c is still present in vrouter an-jenkins-deploy-platform-ansible-os-4236-1 uve 2025-10-08 15:52:37,295 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-08 15:52:37,295 - DEBUG - interface for vm e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c deleted from vrouter uve ... 2025-10-08 15:52:37,295 - DEBUG - Deleted interface not in error interface list ... 2025-10-08 15:52:40,296 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4236-1:Compute:contrail-vrouter-agent:0?flat 2025-10-08 15:52:40,352 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4236-1?flat 2025-10-08 15:52:40,382 - DEBUG - VM e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c is still present in vrouter an-jenkins-deploy-platform-ansible-os-4236-1 uve 2025-10-08 15:52:40,382 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-08 15:52:40,382 - DEBUG - interface for vm e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c deleted from vrouter uve ... 2025-10-08 15:52:40,382 - DEBUG - Deleted interface not in error interface list ... 2025-10-08 15:52:43,383 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4236-1:Compute:contrail-vrouter-agent:0?flat 2025-10-08 15:52:43,438 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4236-1?flat 2025-10-08 15:52:43,476 - DEBUG - VM e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c is still present in vrouter an-jenkins-deploy-platform-ansible-os-4236-1 uve 2025-10-08 15:52:43,477 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-08 15:52:43,477 - DEBUG - interface for vm e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c deleted from vrouter uve ... 2025-10-08 15:52:43,477 - DEBUG - Deleted interface not in error interface list ... 2025-10-08 15:52:46,478 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4236-1:Compute:contrail-vrouter-agent:0?flat 2025-10-08 15:52:46,541 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4236-1?flat 2025-10-08 15:52:46,575 - DEBUG - VM e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c is not present in vrouter an-jenkins-deploy-platform-ansible-os-4236-1 uve 2025-10-08 15:52:46,575 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-08 15:52:46,575 - DEBUG - interface for vm e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c deleted from vrouter uve ... 2025-10-08 15:52:46,575 - DEBUG - Deleted interface not in error interface list ... 2025-10-08 15:52:46,575 - DEBUG - Validated that VM e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c is removed in Vrouter UVE 2025-10-08 15:52:46,575 - DEBUG - Verifying through opserver in 10.0.0.133 2025-10-08 15:52:46,575 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607?flat 2025-10-08 15:52:46,603 - DEBUG - e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c BM not in default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn2-11998607 uve 2025-10-08 15:52:46,603 - DEBUG - Validated that VM e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c is not present in the VN UVE 2025-10-08 15:52:46,603 - DEBUG - Verifying the e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c virtual network link through opserver 10.0.0.133 2025-10-08 15:52:46,603 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-machines 2025-10-08 15:52:46,617 - INFO - e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c vm uve deleted from opserver 2025-10-08 15:52:46,617 - DEBUG - Verifying the e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c virtual network link through opserver 10.0.0.50 2025-10-08 15:52:46,617 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2025-10-08 15:52:46,631 - INFO - e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c vm uve deleted from opserver 2025-10-08 15:52:46,631 - DEBUG - Verifying the e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c virtual network link through opserver 10.0.0.242 2025-10-08 15:52:46,631 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-machines 2025-10-08 15:52:46,644 - INFO - e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c vm uve deleted from opserver 2025-10-08 15:52:46,644 - INFO - Validated that VM e32d1141-ba9e-4a08-a4fe-1b1dd5626a3c is removed from Opserver 2025-10-08 15:52:46,644 - INFO - Deleting VM ctest-vn1_vm2-02553484 2025-10-08 15:52:46,730 - DEBUG - Verifying in api server 10.0.0.242 2025-10-08 15:52:46,730 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/a4f84b86-49e3-405a-9012-9136204d94f4 2025-10-08 15:52:46,740 - DEBUG - VM ID a4f84b86-49e3-405a-9012-9136204d94f4 of VM ctest-vn1_vm2-02553484 is still found in API Server 2025-10-08 15:52:48,741 - DEBUG - Verifying in api server 10.0.0.242 2025-10-08 15:52:48,742 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/a4f84b86-49e3-405a-9012-9136204d94f4 2025-10-08 15:52:48,751 - DEBUG - Response Code: 404 2025-10-08 15:52:48,751 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/a4f84b86-49e3-405a-9012-9136204d94f4 2025-10-08 15:52:48,759 - DEBUG - Response Code: 404 2025-10-08 15:52:48,759 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-machine/a4f84b86-49e3-405a-9012-9136204d94f4 2025-10-08 15:52:48,767 - DEBUG - Response Code: 404 2025-10-08 15:52:48,768 - INFO - VM ctest-vn1_vm2-02553484 is fully removed in API-Server 2025-10-08 15:52:48,935 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-10-08 15:52:48,978 - DEBUG - Validated that vrouter 10.0.0.61 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 2025-10-08 15:52:49,111 - DEBUG - VM route 192.168.1.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 of compute 10.0.0.241 2025-10-08 15:52:49,119 - DEBUG - Validated that vrouter 10.0.0.241 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 2025-10-08 15:52:49,119 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 2025-10-08 15:52:49,120 - INFO - VM ctest-vn1_vm2-02553484 is removed in Compute, and routes are removed in all compute nodes 2025-10-08 15:52:49,341 - INFO - Routes for VM ctest-vn1_vm2-02553484 is removed in all control-nodes 2025-10-08 15:52:49,341 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4236-2:Compute:contrail-vrouter-agent:0?flat 2025-10-08 15:52:49,468 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4236-2?flat 2025-10-08 15:52:49,510 - DEBUG - VM a4f84b86-49e3-405a-9012-9136204d94f4 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4236-2 uve 2025-10-08 15:52:49,510 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-08 15:52:49,511 - DEBUG - interface for vm a4f84b86-49e3-405a-9012-9136204d94f4 deleted from vrouter uve ... 2025-10-08 15:52:49,511 - DEBUG - Deleted interface not in error interface list ... 2025-10-08 15:52:52,512 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4236-2:Compute:contrail-vrouter-agent:0?flat 2025-10-08 15:52:52,562 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4236-2?flat 2025-10-08 15:52:52,603 - DEBUG - VM a4f84b86-49e3-405a-9012-9136204d94f4 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4236-2 uve 2025-10-08 15:52:52,603 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-08 15:52:52,603 - DEBUG - interface for vm a4f84b86-49e3-405a-9012-9136204d94f4 deleted from vrouter uve ... 2025-10-08 15:52:52,603 - DEBUG - Deleted interface not in error interface list ... 2025-10-08 15:52:55,604 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4236-2:Compute:contrail-vrouter-agent:0?flat 2025-10-08 15:52:55,656 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4236-2?flat 2025-10-08 15:52:55,701 - DEBUG - VM a4f84b86-49e3-405a-9012-9136204d94f4 is not present in vrouter an-jenkins-deploy-platform-ansible-os-4236-2 uve 2025-10-08 15:52:55,701 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-08 15:52:55,701 - DEBUG - interface for vm a4f84b86-49e3-405a-9012-9136204d94f4 deleted from vrouter uve ... 2025-10-08 15:52:55,701 - DEBUG - Deleted interface not in error interface list ... 2025-10-08 15:52:55,701 - DEBUG - Validated that VM a4f84b86-49e3-405a-9012-9136204d94f4 is removed in Vrouter UVE 2025-10-08 15:52:55,701 - DEBUG - Verifying through opserver in 10.0.0.133 2025-10-08 15:52:55,701 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416?flat 2025-10-08 15:52:55,729 - DEBUG - a4f84b86-49e3-405a-9012-9136204d94f4 BM not in default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 uve 2025-10-08 15:52:55,729 - DEBUG - Validated that VM a4f84b86-49e3-405a-9012-9136204d94f4 is not present in the VN UVE 2025-10-08 15:52:55,729 - DEBUG - Verifying the a4f84b86-49e3-405a-9012-9136204d94f4 virtual network link through opserver 10.0.0.133 2025-10-08 15:52:55,729 - DEBUG - Requesting: http://10.0.0.133:8081/analytics/uves/virtual-machines 2025-10-08 15:52:55,743 - INFO - a4f84b86-49e3-405a-9012-9136204d94f4 vm uve deleted from opserver 2025-10-08 15:52:55,743 - DEBUG - Verifying the a4f84b86-49e3-405a-9012-9136204d94f4 virtual network link through opserver 10.0.0.50 2025-10-08 15:52:55,743 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines 2025-10-08 15:52:55,757 - INFO - a4f84b86-49e3-405a-9012-9136204d94f4 vm uve deleted from opserver 2025-10-08 15:52:55,757 - DEBUG - Verifying the a4f84b86-49e3-405a-9012-9136204d94f4 virtual network link through opserver 10.0.0.242 2025-10-08 15:52:55,757 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-machines 2025-10-08 15:52:55,770 - INFO - a4f84b86-49e3-405a-9012-9136204d94f4 vm uve deleted from opserver 2025-10-08 15:52:55,770 - INFO - Validated that VM a4f84b86-49e3-405a-9012-9136204d94f4 is removed from Opserver 2025-10-08 15:52:55,770 - INFO - Deleting VN ctest-vn3-57172718 2025-10-08 15:52:55,877 - INFO - Deleting VM ctest-vn2_vm1-83080481 2025-10-08 15:52:56,006 - INFO - Deleting VM ctest-vn1_vm1-37998096 2025-10-08 15:52:56,256 - INFO - Deleting VN ctest-vn2-11998607 2025-10-08 15:52:56,316 - DEBUG - RefsExistError Delete when resource still referred: ['http://10.0.0.242:8082/virtual-machine-interface/4ce3468b-6957-40f7-a17e-03af90bab406'] while deleting VN ctest-vn2-11998607..Will retry 2025-10-08 15:52:58,439 - DEBUG - Requesting: http://10.0.0.133:8082/routing-instance/95310814-43d1-496c-96be-cda460a203d9 2025-10-08 15:52:58,446 - DEBUG - Response Code: 404 2025-10-08 15:52:58,447 - DEBUG - Requesting: http://10.0.0.133:8082/domains 2025-10-08 15:52:58,456 - DEBUG - Requesting: http://10.0.0.133:8082/domain/adc52b80-2ed1-485c-90d4-83e78c8e1400 2025-10-08 15:52:58,472 - DEBUG - Requesting: http://10.0.0.133:8082/project/2d0f9370-44fd-4f7b-99c3-73540ae4a690 2025-10-08 15:52:58,590 - INFO - Validated that VN ctest-vn2-11998607 is not found in API Server 2025-10-08 15:52:58,603 - DEBUG - VN ctest-vn2-11998607 is not present in Agent 10.0.0.61 2025-10-08 15:52:58,615 - DEBUG - VN ctest-vn2-11998607 is not present in Agent 10.0.0.241 2025-10-08 15:52:58,615 - INFO - Validated that VN ctest-vn2-11998607 is not in any agent 2025-10-08 15:52:58,660 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn2-11998607 info 2025-10-08 15:52:58,821 - INFO - Deleting VN ctest-vn1-31040416 2025-10-08 15:52:58,968 - DEBUG - Requesting: http://10.0.0.133:8082/routing-instance/d733f0ef-873d-40f1-ba17-4e5de4976db9 2025-10-08 15:52:58,975 - DEBUG - Response Code: 404 2025-10-08 15:52:58,975 - DEBUG - Requesting: http://10.0.0.133:8082/domains 2025-10-08 15:52:58,982 - DEBUG - Requesting: http://10.0.0.133:8082/domain/adc52b80-2ed1-485c-90d4-83e78c8e1400 2025-10-08 15:52:59,015 - DEBUG - Requesting: http://10.0.0.133:8082/project/2d0f9370-44fd-4f7b-99c3-73540ae4a690 2025-10-08 15:52:59,137 - INFO - Validated that VN ctest-vn1-31040416 is not found in API Server 2025-10-08 15:52:59,152 - DEBUG - VN ctest-vn1-31040416 is not present in Agent 10.0.0.61 2025-10-08 15:52:59,164 - DEBUG - VN ctest-vn1-31040416 is not present in Agent 10.0.0.241 2025-10-08 15:52:59,164 - INFO - Validated that VN ctest-vn1-31040416 is not in any agent 2025-10-08 15:52:59,171 - DEBUG - VRF 2 is not seen in agent 10.0.0.241 2025-10-08 15:52:59,177 - DEBUG - Vrouter 10.0.0.241 does not have vrf 2 for VN ctest-vn1-31040416 2025-10-08 15:52:59,177 - INFO - Validated that all vrouters do not have the route table for VN default-domain:ctest-TestBasicVMVN0-24481103:ctest-vn1-31040416 2025-10-08 15:52:59,202 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-31040416 info 2025-10-08 15:52:59,242 - INFO - Deleted policy policy2 2025-10-08 15:52:59,284 - INFO - Deleted policy policy1 2025-10-08 15:53:00,577 - DEBUG - Skipping xmpp flap check 2025-10-08 15:53:00,577 - INFO - END TEST : test_process_restart_in_policy_between_vns : PASSED[0:05:52] 2025-10-08 15:53:00,577 - INFO - -------------------------------------------------------------------------------- 2025-10-08 15:53:00,580 - INFO - ================================================================================ 2025-10-08 15:53:00,580 - INFO - STARTING TEST : test_underlay_broadcast_traffic_handling 2025-10-08 15:53:00,580 - INFO - TEST DESCRIPTION : Test the underlay brocast traffic handling by vrouter. (Bug-1545229). 1. Send broadcast traffic from one compute node. 2. Other compute in same subnet should receive that traffic. 3. Receiving compute should treat this traffic as underlay. 4. Compute should not replicate the packet and send the copy back. Pass criteria: Step 3-4 should pass Maintainer : chhandak@juniper.net 2025-10-08 15:53:01,861 - DEBUG - Skipping xmpp flap check 2025-10-08 15:53:01,861 - INFO - Initial checks done. Running the testcase now 2025-10-08 15:53:01,861 - INFO - 2025-10-08 15:53:02,274 - INFO - Agent interface name: ens6 2025-10-08 15:53:02,358 - DEBUG - Executing command: sudo tcpdump -nni ens6 -U host 0.0.0.0 -w /tmp/ens6_ctest-random-87844150.pcap 2025-10-08 15:53:07,727 - INFO - Pinging broacast address 0.0.0.0 from compute 10.20.0.241 2025-10-08 15:53:22,042 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-10-08 15:53:24,188 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-87844150.pcap | grep -c length 2025-10-08 15:53:24,201 - DEBUG - STDOUT: 9 2025-10-08 15:53:24,201 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-87844150.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-10-08 15:53:24,201 - WARNING - 9 packets are found in tcpdump output file /tmp/ens6_ctest-random-87844150.pcap but expected 10 2025-10-08 15:53:26,202 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-87844150.pcap | grep -c length 2025-10-08 15:53:26,217 - DEBUG - STDOUT: 9 2025-10-08 15:53:26,217 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-87844150.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-10-08 15:53:26,217 - WARNING - 9 packets are found in tcpdump output file /tmp/ens6_ctest-random-87844150.pcap but expected 10 2025-10-08 15:53:28,217 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-87844150.pcap | grep -c length 2025-10-08 15:53:28,231 - DEBUG - STDOUT: 9 2025-10-08 15:53:28,232 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-87844150.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-10-08 15:53:28,232 - WARNING - 9 packets are found in tcpdump output file /tmp/ens6_ctest-random-87844150.pcap but expected 10 2025-10-08 15:53:30,232 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-87844150.pcap | grep -c length 2025-10-08 15:53:30,246 - DEBUG - STDOUT: 9 2025-10-08 15:53:30,246 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-87844150.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-10-08 15:53:30,246 - WARNING - 9 packets are found in tcpdump output file /tmp/ens6_ctest-random-87844150.pcap but expected 10 2025-10-08 15:53:32,247 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-87844150.pcap | grep -c length 2025-10-08 15:53:32,261 - DEBUG - STDOUT: 9 2025-10-08 15:53:32,261 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-87844150.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-10-08 15:53:32,261 - WARNING - 9 packets are found in tcpdump output file /tmp/ens6_ctest-random-87844150.pcap but expected 10 2025-10-08 15:53:34,261 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-87844150.pcap | grep -c length 2025-10-08 15:53:34,274 - DEBUG - STDOUT: 9 2025-10-08 15:53:34,275 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-87844150.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-10-08 15:53:34,275 - WARNING - 9 packets are found in tcpdump output file /tmp/ens6_ctest-random-87844150.pcap but expected 10 2025-10-08 15:53:36,276 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-87844150.pcap | grep -c length 2025-10-08 15:53:36,289 - DEBUG - STDOUT: 9 2025-10-08 15:53:36,289 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-87844150.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-10-08 15:53:36,289 - WARNING - 9 packets are found in tcpdump output file /tmp/ens6_ctest-random-87844150.pcap but expected 10 2025-10-08 15:53:37,569 - ERROR - AssertionError Python 3.9.21: /usr/bin/python3 Wed Oct 8 15:53:36 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={}) 78 log.info('Initial checks done. Running the testcase now') 79 log.info('') 80 result = function(self, *args, **kwargs) 81 if self.inputs.upgrade: 82 pid = os.getpid() result = None function = self = args = () kwargs = {} /contrail-test/serial_scripts/vm_regression/test_vm_serial.py in test_underlay_broadcast_traffic_handling(self=) 948 949 # Analyze pcap 950 assert verify_tcpdump_count(self, session, pcap, exp_count=packet_count), "There should only be %s\ 951 packet from source %s on compute %s" %(packet_count, broadcast_address, compute_ip) 952 self.logger.info( global verify_tcpdump_count = .deco_retry..f_retry> self = session = pcap = '/tmp/ens6_ctest-random-87844150.pcap' exp_count undefined packet_count = 10 broadcast_address = '0.0.0.0' compute_ip = '10.0.0.241' AssertionError: There should only be 10 packet from source 0.0.0.0 on compute 10.0.0.241 __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {} __dir__ = __doc__ = 'Assertion failed.' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = args = ('There should only be 10 ... packet from source 0.0.0.0 on compute 10.0.0.241',) with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper result = function(self, *args, **kwargs) File "/contrail-test/serial_scripts/vm_regression/test_vm_serial.py", line 950, in test_underlay_broadcast_traffic_handling assert verify_tcpdump_count(self, session, pcap, exp_count=packet_count), "There should only be %s\ AssertionError: There should only be 10 packet from source 0.0.0.0 on compute 10.0.0.241 2025-10-08 15:53:37,569 - DEBUG - Skipping xmpp flap check 2025-10-08 15:53:37,569 - INFO - 2025-10-08 15:53:37,569 - INFO - END TEST : test_underlay_broadcast_traffic_handling : FAILED[0:00:37] 2025-10-08 15:53:37,570 - INFO - -------------------------------------------------------------------------------- 2025-10-08 15:53:38,393 - INFO - Deleted project: ctest-TestBasicVMVN0-24481103, ID : 2d0f9370-44fd-4f7b-99c3-73540ae4a690