2025-10-29 21:33:02,891 - INFO - Domain Default found not creating 2025-10-29 21:33:03,052 - INFO - Project ctest-TestBasicVMVN0-91865949 not found, creating it 2025-10-29 21:33:03,521 - INFO - Created Project:ctest-TestBasicVMVN0-91865949, ID : 9b488c42-e15b-4391-a1ef-0c96a0dda6cf 2025-10-29 21:33:04,987 - INFO - ================================================================================ 2025-10-29 21:33:04,987 - INFO - STARTING TEST : test_control_node_switchover 2025-10-29 21:33:04,987 - INFO - TEST DESCRIPTION : Stop the control node and check peering with agent fallback to other control node. 1. Pick one VN from respource pool which has 2 VM's in it 2. Verify ping between VM's 3. Find active control node in cluster by agent inspect 4. Stop control service on active control node 5. Verify agents are connected to new active control-node using xmpp connections 6. Bring back control service on previous active node 7. Verify ping between VM's again after bringing up control serveice Pass criteria: Step 2,5 and 7 should pass 2025-10-29 21:33:06,298 - DEBUG - Skipping xmpp flap check 2025-10-29 21:33:06,298 - INFO - Initial checks done. Running the testcase now 2025-10-29 21:33:06,298 - INFO - 2025-10-29 21:33:06,952 - DEBUG - Response for create_network : {'network': {'id': 'c4d45393-8c58-4fa7-8953-3e3277504438', 'name': 'ctest-vn1-81904884', 'tenant_id': '9b488c42e15b4391a1ef0c96a0dda6cf', 'project_id': '9b488c42e15b4391a1ef0c96a0dda6cf', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestBasicVMVN0-91865949', 'ctest-vn1-81904884'], 'port_security_enabled': True, 'description': ''}} 2025-10-29 21:33:07,230 - DEBUG - Response for create_subnet : {'subnet': {'id': '3573be30-9be2-4785-8e6d-57f31a3edab7', 'name': '', 'tenant_id': '9b488c42e15b4391a1ef0c96a0dda6cf', 'network_id': 'c4d45393-8c58-4fa7-8953-3e3277504438', 'ip_version': 4, 'cidr': '192.168.1.0/24', 'allocation_pools': [{'start': '192.168.1.2', 'end': '192.168.1.254'}], 'gateway_ip': '192.168.1.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '192.168.1.2', 'tags': [], 'project_id': '9b488c42e15b4391a1ef0c96a0dda6cf'}} 2025-10-29 21:33:07,256 - INFO - Created VN ctest-vn1-81904884 2025-10-29 21:33:07,268 - DEBUG - VN ctest-vn1-81904884 UUID is c4d45393-8c58-4fa7-8953-3e3277504438 2025-10-29 21:33:07,344 - DEBUG - Requesting: http://10.0.0.254:8082/domains 2025-10-29 21:33:07,683 - DEBUG - Requesting: http://10.0.0.254:8082/domains 2025-10-29 21:33:07,731 - DEBUG - Requesting: http://10.0.0.254:8082/domain/a090225f-9b9e-4366-926c-6397a8c195bb 2025-10-29 21:33:07,747 - DEBUG - Requesting: http://10.0.0.254:8082/project/9b488c42-e15b-4391-a1ef-0c96a0dda6cf 2025-10-29 21:33:07,809 - DEBUG - Requesting: http://10.0.0.254:8082/virtual-network/c4d45393-8c58-4fa7-8953-3e3277504438 2025-10-29 21:33:07,843 - DEBUG - Requesting: http://10.0.0.254:8082/virtual-network/c4d45393-8c58-4fa7-8953-3e3277504438 2025-10-29 21:33:07,855 - DEBUG - Requesting: http://10.0.0.254:8082/routing-instance/798e55d5-8d6a-469c-b488-a0236a8ca72b 2025-10-29 21:33:07,865 - DEBUG - Requesting: http://10.0.0.254:8082/routing-instance/798e55d5-8d6a-469c-b488-a0236a8ca72b 2025-10-29 21:33:07,872 - DEBUG - Requesting: http://10.0.0.254:8082/route-target/afd8f047-05f4-4116-b538-90b5e6d20b08 2025-10-29 21:33:07,881 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-29 21:33:07,881 - DEBUG - Requesting: http://10.0.0.254:8082/virtual-network/c4d45393-8c58-4fa7-8953-3e3277504438 2025-10-29 21:33:07,891 - DEBUG - Requesting: http://10.0.0.254:8082/routing-instance/798e55d5-8d6a-469c-b488-a0236a8ca72b 2025-10-29 21:33:07,913 - INFO - Verified VN network id 15 for VN c4d45393-8c58-4fa7-8953-3e3277504438 2025-10-29 21:33:07,913 - INFO - Verifications in API Server for VN ctest-vn1-81904884 passed 2025-10-29 21:33:07,913 - DEBUG - Requesting: http://10.0.0.254:8082/virtual-network/c4d45393-8c58-4fa7-8953-3e3277504438 2025-10-29 21:33:07,940 - DEBUG - Requesting: http://10.0.0.254:8082/routing-instance/798e55d5-8d6a-469c-b488-a0236a8ca72b 2025-10-29 21:33:07,948 - DEBUG - Requesting: http://10.0.0.254:8082/routing-instance/798e55d5-8d6a-469c-b488-a0236a8ca72b 2025-10-29 21:33:07,956 - DEBUG - Requesting: http://10.0.0.254:8082/route-target/afd8f047-05f4-4116-b538-90b5e6d20b08 2025-10-29 21:33:07,970 - DEBUG - Control-node 10.0.0.254 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91865949', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '14183053019676430247', 'uuid-lslong': '9895321192734147640'}, 'enable': 'true', 'created': '2025-10-29T21:33:06', 'last-modified': '2025-10-29T21:33:07', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.799167'} 2025-10-29 21:33:07,977 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-29 21:33:07,983 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91865949', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '14183053019676430247', 'uuid-lslong': '9895321192734147640'}, 'enable': 'true', 'created': '2025-10-29T21:33:06', 'last-modified': '2025-10-29T21:33:07', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.815095'} 2025-10-29 21:33:07,990 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-29 21:33:07,995 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91865949', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '14183053019676430247', 'uuid-lslong': '9895321192734147640'}, 'enable': 'true', 'created': '2025-10-29T21:33:06', 'last-modified': '2025-10-29T21:33:07', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.826472'} 2025-10-29 21:33:08,001 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-29 21:33:08,001 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-81904884 passed 2025-10-29 21:33:08,001 - DEBUG - ====Verifying policy data for ctest-vn1-81904884 in API_Server ====== 2025-10-29 21:33:08,001 - DEBUG - Requesting: http://10.0.0.254:8082/domains 2025-10-29 21:33:08,007 - DEBUG - Requesting: http://10.0.0.254:8082/domain/a090225f-9b9e-4366-926c-6397a8c195bb 2025-10-29 21:33:08,020 - DEBUG - Requesting: http://10.0.0.254:8082/project/9b488c42-e15b-4391-a1ef-0c96a0dda6cf 2025-10-29 21:33:08,161 - DEBUG - Requesting: http://10.0.0.254:8082/virtual-network/c4d45393-8c58-4fa7-8953-3e3277504438 2025-10-29 21:33:08,173 - DEBUG - =>VN ctest-vn1-81904884 has no policy to be verified 2025-10-29 21:33:08,173 - DEBUG - Verifying the vn in opserver 2025-10-29 21:33:08,173 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 virtual network link through opserver 10.0.0.254 2025-10-29 21:33:08,174 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-networks 2025-10-29 21:33:08,566 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-networks 2025-10-29 21:33:08,625 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884', 'href': 'http://10.0.0.254:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884?flat'} 2025-10-29 21:33:08,625 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 virtual network link through opserver 10.0.0.38 2025-10-29 21:33:08,626 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-10-29 21:33:08,924 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-10-29 21:33:08,978 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884?flat'} 2025-10-29 21:33:08,978 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 virtual network link through opserver 10.0.0.33 2025-10-29 21:33:08,979 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2025-10-29 21:33:09,270 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2025-10-29 21:33:09,336 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884?flat'} 2025-10-29 21:33:09,336 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 is found in opserver 2025-10-29 21:33:09,366 - DEBUG - Do not have enough data to verify VN in agent 2025-10-29 21:33:09,383 - DEBUG - VRF ids for VN ctest-vn1-81904884: {} 2025-10-29 21:33:09,439 - DEBUG - Services list from nova: [, , , ] 2025-10-29 21:33:10,471 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4527-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4527-2) 2025-10-29 21:33:11,923 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4527-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4527-1) 2025-10-29 21:33:11,923 - INFO - Waiting for VM ctest-vn1_vm1-53214933 to be up.. 2025-10-29 21:33:12,041 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-10-29 21:33:17,119 - DEBUG - VM is in ACTIVE state now 2025-10-29 21:33:17,119 - INFO - VM name : ctest-vn1_vm1-53214933 2025-10-29 21:33:17,194 - DEBUG - VM ctest-vn1_vm1-53214933 ID is ef51fcc9-8db7-40e5-8492-b9071d078e17 2025-10-29 21:33:17,218 - DEBUG - VM ctest-vn1_vm1-53214933 launched on Node an-jenkins-deploy-platform-ansible-os-4527-2 2025-10-29 21:33:17,294 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/ef51fcc9-8db7-40e5-8492-b9071d078e17 2025-10-29 21:33:17,590 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/ef51fcc9-8db7-40e5-8492-b9071d078e17 2025-10-29 21:33:17,625 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine-interface/a4e6a4bd-2991-4878-a750-bd0068a81dfa 2025-10-29 21:33:20,797 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2025-10-29 21:33:20,798 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-53214933 failed! 2025-10-29 21:33:20,810 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 is 192.168.1.1 and allocation pool is NOT set 2025-10-29 21:33:24,898 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2025-10-29 21:33:24,898 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-53214933 failed! 2025-10-29 21:33:24,912 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 is 192.168.1.1 and allocation pool is NOT set 2025-10-29 21:33:28,990 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-10-29 21:33:28,990 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-53214933 failed! 2025-10-29 21:33:29,005 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 is 192.168.1.1 and allocation pool is NOT set 2025-10-29 21:33:33,088 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2025-10-29 21:33:33,088 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-53214933 failed! 2025-10-29 21:33:33,132 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 is 192.168.1.1 and allocation pool is NOT set 2025-10-29 21:33:37,214 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2025-10-29 21:33:37,214 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-53214933 failed! 2025-10-29 21:33:37,229 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 is 192.168.1.1 and allocation pool is NOT set 2025-10-29 21:33:41,310 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-10-29 21:33:41,310 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-53214933 failed! 2025-10-29 21:33:41,325 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 is 192.168.1.1 and allocation pool is NOT set 2025-10-29 21:33:45,402 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-10-29 21:33:45,402 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-53214933 failed! 2025-10-29 21:33:45,417 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 is 192.168.1.1 and allocation pool is NOT set 2025-10-29 21:33:49,497 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=5.66 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 1 received, 50% packet loss, time 1023ms\r\nrtt min/avg/max/mdev = 5.659/5.659/5.659/0.000 ms') 2025-10-29 21:33:49,497 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-53214933 passed 2025-10-29 21:33:49,568 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:33:49,568 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-53214933, IP 192.168.1.3, Port 22 2025-10-29 21:33:49,635 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-29 21:33:49,762 - DEBUG - VM ctest-vn1_vm1-53214933 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-29 21:33:54,763 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:33:54,763 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-53214933, IP 192.168.1.3, Port 22 2025-10-29 21:33:54,832 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-29 21:33:54,923 - DEBUG - VM ctest-vn1_vm1-53214933 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-29 21:33:59,924 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:33:59,924 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-53214933, IP 192.168.1.3, Port 22 2025-10-29 21:33:59,982 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-29 21:34:00,078 - DEBUG - VM ctest-vn1_vm1-53214933 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-29 21:34:05,079 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:34:05,079 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-53214933, IP 192.168.1.3, Port 22 2025-10-29 21:34:05,144 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-29 21:34:05,236 - DEBUG - VM ctest-vn1_vm1-53214933 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-29 21:34:10,237 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:34:10,237 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-53214933, IP 192.168.1.3, Port 22 2025-10-29 21:34:10,404 - DEBUG - VM ctest-vn1_vm1-53214933 is ready for SSH connections 2025-10-29 21:34:10,404 - INFO - Waiting for VM ctest-vn1_vm2-54840660 to be up.. 2025-10-29 21:34:10,512 - DEBUG - VM is in ACTIVE state now 2025-10-29 21:34:10,512 - INFO - VM name : ctest-vn1_vm2-54840660 2025-10-29 21:34:10,601 - DEBUG - VM ctest-vn1_vm2-54840660 ID is f0e1612b-b17c-424e-8322-27fc096c3428 2025-10-29 21:34:10,601 - DEBUG - VM ctest-vn1_vm2-54840660 launched on Node an-jenkins-deploy-platform-ansible-os-4527-1 2025-10-29 21:34:10,691 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/f0e1612b-b17c-424e-8322-27fc096c3428 2025-10-29 21:34:10,701 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine-interface/c4abcbb6-d499-421d-8fdb-99866b42d406 2025-10-29 21:34:11,907 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=5.72 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=2.88 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 2.881/4.299/5.717/1.418 ms') 2025-10-29 21:34:11,907 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm2-54840660 passed 2025-10-29 21:34:11,978 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:34:11,978 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-54840660, IP 192.168.1.4, Port 22 2025-10-29 21:34:12,057 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-29 21:34:12,151 - DEBUG - VM ctest-vn1_vm2-54840660 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-29 21:34:17,151 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:34:17,151 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-54840660, IP 192.168.1.4, Port 22 2025-10-29 21:34:17,353 - DEBUG - VM ctest-vn1_vm2-54840660 is ready for SSH connections 2025-10-29 21:34:17,353 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:34:17,353 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.242, gateway password: c0ntrail123 2025-10-29 21:34:17,354 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4 2025-10-29 21:35:02,566 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data. 64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=21.0 ms 64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.58 ms 64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.62 ms --- 192.168.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.585/8.073/21.015/9.151 ms 2025-10-29 21:35:02,566 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-53214933 passed 2025-10-29 21:35:02,566 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:35:02,566 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.241, gateway password: c0ntrail123 2025-10-29 21:35:02,566 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3 2025-10-29 21:35:43,783 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. 64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=7.08 ms 64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.76 ms 64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.91 ms --- 192.168.1.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2005ms rtt min/avg/max/mdev = 1.769/3.587/7.080/2.470 ms 2025-10-29 21:35:43,783 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-54840660 passed 2025-10-29 21:35:43,790 - INFO - Active control node from the Agent 10.0.0.242 is 10.0.0.38 2025-10-29 21:35:43,791 - INFO - Stoping the Control service in 10.0.0.38 2025-10-29 21:35:44,136 - INFO - 10.0.0.38 2025-10-29 21:35:44,136 - INFO - control:inactive 2025-10-29 21:35:44,137 - INFO - Contrail services control are down on nodes 10.0.0.38 2025-10-29 21:35:49,145 - INFO - Active control node from the Agent 10.0.0.242 is 10.0.0.254 2025-10-29 21:35:49,145 - INFO - Starting the Control service in 10.0.0.38 2025-10-29 21:35:49,453 - INFO - 10.0.0.38 2025-10-29 21:35:49,457 - INFO - control:initializing 2025-10-29 21:35:49,457 - DEBUG - defaultdict(, {'10.0.0.38': {'control': {'status': 'initializing', 'description': None}}}) 2025-10-29 21:35:49,457 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-29 21:35:54,556 - INFO - 10.0.0.38 2025-10-29 21:35:54,562 - INFO - control:active 2025-10-29 21:35:54,562 - INFO - Contrail services ['control'] are up on nodes 10.0.0.38 2025-10-29 21:35:59,652 - DEBUG - VM is in ACTIVE state now 2025-10-29 21:35:59,652 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:35:59,652 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/ef51fcc9-8db7-40e5-8492-b9071d078e17 2025-10-29 21:35:59,659 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine-interface/a4e6a4bd-2991-4878-a750-bd0068a81dfa 2025-10-29 21:35:59,669 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/ef51fcc9-8db7-40e5-8492-b9071d078e17 2025-10-29 21:35:59,677 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine-interface/a4e6a4bd-2991-4878-a750-bd0068a81dfa 2025-10-29 21:35:59,686 - DEBUG - Requesting: http://10.0.0.33:8082/instance-ip/d04c3803-bdf5-4009-8ccf-c6c084bcad4c 2025-10-29 21:35:59,695 - DEBUG - Verifying in api server 10.0.0.33 2025-10-29 21:35:59,695 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/ef51fcc9-8db7-40e5-8492-b9071d078e17 2025-10-29 21:35:59,703 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine-interface/a4e6a4bd-2991-4878-a750-bd0068a81dfa 2025-10-29 21:35:59,712 - DEBUG - Requesting: http://10.0.0.33:8082/instance-ip/d04c3803-bdf5-4009-8ccf-c6c084bcad4c 2025-10-29 21:35:59,722 - INFO - VM ctest-vn1_vm1-53214933 verfication in all API Servers passed 2025-10-29 21:35:59,738 - DEBUG - VM ctest-vn1_vm1-53214933 Tap interface: {'index': '3', 'name': 'tapa4e6a4bd-29', 'uuid': 'a4e6a4bd-2991-4878-a750-bd0068a81dfa', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884:ctest-vn1-81904884', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884', 'vm_uuid': 'ef51fcc9-8db7-40e5-8492-b9071d078e17', 'vm_name': 'ctest-vn1_vm1-53214933', 'ip_addr': '192.168.1.3', 'mac_addr': '02:a4:e6:a4:bd:29', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '165', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-91865949:a4e6a4bd-2991-4878-a750-bd0068a81dfa', 'sg_uuid_list': ['9976a145-8190-42a4-82b7-e413e90cdc3e'], 'static_route_list': None, 'vm_project_uuid': '9b488c42-e15b-4391-a1ef-0c96a0dda6cf', 'admin_state': 'Enabled', 'flow_key_idx': '25', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-29 21:35:59,738 - DEBUG - Agent 10.0.0.242 vrf name: default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884:ctest-vn1-81904884 2025-10-29 21:35:59,743 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884:ctest-vn1-81904884', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884', 'table_label': '-1', 'vxlan_id': '15', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.19:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-10-29 21:35:59,820 - DEBUG - VM ctest-vn1_vm1-53214933 : Tap interface tapa4e6a4bd-29 is set to Active 2025-10-29 21:35:59,820 - DEBUG - Tap interface tapa4e6a4bd-29 detail : {'index': '3', 'name': 'tapa4e6a4bd-29', 'uuid': 'a4e6a4bd-2991-4878-a750-bd0068a81dfa', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884:ctest-vn1-81904884', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884', 'vm_uuid': 'ef51fcc9-8db7-40e5-8492-b9071d078e17', 'vm_name': 'ctest-vn1_vm1-53214933', 'ip_addr': '192.168.1.3', 'mac_addr': '02:a4:e6:a4:bd:29', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '165', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-91865949:a4e6a4bd-2991-4878-a750-bd0068a81dfa', 'sg_uuid_list': ['9976a145-8190-42a4-82b7-e413e90cdc3e'], 'static_route_list': None, 'vm_project_uuid': '9b488c42-e15b-4391-a1ef-0c96a0dda6cf', 'admin_state': 'Enabled', 'flow_key_idx': '25', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-29 21:35:59,928 - DEBUG - Starting Layer 2 verification in Agent 2025-10-29 21:35:59,938 - DEBUG - Layer 2 path is seen for VM MAC 02:a4:e6:a4:bd:29 in agent 10.0.0.242 2025-10-29 21:35:59,938 - DEBUG - Active layer 2 route in agent is present for VMI tapa4e6a4bd-29 2025-10-29 21:35:59,938 - DEBUG - L2 label(29) matches bw route table and interface table 2025-10-29 21:35:59,949 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 is found in Agent of node 10.0.0.241 2025-10-29 21:36:00,094 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 is consistent in agent 10.0.0.241 2025-10-29 21:36:00,094 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.241 2025-10-29 21:36:00,094 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 verification for VM ctest-vn1_vm1-53214933 in Agent 10.0.0.241 passed 2025-10-29 21:36:00,206 - DEBUG - Starting all layer 2 verification in agent 10.0.0.241 2025-10-29 21:36:00,223 - DEBUG - Route for VM MAC 02:a4:e6:a4:bd:29 is consistent in agent 10.0.0.241 2025-10-29 21:36:00,230 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 is found in Agent of node 10.0.0.242 2025-10-29 21:36:00,361 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 is consistent in agent 10.0.0.242 2025-10-29 21:36:00,362 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.242 2025-10-29 21:36:00,362 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 verification for VM ctest-vn1_vm1-53214933 in Agent 10.0.0.242 passed 2025-10-29 21:36:00,470 - DEBUG - Starting all layer 2 verification in agent 10.0.0.242 2025-10-29 21:36:00,480 - DEBUG - Route for VM MAC 02:a4:e6:a4:bd:29 is consistent in agent 10.0.0.242 2025-10-29 21:36:00,481 - INFO - VM ctest-vn1_vm1-53214933 verifications in Compute nodes passed 2025-10-29 21:36:00,735 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.241 2025-10-29 21:36:00,741 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.19, 25; Got 10.20.0.19,25 2025-10-29 21:36:00,741 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.241 2025-10-29 21:36:00,844 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.242 2025-10-29 21:36:00,850 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-10-29 21:36:00,850 - INFO - Validated routes of VM ctest-vn1_vm1-53214933 in all vrouters 2025-10-29 21:36:01,331 - DEBUG - Starting all layer2 verification in 10.0.0.33 Control Node 2025-10-29 21:36:01,452 - DEBUG - Layer2 route found for VM MAC 02:a4:e6:a4:bd:29 in Control-node 10.0.0.33 2025-10-29 21:36:01,452 - DEBUG - L2 Label for VM ctest-vn1_vm1-53214933 same between Control-node 10.0.0.33 and Agent, Expected: 29, Seen: 29 2025-10-29 21:36:01,452 - DEBUG - Starting all layer2 verification in 10.0.0.38 Control Node 2025-10-29 21:36:01,575 - DEBUG - Layer2 route found for VM MAC 02:a4:e6:a4:bd:29 in Control-node 10.0.0.38 2025-10-29 21:36:01,575 - DEBUG - L2 Label for VM ctest-vn1_vm1-53214933 same between Control-node 10.0.0.38 and Agent, Expected: 29, Seen: 29 2025-10-29 21:36:01,575 - DEBUG - Starting all layer2 verification in 10.0.0.254 Control Node 2025-10-29 21:36:01,694 - DEBUG - Layer2 route found for VM MAC 02:a4:e6:a4:bd:29 in Control-node 10.0.0.254 2025-10-29 21:36:01,694 - DEBUG - L2 Label for VM ctest-vn1_vm1-53214933 same between Control-node 10.0.0.254 and Agent, Expected: 29, Seen: 29 2025-10-29 21:36:01,694 - INFO - Verification in Control-nodes for VM ctest-vn1_vm1-53214933 passed 2025-10-29 21:36:01,694 - DEBUG - Verifying the vm in opserver 2025-10-29 21:36:01,695 - DEBUG - Verifying in collector 10.0.0.33 ... 2025-10-29 21:36:01,695 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machine/ef51fcc9-8db7-40e5-8492-b9071d078e17?flat 2025-10-29 21:36:01,710 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91865949:a4e6a4bd-2991-4878-a750-bd0068a81dfa?flat 2025-10-29 21:36:01,729 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91865949:a4e6a4bd-2991-4878-a750-bd0068a81dfa?flat 2025-10-29 21:36:01,795 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machine/ef51fcc9-8db7-40e5-8492-b9071d078e17?flat 2025-10-29 21:36:01,854 - DEBUG - Verifying vm in vn uve 2025-10-29 21:36:01,854 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91865949:a4e6a4bd-2991-4878-a750-bd0068a81dfa?flat 2025-10-29 21:36:01,926 - DEBUG - VM uve shows interface as {'ip6_address': '::', 'ip6_active': False, 'ip4_active': True, 'vm_uuid': 'ef51fcc9-8db7-40e5-8492-b9071d078e17', 'fixed_ip4_list': ['192.168.1.3'], 'vhostuser_mode': 0, 'is_health_check_active': True, '__T': 1761773754872298, 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884', 'uuid': 'a4e6a4bd-2991-4878-a750-bd0068a81dfa', 'label': 25, 'admin_state': True, 'ip_address': '192.168.1.3', 'rx_vlan': 65535, 'mac_address': '02:a4:e6:a4:bd:29', 'vm_name': 'ctest-vn1_vm1-53214933', 'vn_uuid': 'c4d45393-8c58-4fa7-8953-3e3277504438', 'gateway': '192.168.1.1', 'tx_vlan': 65535, 'active': True, 'port_mirror_enabled': False, 'l2_active': True} 2025-10-29 21:36:01,926 - DEBUG - VM uve shows ip address as ['192.168.1.3', '::'] 2025-10-29 21:36:01,926 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 2025-10-29 21:36:01,926 - INFO - Verifying through opserver in 10.0.0.254 2025-10-29 21:36:01,926 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884?flat 2025-10-29 21:36:01,953 - DEBUG - expected vm list ef51fcc9-8db7-40e5-8492-b9071d078e17 2025-10-29 21:36:01,953 - DEBUG - Extracted vm list ['ef51fcc9-8db7-40e5-8492-b9071d078e17'] 2025-10-29 21:36:01,953 - INFO - VM ef51fcc9-8db7-40e5-8492-b9071d078e17 is present in default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 2025-10-29 21:36:01,954 - DEBUG - Verifying vm in vrouter uve 2025-10-29 21:36:01,954 - DEBUG - Getting info from collector 10.0.0.33.. 2025-10-29 21:36:01,954 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machine/ef51fcc9-8db7-40e5-8492-b9071d078e17?flat 2025-10-29 21:36:02,012 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4527-2'] 2025-10-29 21:36:02,012 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4527-2:Compute:contrail-vrouter-agent:0?flat 2025-10-29 21:36:02,062 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-2?flat 2025-10-29 21:36:02,105 - DEBUG - VM ef51fcc9-8db7-40e5-8492-b9071d078e17 is present in vrouter an-jenkins-deploy-platform-ansible-os-4527-2 uve 2025-10-29 21:36:02,105 - DEBUG - Validated that VM ctest-vn1_vm1-53214933 is in Vrouter an-jenkins-deploy-platform-ansible-os-4527-2 UVE 2025-10-29 21:36:02,105 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-10-29 21:36:02,113 - DEBUG - Expected tap interface of VM uuid ef51fcc9-8db7-40e5-8492-b9071d078e17 is default-domain:ctest-TestBasicVMVN0-91865949:a4e6a4bd-2991-4878-a750-bd0068a81dfa 2025-10-29 21:36:02,113 - DEBUG - Expected VN of VM uuid ef51fcc9-8db7-40e5-8492-b9071d078e17 is default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 2025-10-29 21:36:02,113 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4527-2:Compute:contrail-vrouter-agent:0?flat 2025-10-29 21:36:02,171 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-2?flat 2025-10-29 21:36:02,205 - DEBUG - VM ef51fcc9-8db7-40e5-8492-b9071d078e17 is present in vrouter an-jenkins-deploy-platform-ansible-os-4527-2 uve 2025-10-29 21:36:02,205 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-91865949:a4e6a4bd-2991-4878-a750-bd0068a81dfa of vm ef51fcc9-8db7-40e5-8492-b9071d078e17 is present in vrouter an-jenkins-deploy-platform-ansible-os-4527-2 uve 2025-10-29 21:36:02,205 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 of vm ef51fcc9-8db7-40e5-8492-b9071d078e17 is present in vrouter an-jenkins-deploy-platform-ansible-os-4527-2 uve 2025-10-29 21:36:02,205 - DEBUG - Verifying the ef51fcc9-8db7-40e5-8492-b9071d078e17 virtual network link through opserver 10.0.0.254 2025-10-29 21:36:02,205 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-machines 2025-10-29 21:36:02,223 - DEBUG - VM link and name as {'name': 'ef51fcc9-8db7-40e5-8492-b9071d078e17', 'href': 'http://10.0.0.254:8081/analytics/uves/virtual-machine/ef51fcc9-8db7-40e5-8492-b9071d078e17?flat'} 2025-10-29 21:36:02,223 - DEBUG - Verifying the ef51fcc9-8db7-40e5-8492-b9071d078e17 virtual network link through opserver 10.0.0.38 2025-10-29 21:36:02,223 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-10-29 21:36:02,239 - DEBUG - VM link and name as {'name': 'ef51fcc9-8db7-40e5-8492-b9071d078e17', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-machine/ef51fcc9-8db7-40e5-8492-b9071d078e17?flat'} 2025-10-29 21:36:02,239 - DEBUG - Verifying the ef51fcc9-8db7-40e5-8492-b9071d078e17 virtual network link through opserver 10.0.0.33 2025-10-29 21:36:02,240 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-10-29 21:36:02,269 - DEBUG - VM link and name as {'name': 'ef51fcc9-8db7-40e5-8492-b9071d078e17', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-machine/ef51fcc9-8db7-40e5-8492-b9071d078e17?flat'} 2025-10-29 21:36:02,269 - INFO - VM ctest-vn1_vm1-53214933 validations in Opserver passed 2025-10-29 21:36:02,362 - DEBUG - VM is in ACTIVE state now 2025-10-29 21:36:02,362 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:36:02,362 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/f0e1612b-b17c-424e-8322-27fc096c3428 2025-10-29 21:36:02,369 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine-interface/c4abcbb6-d499-421d-8fdb-99866b42d406 2025-10-29 21:36:02,379 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/f0e1612b-b17c-424e-8322-27fc096c3428 2025-10-29 21:36:02,387 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine-interface/c4abcbb6-d499-421d-8fdb-99866b42d406 2025-10-29 21:36:02,396 - DEBUG - Requesting: http://10.0.0.33:8082/instance-ip/9b4ea379-689a-4656-a2b9-54588cbfd4a2 2025-10-29 21:36:02,405 - DEBUG - Verifying in api server 10.0.0.33 2025-10-29 21:36:02,405 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/f0e1612b-b17c-424e-8322-27fc096c3428 2025-10-29 21:36:02,412 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine-interface/c4abcbb6-d499-421d-8fdb-99866b42d406 2025-10-29 21:36:02,421 - DEBUG - Requesting: http://10.0.0.33:8082/instance-ip/9b4ea379-689a-4656-a2b9-54588cbfd4a2 2025-10-29 21:36:02,430 - INFO - VM ctest-vn1_vm2-54840660 verfication in all API Servers passed 2025-10-29 21:36:02,460 - DEBUG - VM ctest-vn1_vm2-54840660 Tap interface: {'index': '3', 'name': 'tapc4abcbb6-d4', 'uuid': 'c4abcbb6-d499-421d-8fdb-99866b42d406', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884:ctest-vn1-81904884', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884', 'vm_uuid': 'f0e1612b-b17c-424e-8322-27fc096c3428', 'vm_name': 'ctest-vn1_vm2-54840660', 'ip_addr': '192.168.1.4', 'mac_addr': '02:c4:ab:cb:b6:d4', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '123', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-91865949:c4abcbb6-d499-421d-8fdb-99866b42d406', 'sg_uuid_list': ['9976a145-8190-42a4-82b7-e413e90cdc3e'], 'static_route_list': None, 'vm_project_uuid': '9b488c42-e15b-4391-a1ef-0c96a0dda6cf', 'admin_state': 'Enabled', 'flow_key_idx': '25', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-29 21:36:02,460 - DEBUG - Agent 10.0.0.241 vrf name: default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884:ctest-vn1-81904884 2025-10-29 21:36:02,466 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884:ctest-vn1-81904884', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884', 'table_label': '-1', 'vxlan_id': '15', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.129:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-10-29 21:36:02,553 - DEBUG - VM ctest-vn1_vm2-54840660 : Tap interface tapc4abcbb6-d4 is set to Active 2025-10-29 21:36:02,554 - DEBUG - Tap interface tapc4abcbb6-d4 detail : {'index': '3', 'name': 'tapc4abcbb6-d4', 'uuid': 'c4abcbb6-d499-421d-8fdb-99866b42d406', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884:ctest-vn1-81904884', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884', 'vm_uuid': 'f0e1612b-b17c-424e-8322-27fc096c3428', 'vm_name': 'ctest-vn1_vm2-54840660', 'ip_addr': '192.168.1.4', 'mac_addr': '02:c4:ab:cb:b6:d4', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '123', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-91865949:c4abcbb6-d499-421d-8fdb-99866b42d406', 'sg_uuid_list': ['9976a145-8190-42a4-82b7-e413e90cdc3e'], 'static_route_list': None, 'vm_project_uuid': '9b488c42-e15b-4391-a1ef-0c96a0dda6cf', 'admin_state': 'Enabled', 'flow_key_idx': '25', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-29 21:36:02,664 - DEBUG - Starting Layer 2 verification in Agent 2025-10-29 21:36:02,676 - DEBUG - Layer 2 path is seen for VM MAC 02:c4:ab:cb:b6:d4 in agent 10.0.0.241 2025-10-29 21:36:02,676 - DEBUG - Active layer 2 route in agent is present for VMI tapc4abcbb6-d4 2025-10-29 21:36:02,676 - DEBUG - L2 label(29) matches bw route table and interface table 2025-10-29 21:36:02,683 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 is found in Agent of node 10.0.0.241 2025-10-29 21:36:02,820 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 is consistent in agent 10.0.0.241 2025-10-29 21:36:02,820 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.241 2025-10-29 21:36:02,820 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 verification for VM ctest-vn1_vm2-54840660 in Agent 10.0.0.241 passed 2025-10-29 21:36:02,932 - DEBUG - Starting all layer 2 verification in agent 10.0.0.241 2025-10-29 21:36:02,950 - DEBUG - Route for VM MAC 02:c4:ab:cb:b6:d4 is consistent in agent 10.0.0.241 2025-10-29 21:36:02,958 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 is found in Agent of node 10.0.0.242 2025-10-29 21:36:03,089 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 is consistent in agent 10.0.0.242 2025-10-29 21:36:03,090 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.242 2025-10-29 21:36:03,090 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 verification for VM ctest-vn1_vm2-54840660 in Agent 10.0.0.242 passed 2025-10-29 21:36:03,202 - DEBUG - Starting all layer 2 verification in agent 10.0.0.242 2025-10-29 21:36:03,212 - DEBUG - Route for VM MAC 02:c4:ab:cb:b6:d4 is consistent in agent 10.0.0.242 2025-10-29 21:36:03,213 - INFO - VM ctest-vn1_vm2-54840660 verifications in Compute nodes passed 2025-10-29 21:36:03,478 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.241 2025-10-29 21:36:03,484 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-10-29 21:36:03,596 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.242 2025-10-29 21:36:03,602 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.129, 25; Got 10.20.0.129,25 2025-10-29 21:36:03,602 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.242 2025-10-29 21:36:03,602 - INFO - Validated routes of VM ctest-vn1_vm2-54840660 in all vrouters 2025-10-29 21:36:04,207 - DEBUG - Starting all layer2 verification in 10.0.0.33 Control Node 2025-10-29 21:36:04,328 - DEBUG - Layer2 route found for VM MAC 02:c4:ab:cb:b6:d4 in Control-node 10.0.0.33 2025-10-29 21:36:04,328 - DEBUG - L2 Label for VM ctest-vn1_vm2-54840660 same between Control-node 10.0.0.33 and Agent, Expected: 29, Seen: 29 2025-10-29 21:36:04,328 - DEBUG - Starting all layer2 verification in 10.0.0.38 Control Node 2025-10-29 21:36:04,448 - DEBUG - Layer2 route found for VM MAC 02:c4:ab:cb:b6:d4 in Control-node 10.0.0.38 2025-10-29 21:36:04,449 - DEBUG - L2 Label for VM ctest-vn1_vm2-54840660 same between Control-node 10.0.0.38 and Agent, Expected: 29, Seen: 29 2025-10-29 21:36:04,449 - DEBUG - Starting all layer2 verification in 10.0.0.254 Control Node 2025-10-29 21:36:04,567 - DEBUG - Layer2 route found for VM MAC 02:c4:ab:cb:b6:d4 in Control-node 10.0.0.254 2025-10-29 21:36:04,567 - DEBUG - L2 Label for VM ctest-vn1_vm2-54840660 same between Control-node 10.0.0.254 and Agent, Expected: 29, Seen: 29 2025-10-29 21:36:04,568 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-54840660 passed 2025-10-29 21:36:04,568 - DEBUG - Verifying the vm in opserver 2025-10-29 21:36:04,568 - DEBUG - Verifying in collector 10.0.0.33 ... 2025-10-29 21:36:04,568 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machine/f0e1612b-b17c-424e-8322-27fc096c3428?flat 2025-10-29 21:36:04,583 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91865949:c4abcbb6-d499-421d-8fdb-99866b42d406?flat 2025-10-29 21:36:04,605 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91865949:c4abcbb6-d499-421d-8fdb-99866b42d406?flat 2025-10-29 21:36:04,623 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machine/f0e1612b-b17c-424e-8322-27fc096c3428?flat 2025-10-29 21:36:04,639 - DEBUG - Verifying vm in vn uve 2025-10-29 21:36:04,639 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91865949:c4abcbb6-d499-421d-8fdb-99866b42d406?flat 2025-10-29 21:36:04,659 - DEBUG - VM uve shows interface as {'vn_uuid': 'c4d45393-8c58-4fa7-8953-3e3277504438', 'rx_vlan': 65535, 'mac_address': '02:c4:ab:cb:b6:d4', 'vm_name': 'ctest-vn1_vm2-54840660', 'gateway': '192.168.1.1', 'is_health_check_active': True, 'l2_active': True, 'label': 25, 'ip_address': '192.168.1.4', 'vm_uuid': 'f0e1612b-b17c-424e-8322-27fc096c3428', '__T': 1761773596526289, 'active': True, 'vhostuser_mode': 0, 'fixed_ip4_list': ['192.168.1.4'], 'ip6_address': '::', 'ip4_active': True, 'port_mirror_enabled': False, 'tx_vlan': 65535, 'ip6_active': False, 'admin_state': True, 'uuid': 'c4abcbb6-d499-421d-8fdb-99866b42d406', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884'} 2025-10-29 21:36:04,659 - DEBUG - VM uve shows ip address as ['192.168.1.4', '::'] 2025-10-29 21:36:04,659 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 2025-10-29 21:36:04,659 - INFO - Verifying through opserver in 10.0.0.254 2025-10-29 21:36:04,659 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884?flat 2025-10-29 21:36:04,685 - DEBUG - expected vm list f0e1612b-b17c-424e-8322-27fc096c3428 2025-10-29 21:36:04,685 - DEBUG - Extracted vm list ['f0e1612b-b17c-424e-8322-27fc096c3428'] 2025-10-29 21:36:04,685 - INFO - VM f0e1612b-b17c-424e-8322-27fc096c3428 is present in default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 2025-10-29 21:36:04,685 - DEBUG - Verifying vm in vrouter uve 2025-10-29 21:36:04,685 - DEBUG - Getting info from collector 10.0.0.33.. 2025-10-29 21:36:04,686 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machine/f0e1612b-b17c-424e-8322-27fc096c3428?flat 2025-10-29 21:36:04,701 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4527-1'] 2025-10-29 21:36:04,701 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4527-1:Compute:contrail-vrouter-agent:0?flat 2025-10-29 21:36:04,735 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-1?flat 2025-10-29 21:36:04,808 - DEBUG - VM f0e1612b-b17c-424e-8322-27fc096c3428 is present in vrouter an-jenkins-deploy-platform-ansible-os-4527-1 uve 2025-10-29 21:36:04,808 - DEBUG - Validated that VM ctest-vn1_vm2-54840660 is in Vrouter an-jenkins-deploy-platform-ansible-os-4527-1 UVE 2025-10-29 21:36:04,808 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-10-29 21:36:04,820 - DEBUG - Expected tap interface of VM uuid f0e1612b-b17c-424e-8322-27fc096c3428 is default-domain:ctest-TestBasicVMVN0-91865949:c4abcbb6-d499-421d-8fdb-99866b42d406 2025-10-29 21:36:04,820 - DEBUG - Expected VN of VM uuid f0e1612b-b17c-424e-8322-27fc096c3428 is default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 2025-10-29 21:36:04,820 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4527-1:Compute:contrail-vrouter-agent:0?flat 2025-10-29 21:36:04,853 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-1?flat 2025-10-29 21:36:04,925 - DEBUG - VM f0e1612b-b17c-424e-8322-27fc096c3428 is present in vrouter an-jenkins-deploy-platform-ansible-os-4527-1 uve 2025-10-29 21:36:04,925 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-91865949:c4abcbb6-d499-421d-8fdb-99866b42d406 of vm f0e1612b-b17c-424e-8322-27fc096c3428 is present in vrouter an-jenkins-deploy-platform-ansible-os-4527-1 uve 2025-10-29 21:36:04,925 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 of vm f0e1612b-b17c-424e-8322-27fc096c3428 is present in vrouter an-jenkins-deploy-platform-ansible-os-4527-1 uve 2025-10-29 21:36:04,925 - DEBUG - Verifying the f0e1612b-b17c-424e-8322-27fc096c3428 virtual network link through opserver 10.0.0.254 2025-10-29 21:36:04,925 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-machines 2025-10-29 21:36:04,940 - DEBUG - VM link and name as {'name': 'f0e1612b-b17c-424e-8322-27fc096c3428', 'href': 'http://10.0.0.254:8081/analytics/uves/virtual-machine/f0e1612b-b17c-424e-8322-27fc096c3428?flat'} 2025-10-29 21:36:04,940 - DEBUG - Verifying the f0e1612b-b17c-424e-8322-27fc096c3428 virtual network link through opserver 10.0.0.38 2025-10-29 21:36:04,940 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-10-29 21:36:04,953 - DEBUG - VM link and name as {'name': 'f0e1612b-b17c-424e-8322-27fc096c3428', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-machine/f0e1612b-b17c-424e-8322-27fc096c3428?flat'} 2025-10-29 21:36:04,953 - DEBUG - Verifying the f0e1612b-b17c-424e-8322-27fc096c3428 virtual network link through opserver 10.0.0.33 2025-10-29 21:36:04,953 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-10-29 21:36:04,965 - DEBUG - VM link and name as {'name': 'f0e1612b-b17c-424e-8322-27fc096c3428', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-machine/f0e1612b-b17c-424e-8322-27fc096c3428?flat'} 2025-10-29 21:36:04,965 - INFO - VM ctest-vn1_vm2-54840660 validations in Opserver passed 2025-10-29 21:36:04,965 - INFO - Checking the ping between the VM again 2025-10-29 21:36:04,965 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:36:04,965 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.242, gateway password: c0ntrail123 2025-10-29 21:36:04,965 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4 2025-10-29 21:36:07,466 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data. 64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=6.48 ms 64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.55 ms 64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.68 ms --- 192.168.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.557/3.244/6.486/2.293 ms 2025-10-29 21:36:07,466 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-53214933 passed 2025-10-29 21:36:07,466 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:36:07,466 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.241, gateway password: c0ntrail123 2025-10-29 21:36:07,466 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3 2025-10-29 21:36:10,073 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. 64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=3.06 ms 64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=2.04 ms 64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.73 ms --- 192.168.1.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2005ms rtt min/avg/max/mdev = 1.734/2.281/3.068/0.572 ms 2025-10-29 21:36:10,073 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-54840660 passed 2025-10-29 21:36:10,073 - INFO - Deleting VM ctest-vn1_vm2-54840660 2025-10-29 21:36:10,146 - DEBUG - Verifying in api server 10.0.0.33 2025-10-29 21:36:10,147 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/f0e1612b-b17c-424e-8322-27fc096c3428 2025-10-29 21:36:10,155 - DEBUG - VM ID f0e1612b-b17c-424e-8322-27fc096c3428 of VM ctest-vn1_vm2-54840660 is still found in API Server 2025-10-29 21:36:12,156 - DEBUG - Verifying in api server 10.0.0.33 2025-10-29 21:36:12,156 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/f0e1612b-b17c-424e-8322-27fc096c3428 2025-10-29 21:36:12,163 - DEBUG - Response Code: 404 2025-10-29 21:36:12,163 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/f0e1612b-b17c-424e-8322-27fc096c3428 2025-10-29 21:36:12,175 - DEBUG - Response Code: 404 2025-10-29 21:36:12,176 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/f0e1612b-b17c-424e-8322-27fc096c3428 2025-10-29 21:36:12,188 - DEBUG - Response Code: 404 2025-10-29 21:36:12,188 - INFO - VM ctest-vn1_vm2-54840660 is fully removed in API-Server 2025-10-29 21:36:12,353 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-10-29 21:36:12,414 - DEBUG - Validated that vrouter 10.0.0.241 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 2025-10-29 21:36:12,530 - DEBUG - VM route 192.168.1.4 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 of compute 10.0.0.242 2025-10-29 21:36:12,535 - DEBUG - Validated that vrouter 10.0.0.242 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 2025-10-29 21:36:12,535 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 2025-10-29 21:36:12,535 - INFO - VM ctest-vn1_vm2-54840660 is removed in Compute, and routes are removed in all compute nodes 2025-10-29 21:36:12,801 - INFO - Routes for VM ctest-vn1_vm2-54840660 is removed in all control-nodes 2025-10-29 21:36:12,801 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4527-1:Compute:contrail-vrouter-agent:0?flat 2025-10-29 21:36:12,849 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-1?flat 2025-10-29 21:36:12,884 - DEBUG - VM f0e1612b-b17c-424e-8322-27fc096c3428 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4527-1 uve 2025-10-29 21:36:12,884 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-29 21:36:12,884 - DEBUG - interface for vm f0e1612b-b17c-424e-8322-27fc096c3428 deleted from vrouter uve ... 2025-10-29 21:36:12,884 - DEBUG - Deleted interface not in error interface list ... 2025-10-29 21:36:15,887 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4527-1:Compute:contrail-vrouter-agent:0?flat 2025-10-29 21:36:15,920 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-1?flat 2025-10-29 21:36:15,948 - DEBUG - VM f0e1612b-b17c-424e-8322-27fc096c3428 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4527-1 uve 2025-10-29 21:36:15,948 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-29 21:36:15,948 - DEBUG - interface for vm f0e1612b-b17c-424e-8322-27fc096c3428 deleted from vrouter uve ... 2025-10-29 21:36:15,948 - DEBUG - Deleted interface not in error interface list ... 2025-10-29 21:36:18,949 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4527-1:Compute:contrail-vrouter-agent:0?flat 2025-10-29 21:36:18,982 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-1?flat 2025-10-29 21:36:19,010 - DEBUG - VM f0e1612b-b17c-424e-8322-27fc096c3428 is not present in vrouter an-jenkins-deploy-platform-ansible-os-4527-1 uve 2025-10-29 21:36:19,010 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-29 21:36:19,010 - DEBUG - interface for vm f0e1612b-b17c-424e-8322-27fc096c3428 deleted from vrouter uve ... 2025-10-29 21:36:19,010 - DEBUG - Deleted interface not in error interface list ... 2025-10-29 21:36:19,010 - DEBUG - Validated that VM f0e1612b-b17c-424e-8322-27fc096c3428 is removed in Vrouter UVE 2025-10-29 21:36:19,010 - DEBUG - Verifying through opserver in 10.0.0.254 2025-10-29 21:36:19,011 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884?flat 2025-10-29 21:36:19,034 - DEBUG - f0e1612b-b17c-424e-8322-27fc096c3428 BM not in default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 uve 2025-10-29 21:36:19,034 - DEBUG - Validated that VM f0e1612b-b17c-424e-8322-27fc096c3428 is not present in the VN UVE 2025-10-29 21:36:19,034 - DEBUG - Verifying the f0e1612b-b17c-424e-8322-27fc096c3428 virtual network link through opserver 10.0.0.254 2025-10-29 21:36:19,034 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-machines 2025-10-29 21:36:19,045 - INFO - f0e1612b-b17c-424e-8322-27fc096c3428 vm uve deleted from opserver 2025-10-29 21:36:19,045 - DEBUG - Verifying the f0e1612b-b17c-424e-8322-27fc096c3428 virtual network link through opserver 10.0.0.38 2025-10-29 21:36:19,045 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-10-29 21:36:19,056 - INFO - f0e1612b-b17c-424e-8322-27fc096c3428 vm uve deleted from opserver 2025-10-29 21:36:19,056 - DEBUG - Verifying the f0e1612b-b17c-424e-8322-27fc096c3428 virtual network link through opserver 10.0.0.33 2025-10-29 21:36:19,056 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-10-29 21:36:19,068 - INFO - f0e1612b-b17c-424e-8322-27fc096c3428 vm uve deleted from opserver 2025-10-29 21:36:19,068 - INFO - Validated that VM f0e1612b-b17c-424e-8322-27fc096c3428 is removed from Opserver 2025-10-29 21:36:19,068 - INFO - Deleting VM ctest-vn1_vm1-53214933 2025-10-29 21:36:19,137 - DEBUG - Verifying in api server 10.0.0.33 2025-10-29 21:36:19,138 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/ef51fcc9-8db7-40e5-8492-b9071d078e17 2025-10-29 21:36:19,145 - DEBUG - VM ID ef51fcc9-8db7-40e5-8492-b9071d078e17 of VM ctest-vn1_vm1-53214933 is still found in API Server 2025-10-29 21:36:21,146 - DEBUG - Verifying in api server 10.0.0.33 2025-10-29 21:36:21,147 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/ef51fcc9-8db7-40e5-8492-b9071d078e17 2025-10-29 21:36:21,154 - DEBUG - Response Code: 404 2025-10-29 21:36:21,154 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/ef51fcc9-8db7-40e5-8492-b9071d078e17 2025-10-29 21:36:21,160 - DEBUG - Response Code: 404 2025-10-29 21:36:21,160 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/ef51fcc9-8db7-40e5-8492-b9071d078e17 2025-10-29 21:36:21,165 - DEBUG - Response Code: 404 2025-10-29 21:36:21,166 - INFO - VM ctest-vn1_vm1-53214933 is fully removed in API-Server 2025-10-29 21:36:21,326 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-10-29 21:36:21,379 - DEBUG - Validated that vrouter 10.0.0.241 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 2025-10-29 21:36:21,390 - DEBUG - Validated that vrouter 10.0.0.242 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 2025-10-29 21:36:21,390 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 2025-10-29 21:36:21,390 - INFO - VM ctest-vn1_vm1-53214933 is removed in Compute, and routes are removed in all compute nodes 2025-10-29 21:36:21,623 - INFO - Routes for VM ctest-vn1_vm1-53214933 is removed in all control-nodes 2025-10-29 21:36:21,623 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4527-2:Compute:contrail-vrouter-agent:0?flat 2025-10-29 21:36:21,669 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-2?flat 2025-10-29 21:36:21,699 - DEBUG - VM ef51fcc9-8db7-40e5-8492-b9071d078e17 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4527-2 uve 2025-10-29 21:36:21,699 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-29 21:36:21,699 - DEBUG - interface for vm ef51fcc9-8db7-40e5-8492-b9071d078e17 deleted from vrouter uve ... 2025-10-29 21:36:21,699 - DEBUG - Deleted interface not in error interface list ... 2025-10-29 21:36:24,699 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4527-2:Compute:contrail-vrouter-agent:0?flat 2025-10-29 21:36:24,745 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-2?flat 2025-10-29 21:36:24,776 - DEBUG - VM ef51fcc9-8db7-40e5-8492-b9071d078e17 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4527-2 uve 2025-10-29 21:36:24,776 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-29 21:36:24,776 - DEBUG - interface for vm ef51fcc9-8db7-40e5-8492-b9071d078e17 deleted from vrouter uve ... 2025-10-29 21:36:24,777 - DEBUG - Deleted interface not in error interface list ... 2025-10-29 21:36:27,777 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4527-2:Compute:contrail-vrouter-agent:0?flat 2025-10-29 21:36:27,824 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-2?flat 2025-10-29 21:36:27,854 - DEBUG - VM ef51fcc9-8db7-40e5-8492-b9071d078e17 is not present in vrouter an-jenkins-deploy-platform-ansible-os-4527-2 uve 2025-10-29 21:36:27,855 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-29 21:36:27,855 - DEBUG - interface for vm ef51fcc9-8db7-40e5-8492-b9071d078e17 deleted from vrouter uve ... 2025-10-29 21:36:27,855 - DEBUG - Deleted interface not in error interface list ... 2025-10-29 21:36:27,855 - DEBUG - Validated that VM ef51fcc9-8db7-40e5-8492-b9071d078e17 is removed in Vrouter UVE 2025-10-29 21:36:27,855 - DEBUG - Verifying through opserver in 10.0.0.254 2025-10-29 21:36:27,855 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884?flat 2025-10-29 21:36:27,872 - DEBUG - ef51fcc9-8db7-40e5-8492-b9071d078e17 BM not in default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-81904884 uve 2025-10-29 21:36:27,872 - DEBUG - Validated that VM ef51fcc9-8db7-40e5-8492-b9071d078e17 is not present in the VN UVE 2025-10-29 21:36:27,872 - DEBUG - Verifying the ef51fcc9-8db7-40e5-8492-b9071d078e17 virtual network link through opserver 10.0.0.254 2025-10-29 21:36:27,872 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-machines 2025-10-29 21:36:27,886 - INFO - ef51fcc9-8db7-40e5-8492-b9071d078e17 vm uve deleted from opserver 2025-10-29 21:36:27,886 - DEBUG - Verifying the ef51fcc9-8db7-40e5-8492-b9071d078e17 virtual network link through opserver 10.0.0.38 2025-10-29 21:36:27,886 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-10-29 21:36:27,899 - INFO - ef51fcc9-8db7-40e5-8492-b9071d078e17 vm uve deleted from opserver 2025-10-29 21:36:27,899 - DEBUG - Verifying the ef51fcc9-8db7-40e5-8492-b9071d078e17 virtual network link through opserver 10.0.0.33 2025-10-29 21:36:27,899 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-10-29 21:36:27,911 - INFO - ef51fcc9-8db7-40e5-8492-b9071d078e17 vm uve deleted from opserver 2025-10-29 21:36:27,911 - INFO - Validated that VM ef51fcc9-8db7-40e5-8492-b9071d078e17 is removed from Opserver 2025-10-29 21:36:27,911 - INFO - Deleting VN ctest-vn1-81904884 2025-10-29 21:36:28,138 - DEBUG - Response for deleting network () 2025-10-29 21:36:28,138 - DEBUG - Requesting: http://10.0.0.254:8082/routing-instance/798e55d5-8d6a-469c-b488-a0236a8ca72b 2025-10-29 21:36:28,143 - DEBUG - Response Code: 404 2025-10-29 21:36:28,144 - DEBUG - Requesting: http://10.0.0.254:8082/domains 2025-10-29 21:36:28,151 - DEBUG - Requesting: http://10.0.0.254:8082/domain/a090225f-9b9e-4366-926c-6397a8c195bb 2025-10-29 21:36:28,217 - DEBUG - Requesting: http://10.0.0.254:8082/project/9b488c42-e15b-4391-a1ef-0c96a0dda6cf 2025-10-29 21:36:28,313 - INFO - Validated that VN ctest-vn1-81904884 is not found in API Server 2025-10-29 21:36:28,335 - DEBUG - VN ctest-vn1-81904884 is not present in Agent 10.0.0.241 2025-10-29 21:36:28,345 - DEBUG - VN ctest-vn1-81904884 is not present in Agent 10.0.0.242 2025-10-29 21:36:28,345 - INFO - Validated that VN ctest-vn1-81904884 is not in any agent 2025-10-29 21:36:28,382 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-81904884 info 2025-10-29 21:36:29,694 - DEBUG - Skipping xmpp flap check 2025-10-29 21:36:29,694 - INFO - END TEST : test_control_node_switchover : PASSED[0:03:25] 2025-10-29 21:36:29,694 - INFO - -------------------------------------------------------------------------------- 2025-10-29 21:36:29,696 - INFO - ================================================================================ 2025-10-29 21:36:29,696 - INFO - STARTING TEST : test_process_restart_in_policy_between_vns 2025-10-29 21:36:29,696 - INFO - TEST DESCRIPTION : Test to validate that with policy having rule to check icmp fwding between VMs on different VNs , ping between VMs should pass with process restarts 1. Pick 2 VN's from resource pool which has one VM each 2. Create policy with icmp allow rule between those VN's and bind it networks 3. Ping from one VM to another VM 4. Restart process 'vrouter' and 'control' on setup 5. Ping again between VM's after process restart Pass criteria: Step 2,3,4 and 5 should pass 2025-10-29 21:36:30,982 - DEBUG - Skipping xmpp flap check 2025-10-29 21:36:30,982 - INFO - Initial checks done. Running the testcase now 2025-10-29 21:36:30,982 - INFO - 2025-10-29 21:36:30,989 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-10-29 21:36:31,306 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-10-29 21:36:31,858 - INFO - Created VN ctest-vn1-38715447, UUID :2710a27a-333f-48f1-9b07-63c28e1b45ec 2025-10-29 21:36:32,198 - DEBUG - Requesting: http://10.0.0.254:8082/domains 2025-10-29 21:36:32,204 - DEBUG - Requesting: http://10.0.0.254:8082/domain/a090225f-9b9e-4366-926c-6397a8c195bb 2025-10-29 21:36:32,216 - DEBUG - Requesting: http://10.0.0.254:8082/project/9b488c42-e15b-4391-a1ef-0c96a0dda6cf 2025-10-29 21:36:32,330 - DEBUG - Requesting: http://10.0.0.254:8082/virtual-network/2710a27a-333f-48f1-9b07-63c28e1b45ec 2025-10-29 21:36:32,346 - DEBUG - Requesting: http://10.0.0.254:8082/virtual-network/2710a27a-333f-48f1-9b07-63c28e1b45ec 2025-10-29 21:36:32,359 - DEBUG - Requesting: http://10.0.0.254:8082/routing-instance/3fef5723-58fa-4212-85fe-12547afd0ca2 2025-10-29 21:36:32,371 - DEBUG - Requesting: http://10.0.0.254:8082/routing-instance/3fef5723-58fa-4212-85fe-12547afd0ca2 2025-10-29 21:36:32,383 - DEBUG - Requesting: http://10.0.0.254:8082/route-target/a3f848ee-adbc-4b02-98ff-3f7dd963c939 2025-10-29 21:36:32,392 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-29 21:36:32,392 - DEBUG - Requesting: http://10.0.0.254:8082/virtual-network/2710a27a-333f-48f1-9b07-63c28e1b45ec 2025-10-29 21:36:32,405 - DEBUG - Requesting: http://10.0.0.254:8082/routing-instance/3fef5723-58fa-4212-85fe-12547afd0ca2 2025-10-29 21:36:32,427 - INFO - Verified VN network id 15 for VN 2710a27a-333f-48f1-9b07-63c28e1b45ec 2025-10-29 21:36:32,427 - INFO - Verifications in API Server for VN ctest-vn1-38715447 passed 2025-10-29 21:36:32,427 - DEBUG - Requesting: http://10.0.0.254:8082/virtual-network/2710a27a-333f-48f1-9b07-63c28e1b45ec 2025-10-29 21:36:32,437 - DEBUG - Requesting: http://10.0.0.254:8082/routing-instance/3fef5723-58fa-4212-85fe-12547afd0ca2 2025-10-29 21:36:32,445 - DEBUG - Requesting: http://10.0.0.254:8082/routing-instance/3fef5723-58fa-4212-85fe-12547afd0ca2 2025-10-29 21:36:32,454 - DEBUG - Requesting: http://10.0.0.254:8082/route-target/a3f848ee-adbc-4b02-98ff-3f7dd963c939 2025-10-29 21:36:32,467 - DEBUG - Control-node 10.0.0.254 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91865949', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '2814928412836055281', 'uuid-lslong': '11171007087974761964'}, 'enable': 'true', 'created': '2025-10-29T21:36:31', 'last-modified': '2025-10-29T21:36:31', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.448354'} 2025-10-29 21:36:32,474 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-29 21:36:32,479 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91865949', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '2814928412836055281', 'uuid-lslong': '11171007087974761964'}, 'enable': 'true', 'created': '2025-10-29T21:36:31', 'last-modified': '2025-10-29T21:36:31', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.477601'} 2025-10-29 21:36:32,484 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-29 21:36:32,490 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91865949', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '2814928412836055281', 'uuid-lslong': '11171007087974761964'}, 'enable': 'true', 'created': '2025-10-29T21:36:31', 'last-modified': '2025-10-29T21:36:31', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.483762'} 2025-10-29 21:36:32,496 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-29 21:36:32,496 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-38715447 passed 2025-10-29 21:36:32,496 - DEBUG - ====Verifying policy data for ctest-vn1-38715447 in API_Server ====== 2025-10-29 21:36:32,496 - DEBUG - Requesting: http://10.0.0.254:8082/domains 2025-10-29 21:36:32,503 - DEBUG - Requesting: http://10.0.0.254:8082/domain/a090225f-9b9e-4366-926c-6397a8c195bb 2025-10-29 21:36:32,516 - DEBUG - Requesting: http://10.0.0.254:8082/project/9b488c42-e15b-4391-a1ef-0c96a0dda6cf 2025-10-29 21:36:32,632 - DEBUG - Requesting: http://10.0.0.254:8082/virtual-network/2710a27a-333f-48f1-9b07-63c28e1b45ec 2025-10-29 21:36:32,645 - DEBUG - =>VN ctest-vn1-38715447 has no policy to be verified 2025-10-29 21:36:32,645 - DEBUG - Verifying the vn in opserver 2025-10-29 21:36:32,645 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 virtual network link through opserver 10.0.0.254 2025-10-29 21:36:32,645 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-networks 2025-10-29 21:36:32,659 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447', 'href': 'http://10.0.0.254:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447?flat'} 2025-10-29 21:36:32,659 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 virtual network link through opserver 10.0.0.38 2025-10-29 21:36:32,659 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-10-29 21:36:32,672 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447?flat'} 2025-10-29 21:36:32,673 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 virtual network link through opserver 10.0.0.33 2025-10-29 21:36:32,673 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2025-10-29 21:36:32,685 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447?flat'} 2025-10-29 21:36:32,685 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 is found in opserver 2025-10-29 21:36:32,715 - DEBUG - Do not have enough data to verify VN in agent 2025-10-29 21:36:32,730 - DEBUG - VRF ids for VN ctest-vn1-38715447: {} 2025-10-29 21:36:33,434 - INFO - Created VN ctest-vn2-07183641, UUID :36b37917-8e24-4e82-88a0-0e8a8805bf6b 2025-10-29 21:36:33,767 - DEBUG - Requesting: http://10.0.0.254:8082/domains 2025-10-29 21:36:33,774 - DEBUG - Requesting: http://10.0.0.254:8082/domain/a090225f-9b9e-4366-926c-6397a8c195bb 2025-10-29 21:36:33,787 - DEBUG - Requesting: http://10.0.0.254:8082/project/9b488c42-e15b-4391-a1ef-0c96a0dda6cf 2025-10-29 21:36:33,879 - DEBUG - Requesting: http://10.0.0.254:8082/virtual-network/36b37917-8e24-4e82-88a0-0e8a8805bf6b 2025-10-29 21:36:33,897 - DEBUG - Requesting: http://10.0.0.254:8082/virtual-network/36b37917-8e24-4e82-88a0-0e8a8805bf6b 2025-10-29 21:36:33,908 - DEBUG - Requesting: http://10.0.0.254:8082/routing-instance/9d3258e4-37c5-48fb-ac56-61dff558b053 2025-10-29 21:36:33,916 - DEBUG - Requesting: http://10.0.0.254:8082/routing-instance/9d3258e4-37c5-48fb-ac56-61dff558b053 2025-10-29 21:36:33,924 - DEBUG - Requesting: http://10.0.0.254:8082/route-target/6046030a-5e7b-45ef-9937-7eb847beb2ae 2025-10-29 21:36:33,933 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-10-29 21:36:33,933 - DEBUG - Requesting: http://10.0.0.254:8082/virtual-network/36b37917-8e24-4e82-88a0-0e8a8805bf6b 2025-10-29 21:36:33,944 - DEBUG - Requesting: http://10.0.0.254:8082/routing-instance/9d3258e4-37c5-48fb-ac56-61dff558b053 2025-10-29 21:36:33,959 - INFO - Verified VN network id 16 for VN 36b37917-8e24-4e82-88a0-0e8a8805bf6b 2025-10-29 21:36:33,959 - INFO - Verifications in API Server for VN ctest-vn2-07183641 passed 2025-10-29 21:36:33,959 - DEBUG - Requesting: http://10.0.0.254:8082/virtual-network/36b37917-8e24-4e82-88a0-0e8a8805bf6b 2025-10-29 21:36:33,972 - DEBUG - Requesting: http://10.0.0.254:8082/routing-instance/9d3258e4-37c5-48fb-ac56-61dff558b053 2025-10-29 21:36:33,979 - DEBUG - Requesting: http://10.0.0.254:8082/routing-instance/9d3258e4-37c5-48fb-ac56-61dff558b053 2025-10-29 21:36:33,986 - DEBUG - Requesting: http://10.0.0.254:8082/route-target/6046030a-5e7b-45ef-9937-7eb847beb2ae 2025-10-29 21:36:33,998 - DEBUG - Control-node 10.0.0.254 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91865949', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3941627240955268738', 'uuid-lslong': '9844884773582258027'}, 'enable': 'true', 'created': '2025-10-29T21:36:33', 'last-modified': '2025-10-29T21:36:33', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.415226'} 2025-10-29 21:36:34,004 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-10-29 21:36:34,009 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91865949', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3941627240955268738', 'uuid-lslong': '9844884773582258027'}, 'enable': 'true', 'created': '2025-10-29T21:36:33', 'last-modified': '2025-10-29T21:36:33', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.424091'} 2025-10-29 21:36:34,015 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-10-29 21:36:34,021 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91865949', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3941627240955268738', 'uuid-lslong': '9844884773582258027'}, 'enable': 'true', 'created': '2025-10-29T21:36:33', 'last-modified': '2025-10-29T21:36:33', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.438092'} 2025-10-29 21:36:34,028 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-10-29 21:36:34,028 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn2-07183641 passed 2025-10-29 21:36:34,028 - DEBUG - ====Verifying policy data for ctest-vn2-07183641 in API_Server ====== 2025-10-29 21:36:34,028 - DEBUG - Requesting: http://10.0.0.254:8082/domains 2025-10-29 21:36:34,034 - DEBUG - Requesting: http://10.0.0.254:8082/domain/a090225f-9b9e-4366-926c-6397a8c195bb 2025-10-29 21:36:34,053 - DEBUG - Requesting: http://10.0.0.254:8082/project/9b488c42-e15b-4391-a1ef-0c96a0dda6cf 2025-10-29 21:36:34,162 - DEBUG - Requesting: http://10.0.0.254:8082/virtual-network/36b37917-8e24-4e82-88a0-0e8a8805bf6b 2025-10-29 21:36:34,174 - DEBUG - =>VN ctest-vn2-07183641 has no policy to be verified 2025-10-29 21:36:34,174 - DEBUG - Verifying the vn in opserver 2025-10-29 21:36:34,174 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 virtual network link through opserver 10.0.0.254 2025-10-29 21:36:34,174 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-networks 2025-10-29 21:36:34,187 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641', 'href': 'http://10.0.0.254:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641?flat'} 2025-10-29 21:36:34,187 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 virtual network link through opserver 10.0.0.38 2025-10-29 21:36:34,187 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-10-29 21:36:34,198 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641?flat'} 2025-10-29 21:36:34,198 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 virtual network link through opserver 10.0.0.33 2025-10-29 21:36:34,198 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2025-10-29 21:36:34,210 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641?flat'} 2025-10-29 21:36:34,210 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 is found in opserver 2025-10-29 21:36:34,230 - DEBUG - Do not have enough data to verify VN in agent 2025-10-29 21:36:34,239 - DEBUG - VRF ids for VN ctest-vn2-07183641: {} 2025-10-29 21:36:35,833 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4527-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4527-2) 2025-10-29 21:36:37,239 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4527-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4527-1) 2025-10-29 21:36:37,239 - INFO - Waiting for VM ctest-vn1_vm1-21582819 to be up.. 2025-10-29 21:36:37,325 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-10-29 21:36:42,406 - DEBUG - VM is in ACTIVE state now 2025-10-29 21:36:42,406 - INFO - VM name : ctest-vn1_vm1-21582819 2025-10-29 21:36:42,483 - DEBUG - VM ctest-vn1_vm1-21582819 ID is e4da1275-21de-4187-9b91-35180443130f 2025-10-29 21:36:42,484 - DEBUG - VM ctest-vn1_vm1-21582819 launched on Node an-jenkins-deploy-platform-ansible-os-4527-2 2025-10-29 21:36:42,564 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/e4da1275-21de-4187-9b91-35180443130f 2025-10-29 21:36:42,573 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine-interface/6c8d3522-d43c-4fcc-a90f-9e4db4036e82 2025-10-29 21:36:45,752 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1027ms') 2025-10-29 21:36:45,752 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-21582819 failed! 2025-10-29 21:36:45,766 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 is 192.168.1.254 and allocation pool is NOT set 2025-10-29 21:36:49,857 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1027ms') 2025-10-29 21:36:49,858 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-21582819 failed! 2025-10-29 21:36:49,870 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 is 192.168.1.254 and allocation pool is NOT set 2025-10-29 21:36:53,956 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2025-10-29 21:36:53,956 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-21582819 failed! 2025-10-29 21:36:53,972 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 is 192.168.1.254 and allocation pool is NOT set 2025-10-29 21:36:58,048 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms') 2025-10-29 21:36:58,048 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-21582819 failed! 2025-10-29 21:36:58,060 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 is 192.168.1.254 and allocation pool is NOT set 2025-10-29 21:37:02,140 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-10-29 21:37:02,140 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-21582819 failed! 2025-10-29 21:37:02,153 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 is 192.168.1.254 and allocation pool is NOT set 2025-10-29 21:37:06,242 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1027ms') 2025-10-29 21:37:06,242 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-21582819 failed! 2025-10-29 21:37:06,256 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 is 192.168.1.254 and allocation pool is NOT set 2025-10-29 21:37:10,329 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1008ms') 2025-10-29 21:37:10,330 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-21582819 failed! 2025-10-29 21:37:10,343 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 is 192.168.1.254 and allocation pool is NOT set 2025-10-29 21:37:14,429 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=9.36 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 1 received, 50% packet loss, time 1023ms\r\nrtt min/avg/max/mdev = 9.361/9.361/9.361/0.000 ms') 2025-10-29 21:37:14,429 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-21582819 passed 2025-10-29 21:37:14,494 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:37:14,494 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-21582819, IP 192.168.1.252, Port 22 2025-10-29 21:37:14,559 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-29 21:37:14,642 - DEBUG - VM ctest-vn1_vm1-21582819 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-29 21:37:19,643 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:37:19,643 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-21582819, IP 192.168.1.252, Port 22 2025-10-29 21:37:19,714 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-29 21:37:19,803 - DEBUG - VM ctest-vn1_vm1-21582819 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-29 21:37:24,803 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:37:24,803 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-21582819, IP 192.168.1.252, Port 22 2025-10-29 21:37:24,871 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-29 21:37:24,956 - DEBUG - VM ctest-vn1_vm1-21582819 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-29 21:37:29,957 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:37:29,958 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-21582819, IP 192.168.1.252, Port 22 2025-10-29 21:37:30,013 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-29 21:37:30,106 - DEBUG - VM ctest-vn1_vm1-21582819 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-29 21:37:35,107 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:37:35,108 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-21582819, IP 192.168.1.252, Port 22 2025-10-29 21:37:35,276 - DEBUG - VM ctest-vn1_vm1-21582819 is ready for SSH connections 2025-10-29 21:37:35,277 - INFO - Waiting for VM ctest-vn2_vm1-43980660 to be up.. 2025-10-29 21:37:35,361 - DEBUG - VM is in ACTIVE state now 2025-10-29 21:37:35,361 - INFO - VM name : ctest-vn2_vm1-43980660 2025-10-29 21:37:35,436 - DEBUG - VM ctest-vn2_vm1-43980660 ID is 013cd078-d26d-4614-ac70-b9d898f8061a 2025-10-29 21:37:35,436 - DEBUG - VM ctest-vn2_vm1-43980660 launched on Node an-jenkins-deploy-platform-ansible-os-4527-1 2025-10-29 21:37:35,517 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/013cd078-d26d-4614-ac70-b9d898f8061a 2025-10-29 21:37:35,526 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine-interface/b77e8099-8a29-4319-abbe-bddecbda5993 2025-10-29 21:37:36,719 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=7.04 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=2.72 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1000ms\r\nrtt min/avg/max/mdev = 2.718/4.878/7.039/2.160 ms') 2025-10-29 21:37:36,719 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn2_vm1-43980660 passed 2025-10-29 21:37:36,787 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:37:36,788 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-43980660, IP 192.168.2.252, Port 22 2025-10-29 21:37:36,860 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-29 21:37:36,940 - DEBUG - VM ctest-vn2_vm1-43980660 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-29 21:37:41,941 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:37:41,941 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-43980660, IP 192.168.2.252, Port 22 2025-10-29 21:37:42,155 - DEBUG - VM ctest-vn2_vm1-43980660 is ready for SSH connections 2025-10-29 21:37:42,155 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:37:42,155 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.252, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.242, gateway password: c0ntrail123 2025-10-29 21:37:42,155 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.252 2025-10-29 21:38:27,751 - DEBUG - PING 192.168.2.252 (192.168.2.252) 56(84) bytes of data. 64 bytes from 192.168.2.252: icmp_req=1 ttl=63 time=14.2 ms 64 bytes from 192.168.2.252: icmp_req=2 ttl=63 time=1.96 ms 64 bytes from 192.168.2.252: icmp_req=3 ttl=63 time=1.89 ms --- 192.168.2.252 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.894/6.033/14.245/5.806 ms 2025-10-29 21:38:27,751 - INFO - Ping to IP 192.168.2.252 from VM ctest-vn1_vm1-21582819 passed 2025-10-29 21:38:31,329 - INFO - 10.0.0.241 2025-10-29 21:38:31,334 - INFO - agent:initializing 2025-10-29 21:38:31,334 - DEBUG - defaultdict(, {'10.0.0.241': {'agent': {'status': 'initializing', 'description': None}}}) 2025-10-29 21:38:31,334 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-29 21:38:36,497 - INFO - 10.0.0.241 2025-10-29 21:38:36,507 - INFO - agent:active 2025-10-29 21:38:36,507 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.241 2025-10-29 21:38:39,880 - INFO - 10.0.0.242 2025-10-29 21:38:39,884 - INFO - agent:initializing 2025-10-29 21:38:39,884 - DEBUG - defaultdict(, {'10.0.0.242': {'agent': {'status': 'initializing', 'description': None}}}) 2025-10-29 21:38:39,884 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-29 21:38:44,973 - INFO - 10.0.0.242 2025-10-29 21:38:44,978 - INFO - agent:active 2025-10-29 21:38:44,978 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.242 2025-10-29 21:38:45,580 - INFO - 10.0.0.254 2025-10-29 21:38:45,583 - INFO - control:initializing 2025-10-29 21:38:45,583 - DEBUG - defaultdict(, {'10.0.0.254': {'control': {'status': 'initializing', 'description': None}}}) 2025-10-29 21:38:45,583 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-29 21:38:50,687 - INFO - 10.0.0.254 2025-10-29 21:38:50,693 - INFO - control:active 2025-10-29 21:38:50,693 - INFO - Contrail services ['control'] are up on nodes 10.0.0.254 2025-10-29 21:38:51,071 - INFO - 10.0.0.38 2025-10-29 21:38:51,075 - INFO - control:initializing 2025-10-29 21:38:51,075 - DEBUG - defaultdict(, {'10.0.0.38': {'control': {'status': 'initializing', 'description': None}}}) 2025-10-29 21:38:51,076 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-29 21:38:56,177 - INFO - 10.0.0.38 2025-10-29 21:38:56,184 - INFO - control:active 2025-10-29 21:38:56,184 - INFO - Contrail services ['control'] are up on nodes 10.0.0.38 2025-10-29 21:38:56,574 - INFO - 10.0.0.33 2025-10-29 21:38:56,579 - INFO - control:initializing 2025-10-29 21:38:56,579 - DEBUG - defaultdict(, {'10.0.0.33': {'control': {'status': 'initializing', 'description': None}}}) 2025-10-29 21:38:56,579 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-29 21:39:01,678 - INFO - 10.0.0.33 2025-10-29 21:39:01,684 - INFO - control:active 2025-10-29 21:39:01,684 - INFO - Contrail services ['control'] are up on nodes 10.0.0.33 2025-10-29 21:39:02,515 - INFO - 10.0.0.254 2025-10-29 21:39:02,519 - INFO - api-server:initializing 2025-10-29 21:39:02,519 - DEBUG - defaultdict(, {'10.0.0.254': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-29 21:39:02,519 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-29 21:39:07,631 - INFO - 10.0.0.254 2025-10-29 21:39:07,634 - INFO - api-server:initializing 2025-10-29 21:39:07,634 - DEBUG - defaultdict(, {'10.0.0.254': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-29 21:39:07,634 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-10-29 21:39:12,771 - INFO - 10.0.0.254 2025-10-29 21:39:12,782 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2025-10-29 21:39:12,784 - DEBUG - defaultdict(, {'10.0.0.254': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2025-10-29 21:39:12,784 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-10-29 21:39:17,898 - INFO - 10.0.0.254 2025-10-29 21:39:17,903 - INFO - api-server:active 2025-10-29 21:39:17,904 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.254 2025-10-29 21:39:18,526 - INFO - 10.0.0.38 2025-10-29 21:39:18,530 - INFO - api-server:initializing 2025-10-29 21:39:18,530 - DEBUG - defaultdict(, {'10.0.0.38': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-29 21:39:18,530 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-29 21:39:23,631 - INFO - 10.0.0.38 2025-10-29 21:39:23,635 - INFO - api-server:initializing 2025-10-29 21:39:23,635 - DEBUG - defaultdict(, {'10.0.0.38': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-29 21:39:23,635 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-10-29 21:39:28,726 - INFO - 10.0.0.38 2025-10-29 21:39:28,731 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2025-10-29 21:39:28,732 - DEBUG - defaultdict(, {'10.0.0.38': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2025-10-29 21:39:28,732 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-10-29 21:39:33,821 - INFO - 10.0.0.38 2025-10-29 21:39:33,827 - INFO - api-server:active 2025-10-29 21:39:33,827 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.38 2025-10-29 21:39:34,407 - INFO - 10.0.0.33 2025-10-29 21:39:34,411 - INFO - api-server:initializing 2025-10-29 21:39:34,411 - DEBUG - defaultdict(, {'10.0.0.33': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-29 21:39:34,411 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-10-29 21:39:39,531 - INFO - 10.0.0.33 2025-10-29 21:39:39,535 - INFO - api-server:initializing 2025-10-29 21:39:39,535 - DEBUG - defaultdict(, {'10.0.0.33': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-10-29 21:39:39,535 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-10-29 21:39:44,634 - INFO - 10.0.0.33 2025-10-29 21:39:44,639 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2025-10-29 21:39:44,639 - DEBUG - defaultdict(, {'10.0.0.33': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2025-10-29 21:39:44,639 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-10-29 21:39:49,726 - INFO - 10.0.0.33 2025-10-29 21:39:49,731 - INFO - api-server:active 2025-10-29 21:39:49,731 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.33 2025-10-29 21:39:49,833 - INFO - 10.0.0.254 2025-10-29 21:39:49,833 - INFO - config 2025-10-29 21:39:49,839 - INFO - config-nodemgr:active 2025-10-29 21:39:49,844 - INFO - api-server:active 2025-10-29 21:39:49,847 - INFO - schema:backup 2025-10-29 21:39:49,850 - INFO - svc-monitor:backup 2025-10-29 21:39:49,853 - INFO - device-manager:backup 2025-10-29 21:39:49,853 - INFO - config-database 2025-10-29 21:39:49,854 - INFO - config-cassandra:active 2025-10-29 21:39:49,854 - INFO - config-zookeeper:active 2025-10-29 21:39:49,854 - INFO - config-rabbitmq:active 2025-10-29 21:39:49,854 - INFO - control 2025-10-29 21:39:49,858 - INFO - control-nodemgr:active 2025-10-29 21:39:49,863 - INFO - control:active 2025-10-29 21:39:49,863 - INFO - named:active 2025-10-29 21:39:49,867 - INFO - dns:active 2025-10-29 21:39:49,868 - INFO - analytics 2025-10-29 21:39:49,872 - INFO - analytics-nodemgr:active 2025-10-29 21:39:49,875 - INFO - analytics-api:active 2025-10-29 21:39:49,881 - INFO - collector:active 2025-10-29 21:39:49,881 - INFO - analytics-database 2025-10-29 21:39:49,881 - INFO - analytics-cassandra:active 2025-10-29 21:39:49,886 - INFO - analyticsdb-nodemgr:active 2025-10-29 21:39:49,892 - INFO - query-engine:active 2025-10-29 21:39:49,892 - INFO - webui 2025-10-29 21:39:49,892 - INFO - webui:active 2025-10-29 21:39:49,892 - INFO - webui-middleware:active 2025-10-29 21:39:49,892 - INFO - redis:active 2025-10-29 21:39:49,892 - INFO - analytics_snmp 2025-10-29 21:39:49,897 - INFO - snmp-collector:active 2025-10-29 21:39:49,897 - INFO - snmp-topology:active 2025-10-29 21:39:49,897 - INFO - analytics_alarm 2025-10-29 21:39:49,898 - INFO - alarmgen:active 2025-10-29 21:39:49,984 - INFO - 10.0.0.38 2025-10-29 21:39:49,985 - INFO - config 2025-10-29 21:39:49,990 - INFO - config-nodemgr:active 2025-10-29 21:39:49,995 - INFO - api-server:active 2025-10-29 21:39:50,000 - INFO - schema:active 2025-10-29 21:39:50,005 - INFO - svc-monitor:active 2025-10-29 21:39:50,009 - INFO - device-manager:active 2025-10-29 21:39:50,009 - INFO - config-database 2025-10-29 21:39:50,009 - INFO - config-cassandra:active 2025-10-29 21:39:50,009 - INFO - config-zookeeper:active 2025-10-29 21:39:50,009 - INFO - config-rabbitmq:active 2025-10-29 21:39:50,009 - INFO - control 2025-10-29 21:39:50,014 - INFO - control-nodemgr:active 2025-10-29 21:39:50,018 - INFO - control:active 2025-10-29 21:39:50,019 - INFO - named:active 2025-10-29 21:39:50,033 - INFO - dns:active 2025-10-29 21:39:50,033 - INFO - analytics 2025-10-29 21:39:50,039 - INFO - analytics-nodemgr:active 2025-10-29 21:39:50,044 - INFO - analytics-api:active 2025-10-29 21:39:50,049 - INFO - collector:active 2025-10-29 21:39:50,050 - INFO - analytics-database 2025-10-29 21:39:50,050 - INFO - analytics-cassandra:active 2025-10-29 21:39:50,055 - INFO - analyticsdb-nodemgr:active 2025-10-29 21:39:50,068 - INFO - query-engine:active 2025-10-29 21:39:50,068 - INFO - webui 2025-10-29 21:39:50,068 - INFO - webui:active 2025-10-29 21:39:50,068 - INFO - webui-middleware:active 2025-10-29 21:39:50,068 - INFO - redis:active 2025-10-29 21:39:50,068 - INFO - analytics_snmp 2025-10-29 21:39:50,074 - INFO - snmp-collector:active 2025-10-29 21:39:50,074 - INFO - snmp-topology:active 2025-10-29 21:39:50,074 - INFO - analytics_alarm 2025-10-29 21:39:50,074 - INFO - alarmgen:active 2025-10-29 21:39:50,163 - INFO - 10.0.0.33 2025-10-29 21:39:50,163 - INFO - config 2025-10-29 21:39:50,169 - INFO - config-nodemgr:active 2025-10-29 21:39:50,173 - INFO - api-server:active 2025-10-29 21:39:50,176 - INFO - schema:backup 2025-10-29 21:39:50,179 - INFO - svc-monitor:backup 2025-10-29 21:39:50,183 - INFO - device-manager:backup 2025-10-29 21:39:50,183 - INFO - config-database 2025-10-29 21:39:50,183 - INFO - config-cassandra:active 2025-10-29 21:39:50,183 - INFO - config-zookeeper:active 2025-10-29 21:39:50,183 - INFO - config-rabbitmq:active 2025-10-29 21:39:50,184 - INFO - control 2025-10-29 21:39:50,188 - INFO - control-nodemgr:active 2025-10-29 21:39:50,194 - INFO - control:active 2025-10-29 21:39:50,194 - INFO - named:active 2025-10-29 21:39:50,199 - INFO - dns:active 2025-10-29 21:39:50,199 - INFO - analytics 2025-10-29 21:39:50,204 - INFO - analytics-nodemgr:active 2025-10-29 21:39:50,210 - INFO - analytics-api:active 2025-10-29 21:39:50,215 - INFO - collector:active 2025-10-29 21:39:50,215 - INFO - analytics-database 2025-10-29 21:39:50,215 - INFO - analytics-cassandra:active 2025-10-29 21:39:50,220 - INFO - analyticsdb-nodemgr:active 2025-10-29 21:39:50,226 - INFO - query-engine:active 2025-10-29 21:39:50,226 - INFO - webui 2025-10-29 21:39:50,226 - INFO - webui:active 2025-10-29 21:39:50,226 - INFO - webui-middleware:active 2025-10-29 21:39:50,226 - INFO - redis:active 2025-10-29 21:39:50,226 - INFO - analytics_snmp 2025-10-29 21:39:50,230 - INFO - snmp-collector:active 2025-10-29 21:39:50,230 - INFO - snmp-topology:active 2025-10-29 21:39:50,230 - INFO - analytics_alarm 2025-10-29 21:39:50,230 - INFO - alarmgen:active 2025-10-29 21:39:50,340 - INFO - 10.0.0.241 2025-10-29 21:39:50,340 - INFO - vrouter 2025-10-29 21:39:50,349 - INFO - vrouter-nodemgr:active 2025-10-29 21:39:50,360 - INFO - agent:active 2025-10-29 21:39:50,449 - INFO - 10.0.0.242 2025-10-29 21:39:50,449 - INFO - vrouter 2025-10-29 21:39:50,454 - INFO - vrouter-nodemgr:active 2025-10-29 21:39:50,459 - INFO - agent:active 2025-10-29 21:39:50,459 - INFO - All the contrail services are up on all nodes 2025-10-29 21:39:50,459 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-1?flat 2025-10-29 21:39:50,848 - INFO - an-jenkins-deploy-platform-ansible-os-4527-1:contrail-vrouter-agent module connection to ['10.20.0.14:5269'] servers UP 2025-10-29 21:39:50,848 - INFO - an-jenkins-deploy-platform-ansible-os-4527-1 connected to xmpp 10.20.0.14 2025-10-29 21:39:50,848 - WARNING - an-jenkins-deploy-platform-ansible-os-4527-1:contrail-vrouter-agent module connection to ['10.20.0.18:5269'] servers NOT UP 2025-10-29 21:39:50,848 - INFO - an-jenkins-deploy-platform-ansible-os-4527-1:contrail-vrouter-agent module connection to ['10.20.0.254:5269'] servers UP 2025-10-29 21:39:50,848 - INFO - an-jenkins-deploy-platform-ansible-os-4527-1 connected to xmpp 10.20.0.254 2025-10-29 21:39:50,848 - INFO - an-jenkins-deploy-platform-ansible-os-4527-1:contrail-vrouter-agent module connection to ['10.20.0.14:53'] servers UP 2025-10-29 21:39:50,848 - INFO - an-jenkins-deploy-platform-ansible-os-4527-1 connected to dns 10.20.0.14 2025-10-29 21:39:50,849 - INFO - an-jenkins-deploy-platform-ansible-os-4527-1:contrail-vrouter-agent module connection to ['10.20.0.18:53'] servers UP 2025-10-29 21:39:50,849 - INFO - an-jenkins-deploy-platform-ansible-os-4527-1 connected to dns 10.20.0.18 2025-10-29 21:39:50,849 - WARNING - an-jenkins-deploy-platform-ansible-os-4527-1:contrail-vrouter-agent module connection to ['10.20.0.254:53'] servers NOT UP 2025-10-29 21:39:50,849 - WARNING - an-jenkins-deploy-platform-ansible-os-4527-1:contrail-vrouter-agent module connection to ['10.0.0.254:8086'] servers NOT UP 2025-10-29 21:39:50,849 - INFO - an-jenkins-deploy-platform-ansible-os-4527-1:contrail-vrouter-agent module connection to ['10.0.0.38:8086'] servers UP 2025-10-29 21:39:50,849 - INFO - an-jenkins-deploy-platform-ansible-os-4527-1 connected to collector 10.0.0.38 2025-10-29 21:39:50,849 - WARNING - an-jenkins-deploy-platform-ansible-os-4527-1:contrail-vrouter-agent module connection to ['10.0.0.33:8086'] servers NOT UP 2025-10-29 21:39:50,849 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4527-1?flat 2025-10-29 21:39:51,219 - INFO - cn-jenkins-deploy-platform-ansible-os-4527-1:contrail-control module connection to 10.0.0.254:9041 servers UP 2025-10-29 21:39:51,220 - INFO - cn-jenkins-deploy-platform-ansible-os-4527-1:contrail-control module connection to 10.0.0.254:5673 servers UP 2025-10-29 21:39:51,220 - INFO - cn-jenkins-deploy-platform-ansible-os-4527-1:contrail-control module connection to ['10.0.0.254:8086'] servers UP 2025-10-29 21:39:51,220 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4527-2?flat 2025-10-29 21:39:51,286 - INFO - cn-jenkins-deploy-platform-ansible-os-4527-2:contrail-control module connection to 10.0.0.254:9041 servers UP 2025-10-29 21:39:51,286 - INFO - cn-jenkins-deploy-platform-ansible-os-4527-2:contrail-control module connection to 10.0.0.254:5673 servers UP 2025-10-29 21:39:51,286 - INFO - cn-jenkins-deploy-platform-ansible-os-4527-2:contrail-control module connection to ['10.0.0.254:8086'] servers UP 2025-10-29 21:39:51,286 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4527-3?flat 2025-10-29 21:39:51,310 - INFO - cn-jenkins-deploy-platform-ansible-os-4527-3:contrail-control module connection to 10.0.0.254:9041 servers UP 2025-10-29 21:39:51,310 - INFO - cn-jenkins-deploy-platform-ansible-os-4527-3:contrail-control module connection to 10.0.0.254:5673 servers UP 2025-10-29 21:39:51,311 - INFO - cn-jenkins-deploy-platform-ansible-os-4527-3:contrail-control module connection to ['10.0.0.254:8086'] servers UP 2025-10-29 21:39:51,311 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4527-1.?flat 2025-10-29 21:39:51,331 - DEBUG - contrail-api is running 2025-10-29 21:39:51,331 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4527-2.?flat 2025-10-29 21:39:51,358 - DEBUG - contrail-api is running 2025-10-29 21:39:51,358 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4527-3.?flat 2025-10-29 21:39:51,380 - DEBUG - contrail-api is running 2025-10-29 21:39:51,654 - INFO - Created VN ctest-vn3-57553690, UUID :04af3bdd-dd26-4b78-8cd1-0dfcb75eb0db 2025-10-29 21:39:52,041 - DEBUG - Requesting: http://10.0.0.254:8082/domains 2025-10-29 21:39:52,077 - DEBUG - Requesting: http://10.0.0.254:8082/domain/a090225f-9b9e-4366-926c-6397a8c195bb 2025-10-29 21:39:52,089 - DEBUG - Requesting: http://10.0.0.254:8082/project/9b488c42-e15b-4391-a1ef-0c96a0dda6cf 2025-10-29 21:39:52,183 - DEBUG - Requesting: http://10.0.0.254:8082/virtual-network/2710a27a-333f-48f1-9b07-63c28e1b45ec 2025-10-29 21:39:52,207 - DEBUG - Requesting: http://10.0.0.254:8082/virtual-network/2710a27a-333f-48f1-9b07-63c28e1b45ec 2025-10-29 21:39:52,219 - DEBUG - Requesting: http://10.0.0.254:8082/routing-instance/3fef5723-58fa-4212-85fe-12547afd0ca2 2025-10-29 21:39:52,235 - DEBUG - Requesting: http://10.0.0.254:8082/routing-instance/3fef5723-58fa-4212-85fe-12547afd0ca2 2025-10-29 21:39:52,247 - DEBUG - Requesting: http://10.0.0.254:8082/route-target/a3f848ee-adbc-4b02-98ff-3f7dd963c939 2025-10-29 21:39:52,255 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-29 21:39:52,255 - DEBUG - Requesting: http://10.0.0.254:8082/virtual-network/2710a27a-333f-48f1-9b07-63c28e1b45ec 2025-10-29 21:39:52,264 - DEBUG - Requesting: http://10.0.0.254:8082/routing-instance/3fef5723-58fa-4212-85fe-12547afd0ca2 2025-10-29 21:39:52,279 - INFO - Verified VN network id 15 for VN 2710a27a-333f-48f1-9b07-63c28e1b45ec 2025-10-29 21:39:52,279 - INFO - Verifications in API Server for VN ctest-vn1-38715447 passed 2025-10-29 21:39:52,279 - DEBUG - Requesting: http://10.0.0.254:8082/virtual-network/2710a27a-333f-48f1-9b07-63c28e1b45ec 2025-10-29 21:39:52,290 - DEBUG - Requesting: http://10.0.0.254:8082/routing-instance/3fef5723-58fa-4212-85fe-12547afd0ca2 2025-10-29 21:39:52,299 - DEBUG - Requesting: http://10.0.0.254:8082/routing-instance/3fef5723-58fa-4212-85fe-12547afd0ca2 2025-10-29 21:39:52,307 - DEBUG - Requesting: http://10.0.0.254:8082/route-target/a3f848ee-adbc-4b02-98ff-3f7dd963c939 2025-10-29 21:39:52,319 - DEBUG - Control-node 10.0.0.254 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91865949', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '2814928412836055281', 'uuid-lslong': '11171007087974761964'}, 'enable': 'true', 'created': '2025-10-29T21:36:31', 'last-modified': '2025-10-29T21:36:32', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:06.295819'} 2025-10-29 21:39:52,326 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-29 21:39:52,331 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91865949', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '2814928412836055281', 'uuid-lslong': '11171007087974761964'}, 'enable': 'true', 'created': '2025-10-29T21:36:31', 'last-modified': '2025-10-29T21:36:32', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:01.020517'} 2025-10-29 21:39:52,336 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-29 21:39:52,344 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-91865949', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '2814928412836055281', 'uuid-lslong': '11171007087974761964'}, 'enable': 'true', 'created': '2025-10-29T21:36:31', 'last-modified': '2025-10-29T21:36:32', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:55.440037'} 2025-10-29 21:39:52,352 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-10-29 21:39:52,352 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-38715447 passed 2025-10-29 21:39:52,353 - DEBUG - ====Verifying policy data for ctest-vn1-38715447 in API_Server ====== 2025-10-29 21:39:52,353 - DEBUG - Requesting: http://10.0.0.254:8082/domains 2025-10-29 21:39:52,360 - DEBUG - Requesting: http://10.0.0.254:8082/domain/a090225f-9b9e-4366-926c-6397a8c195bb 2025-10-29 21:39:52,373 - DEBUG - Requesting: http://10.0.0.254:8082/project/9b488c42-e15b-4391-a1ef-0c96a0dda6cf 2025-10-29 21:39:52,475 - DEBUG - Requesting: http://10.0.0.254:8082/virtual-network/2710a27a-333f-48f1-9b07-63c28e1b45ec 2025-10-29 21:39:52,487 - DEBUG - ==>Verifying data for policy with id: 7c60056c-d0de-4b0b-a1e7-f5de43874114, fqn: ['default-domain', 'ctest-TestBasicVMVN0-91865949', 'policy1'] 2025-10-29 21:39:52,488 - INFO - VN ctest-vn1-38715447 Policy verification: verify_vn_policy_in_api_server, status: True 2025-10-29 21:39:52,488 - DEBUG - Verifying the vn in opserver 2025-10-29 21:39:52,488 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 virtual network link through opserver 10.0.0.254 2025-10-29 21:39:52,488 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-networks 2025-10-29 21:39:52,504 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447', 'href': 'http://10.0.0.254:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447?flat'} 2025-10-29 21:39:52,504 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 virtual network link through opserver 10.0.0.38 2025-10-29 21:39:52,504 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-10-29 21:39:52,560 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447?flat'} 2025-10-29 21:39:52,560 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 virtual network link through opserver 10.0.0.33 2025-10-29 21:39:52,560 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2025-10-29 21:39:52,615 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447?flat'} 2025-10-29 21:39:52,615 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 is found in opserver 2025-10-29 21:39:52,632 - DEBUG - VRF ids for VN ctest-vn1-38715447: {'10.0.0.242': '2'} 2025-10-29 21:39:53,825 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4527-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4527-2) 2025-10-29 21:39:53,879 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-10-29 21:39:58,955 - DEBUG - VM is in ACTIVE state now 2025-10-29 21:39:58,955 - INFO - VM name : ctest-vn1_vm2-18851643 2025-10-29 21:39:59,038 - DEBUG - VM ctest-vn1_vm2-18851643 ID is c537ba61-c873-4203-aa28-992549ff346c 2025-10-29 21:39:59,116 - DEBUG - VM is in ACTIVE state now 2025-10-29 21:39:59,116 - DEBUG - VM ctest-vn1_vm2-18851643 launched on Node an-jenkins-deploy-platform-ansible-os-4527-2 2025-10-29 21:39:59,196 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/c537ba61-c873-4203-aa28-992549ff346c 2025-10-29 21:39:59,232 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/c537ba61-c873-4203-aa28-992549ff346c 2025-10-29 21:39:59,239 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine-interface/cca351f3-ba85-4d33-a336-251a97c7e77d 2025-10-29 21:39:59,248 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/c537ba61-c873-4203-aa28-992549ff346c 2025-10-29 21:39:59,255 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine-interface/cca351f3-ba85-4d33-a336-251a97c7e77d 2025-10-29 21:39:59,264 - DEBUG - Requesting: http://10.0.0.33:8082/instance-ip/e0eafb94-8d9f-4aec-b355-b6fc02db5cfe 2025-10-29 21:39:59,272 - DEBUG - Verifying in api server 10.0.0.33 2025-10-29 21:39:59,272 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/c537ba61-c873-4203-aa28-992549ff346c 2025-10-29 21:39:59,279 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine-interface/cca351f3-ba85-4d33-a336-251a97c7e77d 2025-10-29 21:39:59,288 - DEBUG - Requesting: http://10.0.0.33:8082/instance-ip/e0eafb94-8d9f-4aec-b355-b6fc02db5cfe 2025-10-29 21:39:59,294 - INFO - VM ctest-vn1_vm2-18851643 verfication in all API Servers passed 2025-10-29 21:39:59,310 - DEBUG - VM ctest-vn1_vm2-18851643 Tap interface: {'index': '4', 'name': 'tapcca351f3-ba', 'uuid': 'cca351f3-ba85-4d33-a336-251a97c7e77d', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447:ctest-vn1-38715447', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447', 'vm_uuid': 'c537ba61-c873-4203-aa28-992549ff346c', 'vm_name': 'ctest-vn1_vm2-18851643', 'ip_addr': '192.168.1.251', 'mac_addr': '02:cc:a3:51:f3:ba', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '169', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-91865949:cca351f3-ba85-4d33-a336-251a97c7e77d', 'sg_uuid_list': ['9976a145-8190-42a4-82b7-e413e90cdc3e'], 'static_route_list': None, 'vm_project_uuid': '9b488c42-e15b-4391-a1ef-0c96a0dda6cf', 'admin_state': 'Enabled', 'flow_key_idx': '28', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-29 21:39:59,310 - DEBUG - Agent 10.0.0.242 vrf name: default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447:ctest-vn1-38715447 2025-10-29 21:39:59,316 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447:ctest-vn1-38715447', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447', 'table_label': '-1', 'vxlan_id': '15', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.19:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-10-29 21:39:59,393 - DEBUG - VM ctest-vn1_vm2-18851643 : Tap interface tapcca351f3-ba is set to Active 2025-10-29 21:39:59,393 - DEBUG - Tap interface tapcca351f3-ba detail : {'index': '4', 'name': 'tapcca351f3-ba', 'uuid': 'cca351f3-ba85-4d33-a336-251a97c7e77d', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447:ctest-vn1-38715447', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447', 'vm_uuid': 'c537ba61-c873-4203-aa28-992549ff346c', 'vm_name': 'ctest-vn1_vm2-18851643', 'ip_addr': '192.168.1.251', 'mac_addr': '02:cc:a3:51:f3:ba', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '169', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-91865949:cca351f3-ba85-4d33-a336-251a97c7e77d', 'sg_uuid_list': ['9976a145-8190-42a4-82b7-e413e90cdc3e'], 'static_route_list': None, 'vm_project_uuid': '9b488c42-e15b-4391-a1ef-0c96a0dda6cf', 'admin_state': 'Enabled', 'flow_key_idx': '28', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-29 21:39:59,502 - DEBUG - Starting Layer 2 verification in Agent 2025-10-29 21:39:59,525 - DEBUG - Layer 2 path is seen for VM MAC 02:cc:a3:51:f3:ba in agent 10.0.0.242 2025-10-29 21:39:59,526 - DEBUG - Active layer 2 route in agent is present for VMI tapcca351f3-ba 2025-10-29 21:39:59,526 - DEBUG - L2 label(34) matches bw route table and interface table 2025-10-29 21:39:59,548 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 is found in Agent of node 10.0.0.242 2025-10-29 21:39:59,639 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 is consistent in agent 10.0.0.242 2025-10-29 21:39:59,640 - DEBUG - Route for VM IP ['192.168.1.251'] is consistent in agent 10.0.0.242 2025-10-29 21:39:59,640 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 verification for VM ctest-vn1_vm2-18851643 in Agent 10.0.0.242 passed 2025-10-29 21:39:59,747 - DEBUG - Starting all layer 2 verification in agent 10.0.0.242 2025-10-29 21:39:59,764 - DEBUG - Route for VM MAC 02:cc:a3:51:f3:ba is consistent in agent 10.0.0.242 2025-10-29 21:40:02,943 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1003ms') 2025-10-29 21:40:02,943 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-18851643 failed! 2025-10-29 21:40:02,958 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 is 192.168.1.254 and allocation pool is NOT set 2025-10-29 21:40:07,032 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2025-10-29 21:40:07,032 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-18851643 failed! 2025-10-29 21:40:07,046 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 is 192.168.1.254 and allocation pool is NOT set 2025-10-29 21:40:11,135 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1027ms') 2025-10-29 21:40:11,135 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-18851643 failed! 2025-10-29 21:40:11,150 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 is 192.168.1.254 and allocation pool is NOT set 2025-10-29 21:40:15,233 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2025-10-29 21:40:15,233 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-18851643 failed! 2025-10-29 21:40:15,245 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 is 192.168.1.254 and allocation pool is NOT set 2025-10-29 21:40:19,327 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-10-29 21:40:19,327 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-18851643 failed! 2025-10-29 21:40:19,341 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 is 192.168.1.254 and allocation pool is NOT set 2025-10-29 21:40:23,421 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-10-29 21:40:23,421 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-18851643 failed! 2025-10-29 21:40:23,434 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 is 192.168.1.254 and allocation pool is NOT set 2025-10-29 21:40:27,518 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-10-29 21:40:27,518 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-18851643 failed! 2025-10-29 21:40:27,531 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 is 192.168.1.254 and allocation pool is NOT set 2025-10-29 21:40:31,616 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=75.9 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 1 received, 50% packet loss, time 1019ms\r\nrtt min/avg/max/mdev = 75.923/75.923/75.923/0.000 ms') 2025-10-29 21:40:31,616 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-18851643 passed 2025-10-29 21:40:31,616 - INFO - VM ctest-vn1_vm2-18851643 verifications in Compute nodes passed 2025-10-29 21:40:31,821 - DEBUG - Validated VM route 192.168.1.251 in vrouter of 10.0.0.242 2025-10-29 21:40:31,827 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-10-29 21:40:31,827 - INFO - Validated routes of VM ctest-vn1_vm2-18851643 in all vrouters 2025-10-29 21:40:32,246 - DEBUG - Starting all layer2 verification in 10.0.0.33 Control Node 2025-10-29 21:40:32,368 - DEBUG - Layer2 route found for VM MAC 02:cc:a3:51:f3:ba in Control-node 10.0.0.33 2025-10-29 21:40:32,368 - DEBUG - L2 Label for VM ctest-vn1_vm2-18851643 same between Control-node 10.0.0.33 and Agent, Expected: 34, Seen: 34 2025-10-29 21:40:32,369 - DEBUG - Starting all layer2 verification in 10.0.0.38 Control Node 2025-10-29 21:40:32,490 - DEBUG - Layer2 route found for VM MAC 02:cc:a3:51:f3:ba in Control-node 10.0.0.38 2025-10-29 21:40:32,490 - DEBUG - L2 Label for VM ctest-vn1_vm2-18851643 same between Control-node 10.0.0.38 and Agent, Expected: 34, Seen: 34 2025-10-29 21:40:32,490 - DEBUG - Starting all layer2 verification in 10.0.0.254 Control Node 2025-10-29 21:40:32,609 - DEBUG - Layer2 route found for VM MAC 02:cc:a3:51:f3:ba in Control-node 10.0.0.254 2025-10-29 21:40:32,609 - DEBUG - L2 Label for VM ctest-vn1_vm2-18851643 same between Control-node 10.0.0.254 and Agent, Expected: 34, Seen: 34 2025-10-29 21:40:32,609 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-18851643 passed 2025-10-29 21:40:32,610 - DEBUG - Verifying the vm in opserver 2025-10-29 21:40:32,610 - DEBUG - Verifying in collector 10.0.0.33 ... 2025-10-29 21:40:32,610 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machine/c537ba61-c873-4203-aa28-992549ff346c?flat 2025-10-29 21:40:32,675 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91865949:cca351f3-ba85-4d33-a336-251a97c7e77d?flat 2025-10-29 21:40:32,693 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91865949:cca351f3-ba85-4d33-a336-251a97c7e77d?flat 2025-10-29 21:40:32,712 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machine/c537ba61-c873-4203-aa28-992549ff346c?flat 2025-10-29 21:40:32,792 - DEBUG - Verifying vm in vn uve 2025-10-29 21:40:32,792 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91865949:cca351f3-ba85-4d33-a336-251a97c7e77d?flat 2025-10-29 21:40:32,809 - DEBUG - VM uve shows interface as {'ip_address': '192.168.1.251', 'ip6_address': '::', 'vhostuser_mode': 0, 'fixed_ip4_list': ['192.168.1.251'], 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447', 'uuid': 'cca351f3-ba85-4d33-a336-251a97c7e77d', 'port_mirror_enabled': False, 'ip6_active': False, '__T': 1761774010417580, 'admin_state': True, 'l2_active': True, 'label': 30, 'vm_name': 'ctest-vn1_vm2-18851643', 'active': True, 'is_health_check_active': True, 'vn_uuid': '2710a27a-333f-48f1-9b07-63c28e1b45ec', 'gateway': '192.168.1.254', 'vm_uuid': 'c537ba61-c873-4203-aa28-992549ff346c', 'tx_vlan': 65535, 'mac_address': '02:cc:a3:51:f3:ba', 'rx_vlan': 65535, 'ip4_active': True} 2025-10-29 21:40:32,809 - DEBUG - VM uve shows ip address as ['192.168.1.251', '::'] 2025-10-29 21:40:32,809 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 2025-10-29 21:40:32,809 - INFO - Verifying through opserver in 10.0.0.254 2025-10-29 21:40:32,809 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447?flat 2025-10-29 21:40:32,831 - DEBUG - expected vm list c537ba61-c873-4203-aa28-992549ff346c 2025-10-29 21:40:32,831 - DEBUG - Extracted vm list ['c537ba61-c873-4203-aa28-992549ff346c'] 2025-10-29 21:40:32,831 - INFO - VM c537ba61-c873-4203-aa28-992549ff346c is present in default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 2025-10-29 21:40:32,831 - DEBUG - Verifying vm in vrouter uve 2025-10-29 21:40:32,831 - DEBUG - Getting info from collector 10.0.0.33.. 2025-10-29 21:40:32,831 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machine/c537ba61-c873-4203-aa28-992549ff346c?flat 2025-10-29 21:40:32,846 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4527-2'] 2025-10-29 21:40:32,846 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4527-2:Compute:contrail-vrouter-agent:0?flat 2025-10-29 21:40:32,911 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-2?flat 2025-10-29 21:40:32,942 - DEBUG - VM c537ba61-c873-4203-aa28-992549ff346c is present in vrouter an-jenkins-deploy-platform-ansible-os-4527-2 uve 2025-10-29 21:40:32,942 - DEBUG - Validated that VM ctest-vn1_vm2-18851643 is in Vrouter an-jenkins-deploy-platform-ansible-os-4527-2 UVE 2025-10-29 21:40:32,942 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-10-29 21:40:32,951 - DEBUG - Expected tap interface of VM uuid c537ba61-c873-4203-aa28-992549ff346c is default-domain:ctest-TestBasicVMVN0-91865949:cca351f3-ba85-4d33-a336-251a97c7e77d 2025-10-29 21:40:32,952 - DEBUG - Expected VN of VM uuid c537ba61-c873-4203-aa28-992549ff346c is default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 2025-10-29 21:40:32,952 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4527-2:Compute:contrail-vrouter-agent:0?flat 2025-10-29 21:40:33,012 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-2?flat 2025-10-29 21:40:33,037 - DEBUG - VM c537ba61-c873-4203-aa28-992549ff346c is present in vrouter an-jenkins-deploy-platform-ansible-os-4527-2 uve 2025-10-29 21:40:33,037 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-91865949:cca351f3-ba85-4d33-a336-251a97c7e77d of vm c537ba61-c873-4203-aa28-992549ff346c is present in vrouter an-jenkins-deploy-platform-ansible-os-4527-2 uve 2025-10-29 21:40:33,037 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 of vm c537ba61-c873-4203-aa28-992549ff346c is present in vrouter an-jenkins-deploy-platform-ansible-os-4527-2 uve 2025-10-29 21:40:33,037 - DEBUG - Verifying the c537ba61-c873-4203-aa28-992549ff346c virtual network link through opserver 10.0.0.254 2025-10-29 21:40:33,037 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-machines 2025-10-29 21:40:33,053 - DEBUG - VM link and name as {'name': 'c537ba61-c873-4203-aa28-992549ff346c', 'href': 'http://10.0.0.254:8081/analytics/uves/virtual-machine/c537ba61-c873-4203-aa28-992549ff346c?flat'} 2025-10-29 21:40:33,053 - DEBUG - Verifying the c537ba61-c873-4203-aa28-992549ff346c virtual network link through opserver 10.0.0.38 2025-10-29 21:40:33,053 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-10-29 21:40:33,113 - DEBUG - VM link and name as {'name': 'c537ba61-c873-4203-aa28-992549ff346c', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-machine/c537ba61-c873-4203-aa28-992549ff346c?flat'} 2025-10-29 21:40:33,113 - DEBUG - Verifying the c537ba61-c873-4203-aa28-992549ff346c virtual network link through opserver 10.0.0.33 2025-10-29 21:40:33,113 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-10-29 21:40:33,125 - DEBUG - VM link and name as {'name': 'c537ba61-c873-4203-aa28-992549ff346c', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-machine/c537ba61-c873-4203-aa28-992549ff346c?flat'} 2025-10-29 21:40:33,125 - INFO - VM ctest-vn1_vm2-18851643 validations in Opserver passed 2025-10-29 21:40:34,334 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4527-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4527-1) 2025-10-29 21:40:34,379 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-10-29 21:40:39,462 - DEBUG - VM is in ACTIVE state now 2025-10-29 21:40:39,463 - INFO - VM name : ctest-vn2_vm2-23734721 2025-10-29 21:40:39,562 - DEBUG - VM ctest-vn2_vm2-23734721 ID is 7da6c3a5-b23e-403d-b548-ce6018ea10c9 2025-10-29 21:40:39,661 - DEBUG - VM is in ACTIVE state now 2025-10-29 21:40:39,661 - DEBUG - VM ctest-vn2_vm2-23734721 launched on Node an-jenkins-deploy-platform-ansible-os-4527-1 2025-10-29 21:40:39,745 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/7da6c3a5-b23e-403d-b548-ce6018ea10c9 2025-10-29 21:40:39,753 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/7da6c3a5-b23e-403d-b548-ce6018ea10c9 2025-10-29 21:40:39,761 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine-interface/187ba937-1b35-4dd5-8346-b6678402e6d5 2025-10-29 21:40:39,772 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/7da6c3a5-b23e-403d-b548-ce6018ea10c9 2025-10-29 21:40:39,779 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine-interface/187ba937-1b35-4dd5-8346-b6678402e6d5 2025-10-29 21:40:39,789 - DEBUG - Requesting: http://10.0.0.33:8082/instance-ip/b941dac4-79b0-48cb-95fb-419e29c5e40f 2025-10-29 21:40:39,798 - DEBUG - Verifying in api server 10.0.0.33 2025-10-29 21:40:39,798 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/7da6c3a5-b23e-403d-b548-ce6018ea10c9 2025-10-29 21:40:39,804 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine-interface/187ba937-1b35-4dd5-8346-b6678402e6d5 2025-10-29 21:40:39,811 - DEBUG - Requesting: http://10.0.0.33:8082/instance-ip/b941dac4-79b0-48cb-95fb-419e29c5e40f 2025-10-29 21:40:39,818 - INFO - VM ctest-vn2_vm2-23734721 verfication in all API Servers passed 2025-10-29 21:40:39,873 - DEBUG - VM ctest-vn2_vm2-23734721 Tap interface: {'index': '4', 'name': 'tap187ba937-1b', 'uuid': '187ba937-1b35-4dd5-8346-b6678402e6d5', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641:ctest-vn2-07183641', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '16', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641', 'vm_uuid': '7da6c3a5-b23e-403d-b548-ce6018ea10c9', 'vm_name': 'ctest-vn2_vm2-23734721', 'ip_addr': '192.168.2.251', 'mac_addr': '02:18:7b:a9:37:1b', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '127', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-91865949:187ba937-1b35-4dd5-8346-b6678402e6d5', 'sg_uuid_list': ['9976a145-8190-42a4-82b7-e413e90cdc3e'], 'static_route_list': None, 'vm_project_uuid': '9b488c42-e15b-4391-a1ef-0c96a0dda6cf', 'admin_state': 'Enabled', 'flow_key_idx': '35', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-29 21:40:39,873 - DEBUG - Agent 10.0.0.241 vrf name: default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641:ctest-vn2-07183641 2025-10-29 21:40:39,889 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641:ctest-vn2-07183641', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641', 'table_label': '-1', 'vxlan_id': '16', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.129:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-10-29 21:40:39,970 - DEBUG - VM ctest-vn2_vm2-23734721 : Tap interface tap187ba937-1b is set to Active 2025-10-29 21:40:39,971 - DEBUG - Tap interface tap187ba937-1b detail : {'index': '4', 'name': 'tap187ba937-1b', 'uuid': '187ba937-1b35-4dd5-8346-b6678402e6d5', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641:ctest-vn2-07183641', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '16', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641', 'vm_uuid': '7da6c3a5-b23e-403d-b548-ce6018ea10c9', 'vm_name': 'ctest-vn2_vm2-23734721', 'ip_addr': '192.168.2.251', 'mac_addr': '02:18:7b:a9:37:1b', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '127', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-91865949:187ba937-1b35-4dd5-8346-b6678402e6d5', 'sg_uuid_list': ['9976a145-8190-42a4-82b7-e413e90cdc3e'], 'static_route_list': None, 'vm_project_uuid': '9b488c42-e15b-4391-a1ef-0c96a0dda6cf', 'admin_state': 'Enabled', 'flow_key_idx': '35', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-10-29 21:40:40,078 - DEBUG - Starting Layer 2 verification in Agent 2025-10-29 21:40:40,092 - DEBUG - Layer 2 path is seen for VM MAC 02:18:7b:a9:37:1b in agent 10.0.0.241 2025-10-29 21:40:40,092 - DEBUG - Active layer 2 route in agent is present for VMI tap187ba937-1b 2025-10-29 21:40:40,093 - DEBUG - L2 label(34) matches bw route table and interface table 2025-10-29 21:40:40,113 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 is found in Agent of node 10.0.0.241 2025-10-29 21:40:40,245 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 is consistent in agent 10.0.0.241 2025-10-29 21:40:40,245 - DEBUG - Route for VM IP ['192.168.2.251'] is consistent in agent 10.0.0.241 2025-10-29 21:40:40,245 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 verification for VM ctest-vn2_vm2-23734721 in Agent 10.0.0.241 passed 2025-10-29 21:40:40,357 - DEBUG - Starting all layer 2 verification in agent 10.0.0.241 2025-10-29 21:40:40,386 - DEBUG - Route for VM MAC 02:18:7b:a9:37:1b is consistent in agent 10.0.0.241 2025-10-29 21:40:43,588 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1016ms') 2025-10-29 21:40:43,588 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-23734721 failed! 2025-10-29 21:40:43,601 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 is 192.168.2.254 and allocation pool is NOT set 2025-10-29 21:40:47,714 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1032ms') 2025-10-29 21:40:47,714 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-23734721 failed! 2025-10-29 21:40:47,730 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 is 192.168.2.254 and allocation pool is NOT set 2025-10-29 21:40:51,809 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1016ms') 2025-10-29 21:40:51,809 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-23734721 failed! 2025-10-29 21:40:51,822 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 is 192.168.2.254 and allocation pool is NOT set 2025-10-29 21:40:55,906 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2025-10-29 21:40:55,906 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-23734721 failed! 2025-10-29 21:40:55,921 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 is 192.168.2.254 and allocation pool is NOT set 2025-10-29 21:41:00,042 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1016ms') 2025-10-29 21:41:00,043 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-23734721 failed! 2025-10-29 21:41:00,056 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 is 192.168.2.254 and allocation pool is NOT set 2025-10-29 21:41:04,170 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1024ms') 2025-10-29 21:41:04,170 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-23734721 failed! 2025-10-29 21:41:04,184 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 is 192.168.2.254 and allocation pool is NOT set 2025-10-29 21:41:08,261 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1003ms') 2025-10-29 21:41:08,261 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-23734721 failed! 2025-10-29 21:41:08,276 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 is 192.168.2.254 and allocation pool is NOT set 2025-10-29 21:41:12,394 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1030ms') 2025-10-29 21:41:12,394 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-23734721 failed! 2025-10-29 21:41:12,408 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 is 192.168.2.254 and allocation pool is NOT set 2025-10-29 21:41:16,490 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=8.97 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 1 received, 50% packet loss, time 1006ms\r\nrtt min/avg/max/mdev = 8.973/8.973/8.973/0.000 ms') 2025-10-29 21:41:16,490 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-23734721 passed 2025-10-29 21:41:16,490 - INFO - VM ctest-vn2_vm2-23734721 verifications in Compute nodes passed 2025-10-29 21:41:16,748 - DEBUG - Validated VM route 192.168.2.251 in vrouter of 10.0.0.241 2025-10-29 21:41:16,755 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-10-29 21:41:16,755 - INFO - Validated routes of VM ctest-vn2_vm2-23734721 in all vrouters 2025-10-29 21:41:17,175 - DEBUG - Starting all layer2 verification in 10.0.0.33 Control Node 2025-10-29 21:41:17,310 - DEBUG - Layer2 route found for VM MAC 02:18:7b:a9:37:1b in Control-node 10.0.0.33 2025-10-29 21:41:17,311 - DEBUG - L2 Label for VM ctest-vn2_vm2-23734721 same between Control-node 10.0.0.33 and Agent, Expected: 34, Seen: 34 2025-10-29 21:41:17,311 - DEBUG - Starting all layer2 verification in 10.0.0.38 Control Node 2025-10-29 21:41:17,438 - DEBUG - Layer2 route found for VM MAC 02:18:7b:a9:37:1b in Control-node 10.0.0.38 2025-10-29 21:41:17,438 - DEBUG - L2 Label for VM ctest-vn2_vm2-23734721 same between Control-node 10.0.0.38 and Agent, Expected: 34, Seen: 34 2025-10-29 21:41:17,438 - DEBUG - Starting all layer2 verification in 10.0.0.254 Control Node 2025-10-29 21:41:17,562 - DEBUG - Layer2 route found for VM MAC 02:18:7b:a9:37:1b in Control-node 10.0.0.254 2025-10-29 21:41:17,562 - DEBUG - L2 Label for VM ctest-vn2_vm2-23734721 same between Control-node 10.0.0.254 and Agent, Expected: 34, Seen: 34 2025-10-29 21:41:17,562 - INFO - Verification in Control-nodes for VM ctest-vn2_vm2-23734721 passed 2025-10-29 21:41:17,562 - DEBUG - Verifying the vm in opserver 2025-10-29 21:41:17,562 - DEBUG - Verifying in collector 10.0.0.33 ... 2025-10-29 21:41:17,563 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machine/7da6c3a5-b23e-403d-b548-ce6018ea10c9?flat 2025-10-29 21:41:17,577 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91865949:187ba937-1b35-4dd5-8346-b6678402e6d5?flat 2025-10-29 21:41:17,597 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91865949:187ba937-1b35-4dd5-8346-b6678402e6d5?flat 2025-10-29 21:41:17,615 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machine/7da6c3a5-b23e-403d-b548-ce6018ea10c9?flat 2025-10-29 21:41:17,629 - DEBUG - Verifying vm in vn uve 2025-10-29 21:41:17,629 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-91865949:187ba937-1b35-4dd5-8346-b6678402e6d5?flat 2025-10-29 21:41:17,648 - DEBUG - VM uve shows interface as {'vn_uuid': '36b37917-8e24-4e82-88a0-0e8a8805bf6b', 'rx_vlan': 65535, 'mac_address': '02:18:7b:a9:37:1b', 'vm_name': 'ctest-vn2_vm2-23734721', 'gateway': '192.168.2.254', 'is_health_check_active': True, 'l2_active': True, 'label': 30, 'ip_address': '192.168.2.251', 'vm_uuid': '7da6c3a5-b23e-403d-b548-ce6018ea10c9', '__T': 1761774062060387, 'active': True, 'vhostuser_mode': 0, 'fixed_ip4_list': ['192.168.2.251'], 'ip6_address': '::', 'ip4_active': True, 'port_mirror_enabled': False, 'tx_vlan': 65535, 'ip6_active': False, 'admin_state': True, 'uuid': '187ba937-1b35-4dd5-8346-b6678402e6d5', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641'} 2025-10-29 21:41:17,648 - DEBUG - VM uve shows ip address as ['192.168.2.251', '::'] 2025-10-29 21:41:17,648 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 2025-10-29 21:41:17,648 - INFO - Verifying through opserver in 10.0.0.254 2025-10-29 21:41:17,648 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641?flat 2025-10-29 21:41:17,668 - DEBUG - expected vm list 7da6c3a5-b23e-403d-b548-ce6018ea10c9 2025-10-29 21:41:17,668 - DEBUG - Extracted vm list ['7da6c3a5-b23e-403d-b548-ce6018ea10c9'] 2025-10-29 21:41:17,668 - INFO - VM 7da6c3a5-b23e-403d-b548-ce6018ea10c9 is present in default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 2025-10-29 21:41:17,668 - DEBUG - Verifying vm in vrouter uve 2025-10-29 21:41:17,668 - DEBUG - Getting info from collector 10.0.0.33.. 2025-10-29 21:41:17,668 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machine/7da6c3a5-b23e-403d-b548-ce6018ea10c9?flat 2025-10-29 21:41:17,682 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4527-1'] 2025-10-29 21:41:17,682 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4527-1:Compute:contrail-vrouter-agent:0?flat 2025-10-29 21:41:17,713 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-1?flat 2025-10-29 21:41:17,741 - DEBUG - VM 7da6c3a5-b23e-403d-b548-ce6018ea10c9 is present in vrouter an-jenkins-deploy-platform-ansible-os-4527-1 uve 2025-10-29 21:41:17,741 - DEBUG - Validated that VM ctest-vn2_vm2-23734721 is in Vrouter an-jenkins-deploy-platform-ansible-os-4527-1 UVE 2025-10-29 21:41:17,741 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-10-29 21:41:17,765 - DEBUG - Expected tap interface of VM uuid 7da6c3a5-b23e-403d-b548-ce6018ea10c9 is default-domain:ctest-TestBasicVMVN0-91865949:187ba937-1b35-4dd5-8346-b6678402e6d5 2025-10-29 21:41:17,765 - DEBUG - Expected VN of VM uuid 7da6c3a5-b23e-403d-b548-ce6018ea10c9 is default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 2025-10-29 21:41:17,765 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4527-1:Compute:contrail-vrouter-agent:0?flat 2025-10-29 21:41:17,799 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-1?flat 2025-10-29 21:41:17,826 - DEBUG - VM 7da6c3a5-b23e-403d-b548-ce6018ea10c9 is present in vrouter an-jenkins-deploy-platform-ansible-os-4527-1 uve 2025-10-29 21:41:17,826 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-91865949:187ba937-1b35-4dd5-8346-b6678402e6d5 of vm 7da6c3a5-b23e-403d-b548-ce6018ea10c9 is present in vrouter an-jenkins-deploy-platform-ansible-os-4527-1 uve 2025-10-29 21:41:17,826 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 of vm 7da6c3a5-b23e-403d-b548-ce6018ea10c9 is present in vrouter an-jenkins-deploy-platform-ansible-os-4527-1 uve 2025-10-29 21:41:17,826 - DEBUG - Verifying the 7da6c3a5-b23e-403d-b548-ce6018ea10c9 virtual network link through opserver 10.0.0.254 2025-10-29 21:41:17,826 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-machines 2025-10-29 21:41:17,839 - DEBUG - VM link and name as {'name': '7da6c3a5-b23e-403d-b548-ce6018ea10c9', 'href': 'http://10.0.0.254:8081/analytics/uves/virtual-machine/7da6c3a5-b23e-403d-b548-ce6018ea10c9?flat'} 2025-10-29 21:41:17,839 - DEBUG - Verifying the 7da6c3a5-b23e-403d-b548-ce6018ea10c9 virtual network link through opserver 10.0.0.38 2025-10-29 21:41:17,839 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-10-29 21:41:17,893 - DEBUG - VM link and name as {'name': '7da6c3a5-b23e-403d-b548-ce6018ea10c9', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-machine/7da6c3a5-b23e-403d-b548-ce6018ea10c9?flat'} 2025-10-29 21:41:17,893 - DEBUG - Verifying the 7da6c3a5-b23e-403d-b548-ce6018ea10c9 virtual network link through opserver 10.0.0.33 2025-10-29 21:41:17,893 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-10-29 21:41:17,905 - DEBUG - VM link and name as {'name': '7da6c3a5-b23e-403d-b548-ce6018ea10c9', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-machine/7da6c3a5-b23e-403d-b548-ce6018ea10c9?flat'} 2025-10-29 21:41:17,905 - INFO - VM ctest-vn2_vm2-23734721 validations in Opserver passed 2025-10-29 21:41:17,905 - INFO - Waiting for VM ctest-vn1_vm2-18851643 to be up.. 2025-10-29 21:41:17,905 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:41:17,906 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/c537ba61-c873-4203-aa28-992549ff346c 2025-10-29 21:41:17,913 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine-interface/cca351f3-ba85-4d33-a336-251a97c7e77d 2025-10-29 21:41:19,069 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=3.01 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.574 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1000ms\r\nrtt min/avg/max/mdev = 0.574/1.793/3.012/1.219 ms') 2025-10-29 21:41:19,069 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-18851643 passed 2025-10-29 21:41:19,143 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:41:19,143 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-18851643, IP 192.168.1.251, Port 22 2025-10-29 21:41:19,312 - DEBUG - VM ctest-vn1_vm2-18851643 is ready for SSH connections 2025-10-29 21:41:19,312 - INFO - Waiting for VM ctest-vn2_vm2-23734721 to be up.. 2025-10-29 21:41:19,312 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:41:19,313 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/7da6c3a5-b23e-403d-b548-ce6018ea10c9 2025-10-29 21:41:19,323 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine-interface/187ba937-1b35-4dd5-8346-b6678402e6d5 2025-10-29 21:41:20,492 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=3.18 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=1.07 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 1.068/2.124/3.181/1.056 ms') 2025-10-29 21:41:20,492 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-23734721 passed 2025-10-29 21:41:20,559 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:41:20,559 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-23734721, IP 192.168.2.251, Port 22 2025-10-29 21:41:20,637 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-29 21:41:20,715 - DEBUG - VM ctest-vn2_vm2-23734721 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-29 21:41:25,716 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:41:25,716 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-23734721, IP 192.168.2.251, Port 22 2025-10-29 21:41:25,832 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-29 21:41:25,909 - DEBUG - VM ctest-vn2_vm2-23734721 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-29 21:41:30,911 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:41:30,911 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-23734721, IP 192.168.2.251, Port 22 2025-10-29 21:41:30,978 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-10-29 21:41:31,060 - DEBUG - VM ctest-vn2_vm2-23734721 is NOT ready for SSH connections, VM status: ACTIVE 2025-10-29 21:41:36,061 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:41:36,062 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-23734721, IP 192.168.2.251, Port 22 2025-10-29 21:41:36,239 - DEBUG - VM ctest-vn2_vm2-23734721 is ready for SSH connections 2025-10-29 21:41:36,239 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-10-29 21:41:36,239 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.251, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.242, gateway password: c0ntrail123 2025-10-29 21:41:36,239 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.251 2025-10-29 21:42:14,534 - DEBUG - PING 192.168.2.251 (192.168.2.251) 56(84) bytes of data. 64 bytes from 192.168.2.251: icmp_req=1 ttl=63 time=18.5 ms 64 bytes from 192.168.2.251: icmp_req=2 ttl=63 time=1.59 ms 64 bytes from 192.168.2.251: icmp_req=3 ttl=63 time=1.74 ms --- 192.168.2.251 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 1.599/7.304/18.572/7.968 ms 2025-10-29 21:42:14,535 - INFO - Ping to IP 192.168.2.251 from VM ctest-vn1_vm2-18851643 passed 2025-10-29 21:42:14,535 - INFO - Deleting VM ctest-vn2_vm2-23734721 2025-10-29 21:42:14,608 - DEBUG - Verifying in api server 10.0.0.33 2025-10-29 21:42:14,608 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/7da6c3a5-b23e-403d-b548-ce6018ea10c9 2025-10-29 21:42:14,617 - DEBUG - VM ID 7da6c3a5-b23e-403d-b548-ce6018ea10c9 of VM ctest-vn2_vm2-23734721 is still found in API Server 2025-10-29 21:42:16,618 - DEBUG - Verifying in api server 10.0.0.33 2025-10-29 21:42:16,618 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/7da6c3a5-b23e-403d-b548-ce6018ea10c9 2025-10-29 21:42:16,624 - DEBUG - Response Code: 404 2025-10-29 21:42:16,624 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/7da6c3a5-b23e-403d-b548-ce6018ea10c9 2025-10-29 21:42:16,629 - DEBUG - Response Code: 404 2025-10-29 21:42:16,629 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/7da6c3a5-b23e-403d-b548-ce6018ea10c9 2025-10-29 21:42:16,633 - DEBUG - Response Code: 404 2025-10-29 21:42:16,634 - INFO - VM ctest-vn2_vm2-23734721 is fully removed in API-Server 2025-10-29 21:42:16,785 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-10-29 21:42:17,049 - DEBUG - VM route 192.168.2.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 of compute 10.0.0.241 2025-10-29 21:42:17,054 - DEBUG - Validated that vrouter 10.0.0.241 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 2025-10-29 21:42:17,054 - DEBUG - Validated that vrouter 10.0.0.242 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 2025-10-29 21:42:17,054 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 2025-10-29 21:42:17,054 - INFO - VM ctest-vn2_vm2-23734721 is removed in Compute, and routes are removed in all compute nodes 2025-10-29 21:42:17,264 - INFO - Routes for VM ctest-vn2_vm2-23734721 is removed in all control-nodes 2025-10-29 21:42:17,265 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4527-1:Compute:contrail-vrouter-agent:0?flat 2025-10-29 21:42:17,299 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-1?flat 2025-10-29 21:42:17,327 - DEBUG - VM 7da6c3a5-b23e-403d-b548-ce6018ea10c9 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4527-1 uve 2025-10-29 21:42:17,327 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-29 21:42:17,327 - DEBUG - interface for vm 7da6c3a5-b23e-403d-b548-ce6018ea10c9 deleted from vrouter uve ... 2025-10-29 21:42:17,327 - DEBUG - Deleted interface not in error interface list ... 2025-10-29 21:42:20,328 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4527-1:Compute:contrail-vrouter-agent:0?flat 2025-10-29 21:42:20,365 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-1?flat 2025-10-29 21:42:20,393 - DEBUG - VM 7da6c3a5-b23e-403d-b548-ce6018ea10c9 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4527-1 uve 2025-10-29 21:42:20,393 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-29 21:42:20,393 - DEBUG - interface for vm 7da6c3a5-b23e-403d-b548-ce6018ea10c9 deleted from vrouter uve ... 2025-10-29 21:42:20,393 - DEBUG - Deleted interface not in error interface list ... 2025-10-29 21:42:23,394 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4527-1:Compute:contrail-vrouter-agent:0?flat 2025-10-29 21:42:23,431 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-1?flat 2025-10-29 21:42:23,463 - DEBUG - VM 7da6c3a5-b23e-403d-b548-ce6018ea10c9 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4527-1 uve 2025-10-29 21:42:23,463 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-29 21:42:23,463 - DEBUG - interface for vm 7da6c3a5-b23e-403d-b548-ce6018ea10c9 deleted from vrouter uve ... 2025-10-29 21:42:23,463 - DEBUG - Deleted interface not in error interface list ... 2025-10-29 21:42:26,464 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4527-1:Compute:contrail-vrouter-agent:0?flat 2025-10-29 21:42:26,499 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-1?flat 2025-10-29 21:42:26,528 - DEBUG - VM 7da6c3a5-b23e-403d-b548-ce6018ea10c9 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4527-1 uve 2025-10-29 21:42:26,528 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-29 21:42:26,528 - DEBUG - interface for vm 7da6c3a5-b23e-403d-b548-ce6018ea10c9 deleted from vrouter uve ... 2025-10-29 21:42:26,528 - DEBUG - Deleted interface not in error interface list ... 2025-10-29 21:42:29,529 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4527-1:Compute:contrail-vrouter-agent:0?flat 2025-10-29 21:42:29,564 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-1?flat 2025-10-29 21:42:29,592 - DEBUG - VM 7da6c3a5-b23e-403d-b548-ce6018ea10c9 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4527-1 uve 2025-10-29 21:42:29,592 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-29 21:42:29,592 - DEBUG - interface for vm 7da6c3a5-b23e-403d-b548-ce6018ea10c9 deleted from vrouter uve ... 2025-10-29 21:42:29,593 - DEBUG - Deleted interface not in error interface list ... 2025-10-29 21:42:32,593 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4527-1:Compute:contrail-vrouter-agent:0?flat 2025-10-29 21:42:32,628 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-1?flat 2025-10-29 21:42:32,654 - DEBUG - VM 7da6c3a5-b23e-403d-b548-ce6018ea10c9 is not present in vrouter an-jenkins-deploy-platform-ansible-os-4527-1 uve 2025-10-29 21:42:32,655 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-29 21:42:32,655 - DEBUG - interface for vm 7da6c3a5-b23e-403d-b548-ce6018ea10c9 deleted from vrouter uve ... 2025-10-29 21:42:32,655 - DEBUG - Deleted interface not in error interface list ... 2025-10-29 21:42:32,655 - DEBUG - Validated that VM 7da6c3a5-b23e-403d-b548-ce6018ea10c9 is removed in Vrouter UVE 2025-10-29 21:42:32,655 - DEBUG - Verifying through opserver in 10.0.0.254 2025-10-29 21:42:32,655 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641?flat 2025-10-29 21:42:32,677 - DEBUG - 7da6c3a5-b23e-403d-b548-ce6018ea10c9 BM not in default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn2-07183641 uve 2025-10-29 21:42:32,677 - DEBUG - Validated that VM 7da6c3a5-b23e-403d-b548-ce6018ea10c9 is not present in the VN UVE 2025-10-29 21:42:32,677 - DEBUG - Verifying the 7da6c3a5-b23e-403d-b548-ce6018ea10c9 virtual network link through opserver 10.0.0.254 2025-10-29 21:42:32,677 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-machines 2025-10-29 21:42:32,689 - INFO - 7da6c3a5-b23e-403d-b548-ce6018ea10c9 vm uve deleted from opserver 2025-10-29 21:42:32,689 - DEBUG - Verifying the 7da6c3a5-b23e-403d-b548-ce6018ea10c9 virtual network link through opserver 10.0.0.38 2025-10-29 21:42:32,689 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-10-29 21:42:32,701 - INFO - 7da6c3a5-b23e-403d-b548-ce6018ea10c9 vm uve deleted from opserver 2025-10-29 21:42:32,701 - DEBUG - Verifying the 7da6c3a5-b23e-403d-b548-ce6018ea10c9 virtual network link through opserver 10.0.0.33 2025-10-29 21:42:32,702 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-10-29 21:42:32,712 - INFO - 7da6c3a5-b23e-403d-b548-ce6018ea10c9 vm uve deleted from opserver 2025-10-29 21:42:32,712 - INFO - Validated that VM 7da6c3a5-b23e-403d-b548-ce6018ea10c9 is removed from Opserver 2025-10-29 21:42:32,712 - INFO - Deleting VM ctest-vn1_vm2-18851643 2025-10-29 21:42:32,803 - DEBUG - Verifying in api server 10.0.0.33 2025-10-29 21:42:32,803 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/c537ba61-c873-4203-aa28-992549ff346c 2025-10-29 21:42:32,815 - DEBUG - VM ID c537ba61-c873-4203-aa28-992549ff346c of VM ctest-vn1_vm2-18851643 is still found in API Server 2025-10-29 21:42:34,816 - DEBUG - Verifying in api server 10.0.0.33 2025-10-29 21:42:34,817 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/c537ba61-c873-4203-aa28-992549ff346c 2025-10-29 21:42:34,823 - DEBUG - Response Code: 404 2025-10-29 21:42:34,823 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/c537ba61-c873-4203-aa28-992549ff346c 2025-10-29 21:42:34,829 - DEBUG - Response Code: 404 2025-10-29 21:42:34,829 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-machine/c537ba61-c873-4203-aa28-992549ff346c 2025-10-29 21:42:34,834 - DEBUG - Response Code: 404 2025-10-29 21:42:34,835 - INFO - VM ctest-vn1_vm2-18851643 is fully removed in API-Server 2025-10-29 21:42:34,986 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-10-29 21:42:35,028 - DEBUG - Validated that vrouter 10.0.0.241 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 2025-10-29 21:42:35,162 - DEBUG - VM route 192.168.1.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 of compute 10.0.0.242 2025-10-29 21:42:35,168 - DEBUG - Validated that vrouter 10.0.0.242 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 2025-10-29 21:42:35,168 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 2025-10-29 21:42:35,168 - INFO - VM ctest-vn1_vm2-18851643 is removed in Compute, and routes are removed in all compute nodes 2025-10-29 21:42:35,389 - INFO - Routes for VM ctest-vn1_vm2-18851643 is removed in all control-nodes 2025-10-29 21:42:35,389 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4527-2:Compute:contrail-vrouter-agent:0?flat 2025-10-29 21:42:35,453 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-2?flat 2025-10-29 21:42:35,485 - DEBUG - VM c537ba61-c873-4203-aa28-992549ff346c is still present in vrouter an-jenkins-deploy-platform-ansible-os-4527-2 uve 2025-10-29 21:42:35,485 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-29 21:42:35,485 - DEBUG - interface for vm c537ba61-c873-4203-aa28-992549ff346c deleted from vrouter uve ... 2025-10-29 21:42:35,485 - DEBUG - Deleted interface not in error interface list ... 2025-10-29 21:42:38,486 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4527-2:Compute:contrail-vrouter-agent:0?flat 2025-10-29 21:42:38,541 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-2?flat 2025-10-29 21:42:38,568 - DEBUG - VM c537ba61-c873-4203-aa28-992549ff346c is still present in vrouter an-jenkins-deploy-platform-ansible-os-4527-2 uve 2025-10-29 21:42:38,568 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-29 21:42:38,568 - DEBUG - interface for vm c537ba61-c873-4203-aa28-992549ff346c deleted from vrouter uve ... 2025-10-29 21:42:38,568 - DEBUG - Deleted interface not in error interface list ... 2025-10-29 21:42:41,569 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4527-2:Compute:contrail-vrouter-agent:0?flat 2025-10-29 21:42:41,624 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4527-2?flat 2025-10-29 21:42:41,652 - DEBUG - VM c537ba61-c873-4203-aa28-992549ff346c is not present in vrouter an-jenkins-deploy-platform-ansible-os-4527-2 uve 2025-10-29 21:42:41,652 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-10-29 21:42:41,652 - DEBUG - interface for vm c537ba61-c873-4203-aa28-992549ff346c deleted from vrouter uve ... 2025-10-29 21:42:41,652 - DEBUG - Deleted interface not in error interface list ... 2025-10-29 21:42:41,652 - DEBUG - Validated that VM c537ba61-c873-4203-aa28-992549ff346c is removed in Vrouter UVE 2025-10-29 21:42:41,652 - DEBUG - Verifying through opserver in 10.0.0.254 2025-10-29 21:42:41,652 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447?flat 2025-10-29 21:42:41,674 - DEBUG - c537ba61-c873-4203-aa28-992549ff346c BM not in default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 uve 2025-10-29 21:42:41,674 - DEBUG - Validated that VM c537ba61-c873-4203-aa28-992549ff346c is not present in the VN UVE 2025-10-29 21:42:41,674 - DEBUG - Verifying the c537ba61-c873-4203-aa28-992549ff346c virtual network link through opserver 10.0.0.254 2025-10-29 21:42:41,674 - DEBUG - Requesting: http://10.0.0.254:8081/analytics/uves/virtual-machines 2025-10-29 21:42:41,685 - INFO - c537ba61-c873-4203-aa28-992549ff346c vm uve deleted from opserver 2025-10-29 21:42:41,685 - DEBUG - Verifying the c537ba61-c873-4203-aa28-992549ff346c virtual network link through opserver 10.0.0.38 2025-10-29 21:42:41,685 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-10-29 21:42:41,698 - INFO - c537ba61-c873-4203-aa28-992549ff346c vm uve deleted from opserver 2025-10-29 21:42:41,698 - DEBUG - Verifying the c537ba61-c873-4203-aa28-992549ff346c virtual network link through opserver 10.0.0.33 2025-10-29 21:42:41,698 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-10-29 21:42:41,710 - INFO - c537ba61-c873-4203-aa28-992549ff346c vm uve deleted from opserver 2025-10-29 21:42:41,710 - INFO - Validated that VM c537ba61-c873-4203-aa28-992549ff346c is removed from Opserver 2025-10-29 21:42:41,710 - INFO - Deleting VN ctest-vn3-57553690 2025-10-29 21:42:41,818 - INFO - Deleting VM ctest-vn2_vm1-43980660 2025-10-29 21:42:41,918 - INFO - Deleting VM ctest-vn1_vm1-21582819 2025-10-29 21:42:42,140 - INFO - Deleting VN ctest-vn2-07183641 2025-10-29 21:42:42,206 - DEBUG - RefsExistError Delete when resource still referred: ['http://10.0.0.33:8082/instance-ip/603b8440-0522-404c-9653-6d264c9018be'] while deleting VN ctest-vn2-07183641..Will retry 2025-10-29 21:42:44,299 - DEBUG - Requesting: http://10.0.0.254:8082/routing-instance/9d3258e4-37c5-48fb-ac56-61dff558b053 2025-10-29 21:42:44,305 - DEBUG - Response Code: 404 2025-10-29 21:42:44,305 - DEBUG - Requesting: http://10.0.0.254:8082/domains 2025-10-29 21:42:44,312 - DEBUG - Requesting: http://10.0.0.254:8082/domain/a090225f-9b9e-4366-926c-6397a8c195bb 2025-10-29 21:42:44,331 - DEBUG - Requesting: http://10.0.0.254:8082/project/9b488c42-e15b-4391-a1ef-0c96a0dda6cf 2025-10-29 21:42:44,424 - INFO - Validated that VN ctest-vn2-07183641 is not found in API Server 2025-10-29 21:42:44,436 - DEBUG - VN ctest-vn2-07183641 is not present in Agent 10.0.0.241 2025-10-29 21:42:44,446 - DEBUG - VN ctest-vn2-07183641 is not present in Agent 10.0.0.242 2025-10-29 21:42:44,446 - INFO - Validated that VN ctest-vn2-07183641 is not in any agent 2025-10-29 21:42:44,476 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn2-07183641 info 2025-10-29 21:42:44,621 - INFO - Deleting VN ctest-vn1-38715447 2025-10-29 21:42:44,722 - DEBUG - Requesting: http://10.0.0.254:8082/routing-instance/3fef5723-58fa-4212-85fe-12547afd0ca2 2025-10-29 21:42:44,727 - DEBUG - Response Code: 404 2025-10-29 21:42:44,728 - DEBUG - Requesting: http://10.0.0.254:8082/domains 2025-10-29 21:42:44,733 - DEBUG - Requesting: http://10.0.0.254:8082/domain/a090225f-9b9e-4366-926c-6397a8c195bb 2025-10-29 21:42:44,768 - DEBUG - Requesting: http://10.0.0.254:8082/project/9b488c42-e15b-4391-a1ef-0c96a0dda6cf 2025-10-29 21:42:44,851 - INFO - Validated that VN ctest-vn1-38715447 is not found in API Server 2025-10-29 21:42:44,875 - DEBUG - VN ctest-vn1-38715447 is not present in Agent 10.0.0.241 2025-10-29 21:42:44,886 - DEBUG - VN ctest-vn1-38715447 is not present in Agent 10.0.0.242 2025-10-29 21:42:44,886 - INFO - Validated that VN ctest-vn1-38715447 is not in any agent 2025-10-29 21:42:44,892 - DEBUG - VRF 2 is not seen in agent 10.0.0.242 2025-10-29 21:42:44,897 - DEBUG - Vrouter 10.0.0.242 does not have vrf 2 for VN ctest-vn1-38715447 2025-10-29 21:42:44,897 - INFO - Validated that all vrouters do not have the route table for VN default-domain:ctest-TestBasicVMVN0-91865949:ctest-vn1-38715447 2025-10-29 21:42:44,915 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-38715447 info 2025-10-29 21:42:44,943 - INFO - Deleted policy policy2 2025-10-29 21:42:44,992 - INFO - Deleted policy policy1 2025-10-29 21:42:46,301 - DEBUG - Skipping xmpp flap check 2025-10-29 21:42:46,301 - INFO - END TEST : test_process_restart_in_policy_between_vns : PASSED[0:06:17] 2025-10-29 21:42:46,301 - INFO - -------------------------------------------------------------------------------- 2025-10-29 21:42:46,304 - INFO - ================================================================================ 2025-10-29 21:42:46,304 - INFO - STARTING TEST : test_underlay_broadcast_traffic_handling 2025-10-29 21:42:46,304 - INFO - TEST DESCRIPTION : Test the underlay brocast traffic handling by vrouter. (Bug-1545229). 1. Send broadcast traffic from one compute node. 2. Other compute in same subnet should receive that traffic. 3. Receiving compute should treat this traffic as underlay. 4. Compute should not replicate the packet and send the copy back. Pass criteria: Step 3-4 should pass Maintainer : chhandak@juniper.net 2025-10-29 21:42:47,621 - DEBUG - Skipping xmpp flap check 2025-10-29 21:42:47,621 - INFO - Initial checks done. Running the testcase now 2025-10-29 21:42:47,621 - INFO - 2025-10-29 21:42:48,054 - INFO - Agent interface name: ens6 2025-10-29 21:42:48,138 - DEBUG - Executing command: sudo tcpdump -nni ens6 -U host 0.0.0.0 -w /tmp/ens6_ctest-random-45494478.pcap 2025-10-29 21:42:53,496 - INFO - Pinging broacast address 0.0.0.0 from compute 10.20.0.129 2025-10-29 21:43:07,918 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-10-29 21:43:10,087 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-45494478.pcap | grep -c length 2025-10-29 21:43:10,100 - DEBUG - STDOUT: 10 2025-10-29 21:43:10,100 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-45494478.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-10-29 21:43:10,100 - INFO - 10 packets are found in tcpdump output as expected 2025-10-29 21:43:10,101 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-10-29 21:43:12,144 - INFO - Packet count matched: Compute 10.0.0.242 has receive only 10 packet from source IP 0.0.0.0. No duplicate packet seen 2025-10-29 21:43:13,461 - DEBUG - Skipping xmpp flap check 2025-10-29 21:43:13,461 - INFO - END TEST : test_underlay_broadcast_traffic_handling : PASSED[0:00:27] 2025-10-29 21:43:13,461 - INFO - -------------------------------------------------------------------------------- 2025-10-29 21:43:14,246 - INFO - Deleted project: ctest-TestBasicVMVN0-91865949, ID : 9b488c42-e15b-4391-a1ef-0c96a0dda6cf