2025-12-22 03:53:38,274 - INFO - Domain Default found not creating 2025-12-22 03:53:38,451 - INFO - Project ctest-TestMd5testsOnControl-67453638 not found, creating it 2025-12-22 03:53:39,188 - INFO - Created Project:ctest-TestMd5testsOnControl-67453638, ID : ea4fd547-b3bf-450d-aae1-8a4cece19ee1 2025-12-22 03:53:41,348 - DEBUG - Response for create_network : {'network': {'id': '5d28afac-3658-4cf6-8b37-c8e0e30391d7', 'name': 'test_vnv6sr', 'tenant_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'project_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestMd5testsOnControl-67453638', 'test_vnv6sr'], 'port_security_enabled': True, 'description': ''}} 2025-12-22 03:53:41,604 - DEBUG - Response for create_subnet : {'subnet': {'id': '7193091e-32d6-4739-aec5-e9225d1c8190', 'name': '', 'tenant_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'network_id': '5d28afac-3658-4cf6-8b37-c8e0e30391d7', 'ip_version': 6, 'cidr': '2001::101:0/120', 'allocation_pools': [{'start': '2001::101:2', 'end': '2001::101:fe'}], 'gateway_ip': '2001::101:1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '2001::101:2', 'tags': [], 'project_id': 'ea4fd547b3bf450daae18a4cece19ee1'}} 2025-12-22 03:53:41,865 - DEBUG - Response for create_subnet : {'subnet': {'id': '609aabfb-0483-442f-b630-a4eedd8a0460', 'name': '', 'tenant_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'network_id': '5d28afac-3658-4cf6-8b37-c8e0e30391d7', 'ip_version': 4, 'cidr': '65.34.210.192/26', 'allocation_pools': [{'start': '65.34.210.194', 'end': '65.34.210.254'}], 'gateway_ip': '65.34.210.193', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '65.34.210.194', 'tags': [], 'project_id': 'ea4fd547b3bf450daae18a4cece19ee1'}} 2025-12-22 03:53:41,892 - INFO - Created VN test_vnv6sr 2025-12-22 03:53:41,904 - DEBUG - VN test_vnv6sr UUID is 5d28afac-3658-4cf6-8b37-c8e0e30391d7 2025-12-22 03:53:42,307 - DEBUG - Response for create_network : {'network': {'id': '7e3b8046-8380-45fd-b424-8e3cf32175b3', 'name': 'test_vnv6dn', 'tenant_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'project_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestMd5testsOnControl-67453638', 'test_vnv6dn'], 'port_security_enabled': True, 'description': ''}} 2025-12-22 03:53:42,556 - DEBUG - Response for create_subnet : {'subnet': {'id': '0fc5d90e-c546-4a27-bc53-39f4c2a1a344', 'name': '', 'tenant_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'network_id': '7e3b8046-8380-45fd-b424-8e3cf32175b3', 'ip_version': 6, 'cidr': '2001::201:0/120', 'allocation_pools': [{'start': '2001::201:2', 'end': '2001::201:fe'}], 'gateway_ip': '2001::201:1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '2001::201:2', 'tags': [], 'project_id': 'ea4fd547b3bf450daae18a4cece19ee1'}} 2025-12-22 03:53:42,827 - DEBUG - Response for create_subnet : {'subnet': {'id': '43bd4c13-3dd9-4448-b5f8-0bb86c5709a9', 'name': '', 'tenant_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'network_id': '7e3b8046-8380-45fd-b424-8e3cf32175b3', 'ip_version': 4, 'cidr': '66.189.20.64/26', 'allocation_pools': [{'start': '66.189.20.66', 'end': '66.189.20.126'}], 'gateway_ip': '66.189.20.65', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '66.189.20.66', 'tags': [], 'project_id': 'ea4fd547b3bf450daae18a4cece19ee1'}} 2025-12-22 03:53:42,852 - INFO - Created VN test_vnv6dn 2025-12-22 03:53:42,865 - DEBUG - VN test_vnv6dn UUID is 7e3b8046-8380-45fd-b424-8e3cf32175b3 2025-12-22 03:53:43,134 - DEBUG - Services list from nova: [, , , ] 2025-12-22 03:53:44,297 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4918-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4918-1) 2025-12-22 03:53:45,562 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4918-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4918-2) 2025-12-22 03:53:45,562 - INFO - Waiting for VM source_vm to be up.. 2025-12-22 03:53:45,627 - DEBUG - VM is in ERROR state now 2025-12-22 03:53:45,627 - ERROR - VM in error state. 2025-12-22 03:53:45,627 - ERROR - VM is not in ACTIVE state 2025-12-22 03:53:45,741 - ERROR - Fatal Nova Exception while getting VM detail 2025-12-22 03:53:45,741 - DEBUG - None 2025-12-22 03:53:45,741 - INFO - Waiting for VM dest_vm to be up.. 2025-12-22 03:53:45,805 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-12-22 03:53:50,900 - DEBUG - VM is in ACTIVE state now 2025-12-22 03:53:50,900 - INFO - VM name : dest_vm 2025-12-22 03:53:51,005 - DEBUG - VM dest_vm ID is 9ece7127-142a-463e-8a50-7c596555a114 2025-12-22 03:53:51,036 - DEBUG - VM dest_vm launched on Node an-jenkins-deploy-platform-ansible-os-4918-2 2025-12-22 03:53:51,145 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/9ece7127-142a-463e-8a50-7c596555a114 2025-12-22 03:53:51,449 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/9ece7127-142a-463e-8a50-7c596555a114 2025-12-22 03:53:51,490 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine-interface/351434ce-8afd-48f5-9649-e22f88e3ee73 2025-12-22 03:53:54,678 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-12-22 03:53:54,679 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM dest_vm failed! 2025-12-22 03:53:54,695 - DEBUG - Gateway for vn default-domain:ctest-TestMd5testsOnControl-67453638:test_vnv6dn is 2001::201:1 and allocation pool is NOT set 2025-12-22 03:53:54,696 - DEBUG - Gateway for vn default-domain:ctest-TestMd5testsOnControl-67453638:test_vnv6dn is 66.189.20.65 and allocation pool is NOT set 2025-12-22 03:53:58,782 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-12-22 03:53:58,782 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM dest_vm failed! 2025-12-22 03:53:58,798 - DEBUG - Gateway for vn default-domain:ctest-TestMd5testsOnControl-67453638:test_vnv6dn is 2001::201:1 and allocation pool is NOT set 2025-12-22 03:53:58,798 - DEBUG - Gateway for vn default-domain:ctest-TestMd5testsOnControl-67453638:test_vnv6dn is 66.189.20.65 and allocation pool is NOT set 2025-12-22 03:54:00,868 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=5.27 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.954 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.954/3.110/5.267/2.156 ms') 2025-12-22 03:54:00,868 - INFO - Ping to Metadata IP 169.254.0.3 of VM dest_vm passed 2025-12-22 03:54:00,947 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-22 03:54:00,947 - DEBUG - Waiting to SSH to VM dest_vm, IP 66.189.20.67, Port 22 2025-12-22 03:54:01,118 - DEBUG - VM dest_vm is ready for SSH connections 2025-12-22 03:54:01,126 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = any, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestMd5testsOnControl-67453638:test_vnv6sr, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = 0, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestMd5testsOnControl-67453638:test_vnv6dn, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = 0, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-12-22 03:54:01,499 - INFO - Attaching policy allow_all to vn test_vnv6sr 2025-12-22 03:54:01,869 - INFO - Attaching policy allow_all to vn test_vnv6dn 2025-12-22 03:54:02,158 - INFO - Configure the policy with allow any 2025-12-22 03:54:02,528 - DEBUG - Response for create_network : {'network': {'id': '465463ab-8fc2-4995-bbd7-deff4eb7d350', 'name': 'vn1', 'tenant_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'project_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestMd5testsOnControl-67453638', 'vn1'], 'port_security_enabled': True, 'description': ''}} 2025-12-22 03:54:02,792 - DEBUG - Response for create_subnet : {'subnet': {'id': 'd01b674a-b597-4fdf-b8c3-7f966396c059', 'name': '', 'tenant_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'network_id': '465463ab-8fc2-4995-bbd7-deff4eb7d350', 'ip_version': 4, 'cidr': '10.1.1.0/24', 'allocation_pools': [{'start': '10.1.1.2', 'end': '10.1.1.254'}], 'gateway_ip': '10.1.1.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '10.1.1.2', 'tags': [], 'project_id': 'ea4fd547b3bf450daae18a4cece19ee1'}} 2025-12-22 03:54:02,818 - INFO - Created VN vn1 2025-12-22 03:54:02,840 - DEBUG - VN vn1 UUID is 465463ab-8fc2-4995-bbd7-deff4eb7d350 2025-12-22 03:54:03,203 - DEBUG - Response for create_network : {'network': {'id': '8f85251a-e61b-4693-b926-736a556d5914', 'name': 'vn2', 'tenant_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'project_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestMd5testsOnControl-67453638', 'vn2'], 'port_security_enabled': True, 'description': ''}} 2025-12-22 03:54:03,497 - DEBUG - Response for create_subnet : {'subnet': {'id': 'feb6aaa0-9e2f-4990-919c-8cde6bbec53a', 'name': '', 'tenant_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'network_id': '8f85251a-e61b-4693-b926-736a556d5914', 'ip_version': 4, 'cidr': '20.1.1.0/24', 'allocation_pools': [{'start': '20.1.1.2', 'end': '20.1.1.254'}], 'gateway_ip': '20.1.1.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '20.1.1.2', 'tags': [], 'project_id': 'ea4fd547b3bf450daae18a4cece19ee1'}} 2025-12-22 03:54:03,526 - INFO - Created VN vn2 2025-12-22 03:54:03,541 - DEBUG - VN vn2 UUID is 8f85251a-e61b-4693-b926-736a556d5914 2025-12-22 03:54:03,618 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = any, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestMd5testsOnControl-67453638:vn1, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = 0, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestMd5testsOnControl-67453638:vn2, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = 0, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-12-22 03:54:03,994 - INFO - Attaching policy sec_grp_policy_ctest-random-70083688 to vn vn1 2025-12-22 03:54:04,377 - INFO - Attaching policy sec_grp_policy_ctest-random-70083688 to vn vn2 2025-12-22 03:54:06,011 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4918-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4918-1) 2025-12-22 03:54:07,377 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4918-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4918-2) 2025-12-22 03:54:07,444 - INFO - ================================================================================ 2025-12-22 03:54:07,445 - INFO - STARTING TEST : test_check_per_peer_on_control 2025-12-22 03:54:07,445 - INFO - TEST DESCRIPTION : Description: Verify per peer md5 and specific protocol on all ports and policy with allow all between VN's 2025-12-22 03:54:08,766 - DEBUG - Skipping xmpp flap check 2025-12-22 03:54:08,766 - INFO - Initial checks done. Running the testcase now 2025-12-22 03:54:08,766 - INFO - 2025-12-22 03:54:08,766 - INFO - config per peer with auth_data None 2025-12-22 03:54:09,370 - INFO - configure MD5 on host 2915c113-4b21-4453-bb1b-c4dc43459b25 2025-12-22 03:54:09,470 - INFO - config per peer with auth_data None 2025-12-22 03:54:10,082 - INFO - configure MD5 on host b22f0de0-e097-4536-a2fb-bd8e856c13a2 2025-12-22 03:54:10,178 - INFO - config per peer with auth_data None 2025-12-22 03:54:10,779 - INFO - configure MD5 on host 21286f36-7d72-499b-ac55-413527aa7bb0 2025-12-22 03:54:10,874 - INFO - check BGP peering is up before setting md5 config 2025-12-22 03:54:10,874 - INFO - Check BGP staus between peers 2025-12-22 03:54:10,892 - INFO - config per peer with auth_data {'key_items': [{'key': 'juniper', 'key_id': 0}], 'key_type': 'md5'} 2025-12-22 03:54:11,542 - INFO - Check BGP staus between peers 2025-12-22 03:54:11,558 - DEBUG - Check the BGP connection on 10.0.0.23 2025-12-22 03:54:11,558 - DEBUG - Check the BGP connection on 10.0.0.23 2025-12-22 03:54:21,559 - INFO - Check BGP staus between peers 2025-12-22 03:54:21,568 - INFO - remove md5 configured on host 2915c113-4b21-4453-bb1b-c4dc43459b25 2025-12-22 03:54:21,568 - INFO - config per peer with auth_data None 2025-12-22 03:54:22,210 - INFO - configure MD5 on host 2915c113-4b21-4453-bb1b-c4dc43459b25 2025-12-22 03:54:22,302 - INFO - remove md5 configured on host b22f0de0-e097-4536-a2fb-bd8e856c13a2 2025-12-22 03:54:22,302 - INFO - config per peer with auth_data None 2025-12-22 03:54:22,962 - INFO - configure MD5 on host b22f0de0-e097-4536-a2fb-bd8e856c13a2 2025-12-22 03:54:23,054 - INFO - remove md5 configured on host 21286f36-7d72-499b-ac55-413527aa7bb0 2025-12-22 03:54:23,054 - INFO - config per peer with auth_data None 2025-12-22 03:54:23,686 - INFO - configure MD5 on host 21286f36-7d72-499b-ac55-413527aa7bb0 2025-12-22 03:54:23,806 - INFO - Skip interface_detach for VM vn2_vm_0 in state BUILD 2025-12-22 03:54:23,806 - INFO - Deleting VM vn2_vm_0 2025-12-22 03:54:23,806 - ERROR - VM vn2_vm_0 has failed to come up 2025-12-22 03:54:23,806 - ERROR - Nova failed to get host of the VM 2025-12-22 03:54:23,806 - ERROR - VM vn2_vm_0 has failed to come up 2025-12-22 03:54:23,806 - ERROR - Nova failed to get host of the VM 2025-12-22 03:54:23,882 - INFO - Skip interface_detach for VM vn1_vm_0 in state ERROR 2025-12-22 03:54:23,882 - INFO - Deleting VM vn1_vm_0 2025-12-22 03:54:23,882 - ERROR - VM vn1_vm_0 has failed to come up 2025-12-22 03:54:23,883 - ERROR - Fault seen in nova show is: {'code': 500, 'created': '2025-12-22T03:54:05Z', 'message': 'No valid host was found. ', 'details': 'Traceback (most recent call last):\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/conductor/manager.py", line 1581, in schedule_and_build_instances\n instance_uuids, return_alternates=True)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/conductor/manager.py", line 942, in _schedule_instances\n return_alternates=return_alternates)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations\n instance_uuids, return_objects, return_alternates)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations\n return cctxt.call(ctxt, \'select_destinations\', **msg_args)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 192, in call\n retry=self.retry, transport_options=self.transport_options)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/transport.py", line 128, in _send\n transport_options=transport_options)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 691, in send\n transport_options=transport_options)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send\n raise result\nnova.exception_Remote.NoValidHost_Remote: No valid host was found. \nTraceback (most recent call last):\n\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner\n return func(*args, **kwargs)\n\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/scheduler/manager.py", line 209, in select_destinations\n raise exception.NoValidHost(reason="")\n\nnova.exception.NoValidHost: No valid host was found. \n\n'} 2025-12-22 03:54:23,885 - INFO - Dettaching policy sec_grp_policy_ctest-random-70083688 from vn vn2 2025-12-22 03:54:24,066 - INFO - Dettaching policy sec_grp_policy_ctest-random-70083688 from vn vn1 2025-12-22 03:54:24,398 - INFO - Deleted policy sec_grp_policy_ctest-random-70083688 2025-12-22 03:54:24,399 - INFO - Deleting VN vn2 2025-12-22 03:54:24,454 - DEBUG - VN 8f85251a-e61b-4693-b926-736a556d5914 still in use: Unable to complete operation on network 8f85251a-e61b-4693-b926-736a556d5914. There are one or more ports still in use on the network. Neutron server returns request_ids: ['req-d7f14558-528b-460d-88e4-bcc9776c3f35'] 2025-12-22 03:54:24,454 - WARNING - Deleting VN vn2 failed..Will retry 2025-12-22 03:54:26,611 - DEBUG - Response for deleting network () 2025-12-22 03:54:26,611 - INFO - Deleting VN vn1 2025-12-22 03:54:26,825 - DEBUG - Response for deleting network () 2025-12-22 03:54:26,825 - INFO - Dettaching policy allow_all from vn test_vnv6dn 2025-12-22 03:54:27,004 - INFO - Dettaching policy allow_all from vn test_vnv6sr 2025-12-22 03:54:27,249 - INFO - Deleted policy allow_all 2025-12-22 03:54:27,249 - INFO - Deleting VM dest_vm 2025-12-22 03:54:27,320 - INFO - Skip interface_detach for VM source_vm in state ERROR 2025-12-22 03:54:27,320 - INFO - Deleting VM source_vm 2025-12-22 03:54:27,320 - ERROR - VM source_vm has failed to come up 2025-12-22 03:54:27,320 - ERROR - Fault seen in nova show is: {'code': 500, 'created': '2025-12-22T03:53:44Z', 'message': 'No valid host was found. ', 'details': 'Traceback (most recent call last):\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/conductor/manager.py", line 1581, in schedule_and_build_instances\n instance_uuids, return_alternates=True)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/conductor/manager.py", line 942, in _schedule_instances\n return_alternates=return_alternates)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations\n instance_uuids, return_objects, return_alternates)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations\n return cctxt.call(ctxt, \'select_destinations\', **msg_args)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 192, in call\n retry=self.retry, transport_options=self.transport_options)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/transport.py", line 128, in _send\n transport_options=transport_options)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 691, in send\n transport_options=transport_options)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send\n raise result\nnova.exception_Remote.NoValidHost_Remote: No valid host was found. \nTraceback (most recent call last):\n\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner\n return func(*args, **kwargs)\n\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/scheduler/manager.py", line 209, in select_destinations\n raise exception.NoValidHost(reason="")\n\nnova.exception.NoValidHost: No valid host was found. \n\n'} 2025-12-22 03:54:27,321 - INFO - Deleting VN test_vnv6dn 2025-12-22 03:54:27,379 - DEBUG - VN 7e3b8046-8380-45fd-b424-8e3cf32175b3 still in use: Unable to complete operation on network 7e3b8046-8380-45fd-b424-8e3cf32175b3. There are one or more ports still in use on the network. Neutron server returns request_ids: ['req-638756b9-4f22-4acc-be9d-ab32fc949d85'] 2025-12-22 03:54:27,379 - WARNING - Deleting VN test_vnv6dn failed..Will retry 2025-12-22 03:54:29,581 - DEBUG - Response for deleting network () 2025-12-22 03:54:29,581 - INFO - Deleting VN test_vnv6sr 2025-12-22 03:54:29,786 - DEBUG - Response for deleting network () 2025-12-22 03:54:31,089 - DEBUG - Skipping xmpp flap check 2025-12-22 03:54:31,089 - INFO - END TEST : test_check_per_peer_on_control : PASSED[0:00:24] 2025-12-22 03:54:31,089 - INFO - -------------------------------------------------------------------------------- 2025-12-22 03:54:31,381 - DEBUG - Response for create_network : {'network': {'id': 'e8916deb-cadb-491b-8b6c-ed2a81672fe7', 'name': 'test_vnv6sr', 'tenant_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'project_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestMd5testsOnControl-67453638', 'test_vnv6sr'], 'port_security_enabled': True, 'description': ''}} 2025-12-22 03:54:31,585 - DEBUG - Response for create_subnet : {'subnet': {'id': '0e6647d3-c4db-41d1-aeb8-41d3c2a48b25', 'name': '', 'tenant_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'network_id': 'e8916deb-cadb-491b-8b6c-ed2a81672fe7', 'ip_version': 6, 'cidr': '2001::101:0/120', 'allocation_pools': [{'start': '2001::101:2', 'end': '2001::101:fe'}], 'gateway_ip': '2001::101:1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '2001::101:2', 'tags': [], 'project_id': 'ea4fd547b3bf450daae18a4cece19ee1'}} 2025-12-22 03:54:31,869 - DEBUG - Response for create_subnet : {'subnet': {'id': '0d98c031-cecf-4981-a9e8-6f15c22188f1', 'name': '', 'tenant_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'network_id': 'e8916deb-cadb-491b-8b6c-ed2a81672fe7', 'ip_version': 4, 'cidr': '107.100.187.128/26', 'allocation_pools': [{'start': '107.100.187.130', 'end': '107.100.187.190'}], 'gateway_ip': '107.100.187.129', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '107.100.187.130', 'tags': [], 'project_id': 'ea4fd547b3bf450daae18a4cece19ee1'}} 2025-12-22 03:54:31,891 - INFO - Created VN test_vnv6sr 2025-12-22 03:54:31,904 - DEBUG - VN test_vnv6sr UUID is e8916deb-cadb-491b-8b6c-ed2a81672fe7 2025-12-22 03:54:32,317 - DEBUG - Response for create_network : {'network': {'id': '9d21a0ad-7ece-497c-b55d-eaba572ec084', 'name': 'test_vnv6dn', 'tenant_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'project_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestMd5testsOnControl-67453638', 'test_vnv6dn'], 'port_security_enabled': True, 'description': ''}} 2025-12-22 03:54:32,557 - DEBUG - Response for create_subnet : {'subnet': {'id': '20b1bc44-4e0a-4d90-91a4-407085d64d58', 'name': '', 'tenant_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'network_id': '9d21a0ad-7ece-497c-b55d-eaba572ec084', 'ip_version': 6, 'cidr': '2001::201:0/120', 'allocation_pools': [{'start': '2001::201:2', 'end': '2001::201:fe'}], 'gateway_ip': '2001::201:1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '2001::201:2', 'tags': [], 'project_id': 'ea4fd547b3bf450daae18a4cece19ee1'}} 2025-12-22 03:54:32,824 - DEBUG - Response for create_subnet : {'subnet': {'id': '764e8d6b-11b0-426b-8f2e-70d0694bbdd7', 'name': '', 'tenant_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'network_id': '9d21a0ad-7ece-497c-b55d-eaba572ec084', 'ip_version': 4, 'cidr': '193.103.23.192/26', 'allocation_pools': [{'start': '193.103.23.194', 'end': '193.103.23.254'}], 'gateway_ip': '193.103.23.193', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '193.103.23.194', 'tags': [], 'project_id': 'ea4fd547b3bf450daae18a4cece19ee1'}} 2025-12-22 03:54:32,847 - INFO - Created VN test_vnv6dn 2025-12-22 03:54:32,859 - DEBUG - VN test_vnv6dn UUID is 9d21a0ad-7ece-497c-b55d-eaba572ec084 2025-12-22 03:54:33,168 - DEBUG - VM source_vm already present, not creating it 2025-12-22 03:54:34,334 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4918-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4918-1) 2025-12-22 03:54:34,334 - INFO - Waiting for VM source_vm to be up.. 2025-12-22 03:54:34,397 - DEBUG - VM is in ERROR state now 2025-12-22 03:54:34,397 - ERROR - VM in error state. 2025-12-22 03:54:34,397 - ERROR - VM is not in ACTIVE state 2025-12-22 03:54:34,470 - ERROR - Fatal Nova Exception while getting VM detail 2025-12-22 03:54:34,470 - DEBUG - None 2025-12-22 03:54:34,471 - INFO - Waiting for VM dest_vm to be up.. 2025-12-22 03:54:34,525 - DEBUG - VM is in ERROR state now 2025-12-22 03:54:34,525 - ERROR - VM in error state. 2025-12-22 03:54:34,525 - ERROR - VM is not in ACTIVE state 2025-12-22 03:54:34,609 - ERROR - Fatal Nova Exception while getting VM detail 2025-12-22 03:54:34,609 - DEBUG - None 2025-12-22 03:54:34,615 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = any, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestMd5testsOnControl-67453638:test_vnv6sr, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = 0, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestMd5testsOnControl-67453638:test_vnv6dn, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = 0, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-12-22 03:54:34,990 - INFO - Attaching policy allow_all to vn test_vnv6sr 2025-12-22 03:54:35,390 - INFO - Attaching policy allow_all to vn test_vnv6dn 2025-12-22 03:54:35,648 - INFO - Configure the policy with allow any 2025-12-22 03:54:35,969 - DEBUG - Response for create_network : {'network': {'id': 'f3445bc3-451f-46d9-815d-d5add93a33f2', 'name': 'vn1', 'tenant_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'project_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestMd5testsOnControl-67453638', 'vn1'], 'port_security_enabled': True, 'description': ''}} 2025-12-22 03:54:36,248 - DEBUG - Response for create_subnet : {'subnet': {'id': '5d480cd5-eccb-446a-9a4b-27a16f668224', 'name': '', 'tenant_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'network_id': 'f3445bc3-451f-46d9-815d-d5add93a33f2', 'ip_version': 4, 'cidr': '10.1.1.0/24', 'allocation_pools': [{'start': '10.1.1.2', 'end': '10.1.1.254'}], 'gateway_ip': '10.1.1.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '10.1.1.2', 'tags': [], 'project_id': 'ea4fd547b3bf450daae18a4cece19ee1'}} 2025-12-22 03:54:36,271 - INFO - Created VN vn1 2025-12-22 03:54:36,283 - DEBUG - VN vn1 UUID is f3445bc3-451f-46d9-815d-d5add93a33f2 2025-12-22 03:54:36,652 - DEBUG - Response for create_network : {'network': {'id': 'f5945947-f0a7-46dd-ac88-32018fff2ea2', 'name': 'vn2', 'tenant_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'project_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestMd5testsOnControl-67453638', 'vn2'], 'port_security_enabled': True, 'description': ''}} 2025-12-22 03:54:36,860 - DEBUG - Response for create_subnet : {'subnet': {'id': '11021492-a7bf-482b-81a9-59d76470d03b', 'name': '', 'tenant_id': 'ea4fd547b3bf450daae18a4cece19ee1', 'network_id': 'f5945947-f0a7-46dd-ac88-32018fff2ea2', 'ip_version': 4, 'cidr': '20.1.1.0/24', 'allocation_pools': [{'start': '20.1.1.2', 'end': '20.1.1.254'}], 'gateway_ip': '20.1.1.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '20.1.1.2', 'tags': [], 'project_id': 'ea4fd547b3bf450daae18a4cece19ee1'}} 2025-12-22 03:54:36,882 - INFO - Created VN vn2 2025-12-22 03:54:36,898 - DEBUG - VN vn2 UUID is f5945947-f0a7-46dd-ac88-32018fff2ea2 2025-12-22 03:54:37,010 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = any, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestMd5testsOnControl-67453638:vn1, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = 0, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestMd5testsOnControl-67453638:vn2, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = 0, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-12-22 03:54:37,349 - INFO - Attaching policy sec_grp_policy_ctest-random-53481838 to vn vn1 2025-12-22 03:54:37,720 - INFO - Attaching policy sec_grp_policy_ctest-random-53481838 to vn vn2 2025-12-22 03:54:38,254 - DEBUG - VM vn1_vm_0 already present, not creating it 2025-12-22 03:54:39,407 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4918-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4918-2) 2025-12-22 03:54:39,471 - INFO - ================================================================================ 2025-12-22 03:54:39,471 - INFO - STARTING TEST : test_create_md5_on_control 2025-12-22 03:54:39,471 - INFO - TEST DESCRIPTION : Description: Verify md5 with allow specific protocol on all ports and policy with allow all between VN's 2025-12-22 03:54:40,785 - DEBUG - Skipping xmpp flap check 2025-12-22 03:54:40,786 - INFO - Initial checks done. Running the testcase now 2025-12-22 03:54:40,786 - INFO - 2025-12-22 03:54:40,786 - INFO - config md5 on each host with auth_data None 2025-12-22 03:54:40,786 - INFO - config per peer with auth_data None 2025-12-22 03:54:41,458 - INFO - configure MD5 on host 2915c113-4b21-4453-bb1b-c4dc43459b25 2025-12-22 03:54:41,550 - INFO - config per peer with auth_data None 2025-12-22 03:54:42,198 - INFO - configure MD5 on host b22f0de0-e097-4536-a2fb-bd8e856c13a2 2025-12-22 03:54:42,291 - INFO - config per peer with auth_data None 2025-12-22 03:54:42,898 - INFO - configure MD5 on host 21286f36-7d72-499b-ac55-413527aa7bb0 2025-12-22 03:54:42,998 - INFO - check if BGP between peers are up before setting md5 2025-12-22 03:54:42,998 - INFO - Check BGP staus between peers 2025-12-22 03:54:43,007 - INFO - setting MD5 config on host 2915c113-4b21-4453-bb1b-c4dc43459b25 2025-12-22 03:54:43,008 - INFO - configure MD5 on host 2915c113-4b21-4453-bb1b-c4dc43459b25 2025-12-22 03:54:43,094 - INFO - setting MD5 config on host b22f0de0-e097-4536-a2fb-bd8e856c13a2 2025-12-22 03:54:43,094 - INFO - configure MD5 on host b22f0de0-e097-4536-a2fb-bd8e856c13a2 2025-12-22 03:54:43,178 - INFO - setting MD5 config on host 21286f36-7d72-499b-ac55-413527aa7bb0 2025-12-22 03:54:43,178 - INFO - configure MD5 on host 21286f36-7d72-499b-ac55-413527aa7bb0 2025-12-22 03:54:43,254 - INFO - check if BGP peering is up after md5 config 2025-12-22 03:54:43,254 - INFO - Check BGP staus between peers 2025-12-22 03:54:43,263 - DEBUG - Check the BGP connection on 10.0.0.23 2025-12-22 03:54:43,263 - DEBUG - Check the BGP connection on 10.0.0.23 2025-12-22 03:54:53,264 - INFO - Check BGP staus between peers 2025-12-22 03:54:53,275 - INFO - remove md5 configured on host 2915c113-4b21-4453-bb1b-c4dc43459b25 2025-12-22 03:54:53,275 - INFO - config per peer with auth_data None 2025-12-22 03:54:53,894 - INFO - configure MD5 on host 2915c113-4b21-4453-bb1b-c4dc43459b25 2025-12-22 03:54:54,002 - INFO - remove md5 configured on host b22f0de0-e097-4536-a2fb-bd8e856c13a2 2025-12-22 03:54:54,002 - INFO - config per peer with auth_data None 2025-12-22 03:54:54,618 - INFO - configure MD5 on host b22f0de0-e097-4536-a2fb-bd8e856c13a2 2025-12-22 03:54:54,702 - INFO - remove md5 configured on host 21286f36-7d72-499b-ac55-413527aa7bb0 2025-12-22 03:54:54,702 - INFO - config per peer with auth_data None 2025-12-22 03:54:55,306 - INFO - configure MD5 on host 21286f36-7d72-499b-ac55-413527aa7bb0 2025-12-22 03:54:55,398 - INFO - Skip interface_detach for VM vn2_vm_0 in state BUILD 2025-12-22 03:54:55,398 - INFO - Deleting VM vn2_vm_0 2025-12-22 03:54:55,398 - ERROR - VM vn2_vm_0 has failed to come up 2025-12-22 03:54:55,398 - ERROR - Nova failed to get host of the VM 2025-12-22 03:54:55,398 - ERROR - VM vn2_vm_0 has failed to come up 2025-12-22 03:54:55,398 - ERROR - Nova failed to get host of the VM 2025-12-22 03:54:55,484 - INFO - Skipping the deletion of VM vn1_vm_0 2025-12-22 03:54:55,484 - INFO - Dettaching policy sec_grp_policy_ctest-random-53481838 from vn vn2 2025-12-22 03:54:55,664 - INFO - Dettaching policy sec_grp_policy_ctest-random-53481838 from vn vn1 2025-12-22 03:54:55,914 - INFO - Deleted policy sec_grp_policy_ctest-random-53481838 2025-12-22 03:54:55,914 - INFO - Deleting VN vn2 2025-12-22 03:54:55,956 - DEBUG - VN f5945947-f0a7-46dd-ac88-32018fff2ea2 still in use: Unable to complete operation on network f5945947-f0a7-46dd-ac88-32018fff2ea2. There are one or more ports still in use on the network. Neutron server returns request_ids: ['req-3f6046c0-36be-4601-bfb7-5193b51f51d8'] 2025-12-22 03:54:55,956 - WARNING - Deleting VN vn2 failed..Will retry 2025-12-22 03:54:58,153 - DEBUG - Response for deleting network () 2025-12-22 03:54:58,153 - INFO - Deleting VN vn1 2025-12-22 03:54:58,307 - DEBUG - Response for deleting network () 2025-12-22 03:54:58,307 - INFO - Dettaching policy allow_all from vn test_vnv6dn 2025-12-22 03:54:58,518 - INFO - Dettaching policy allow_all from vn test_vnv6sr 2025-12-22 03:54:58,798 - INFO - Deleted policy allow_all 2025-12-22 03:54:58,798 - INFO - Skip interface_detach for VM dest_vm in state ERROR 2025-12-22 03:54:58,799 - INFO - Deleting VM dest_vm 2025-12-22 03:54:58,799 - ERROR - VM dest_vm has failed to come up 2025-12-22 03:54:58,799 - ERROR - Fault seen in nova show is: {'code': 500, 'created': '2025-12-22T03:54:34Z', 'message': 'No valid host was found. ', 'details': 'Traceback (most recent call last):\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/conductor/manager.py", line 1581, in schedule_and_build_instances\n instance_uuids, return_alternates=True)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/conductor/manager.py", line 942, in _schedule_instances\n return_alternates=return_alternates)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations\n instance_uuids, return_objects, return_alternates)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations\n return cctxt.call(ctxt, \'select_destinations\', **msg_args)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 192, in call\n retry=self.retry, transport_options=self.transport_options)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/transport.py", line 128, in _send\n transport_options=transport_options)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 691, in send\n transport_options=transport_options)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send\n raise result\nnova.exception_Remote.NoValidHost_Remote: No valid host was found. \nTraceback (most recent call last):\n\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner\n return func(*args, **kwargs)\n\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/scheduler/manager.py", line 209, in select_destinations\n raise exception.NoValidHost(reason="")\n\nnova.exception.NoValidHost: No valid host was found. \n\n'} 2025-12-22 03:54:58,799 - INFO - Skipping the deletion of VM source_vm 2025-12-22 03:54:58,799 - INFO - Deleting VN test_vnv6dn 2025-12-22 03:54:59,013 - DEBUG - Response for deleting network () 2025-12-22 03:54:59,013 - INFO - Deleting VN test_vnv6sr 2025-12-22 03:54:59,238 - DEBUG - Response for deleting network () 2025-12-22 03:55:00,529 - DEBUG - Skipping xmpp flap check 2025-12-22 03:55:00,529 - INFO - END TEST : test_create_md5_on_control : PASSED[0:00:21] 2025-12-22 03:55:00,529 - INFO - -------------------------------------------------------------------------------- 2025-12-22 03:55:01,320 - INFO - Deleted project: ctest-TestMd5testsOnControl-67453638, ID : ea4fd547-b3bf-450d-aae1-8a4cece19ee1