2025-12-22 03:59:32,198 - INFO - Domain Default found not creating 2025-12-22 03:59:32,349 - INFO - Project ctest-FatFlowSerialIpv6-50712872 not found, creating it 2025-12-22 03:59:32,855 - INFO - Created Project:ctest-FatFlowSerialIpv6-50712872, ID : a7a8c85d-44f9-46e9-9e56-c340de4540e4 2025-12-22 03:59:34,438 - INFO - ================================================================================ 2025-12-22 03:59:34,438 - INFO - STARTING TEST : test_fat_flow_intra_vn_inter_node 2025-12-22 03:59:34,438 - INFO - TEST DESCRIPTION : Description: Verify Fat flow for intra-VN inter-Node traffic Steps: 1. create 1 VN and launch 3 VMs in it.client VMs on same node and server VM on different node. 2. on server VM, config Fat flow for udp port 53. 3. from both client VM, send UDP traffic to server on port 53 twice with diff. src ports Pass criteria: 1. on client VMs compute, 4 set of flows and on server compute, 2 set of flows should be created 2. on server compute, flow's source port should be 0 for Fat flow 2025-12-22 03:59:35,719 - DEBUG - Skipping xmpp flap check 2025-12-22 03:59:35,719 - INFO - Initial checks done. Running the testcase now 2025-12-22 03:59:35,719 - INFO - 2025-12-22 03:59:36,403 - DEBUG - Response for create_network : {'network': {'id': 'c4fab8f7-4766-428f-ab83-71bcb2401de2', 'name': 'ctest-vn-56076709', 'tenant_id': 'a7a8c85d44f946e99e56c340de4540e4', 'project_id': 'a7a8c85d44f946e99e56c340de4540e4', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-FatFlowSerialIpv6-50712872', 'ctest-vn-56076709'], 'port_security_enabled': True, 'description': ''}} 2025-12-22 03:59:36,742 - DEBUG - Response for create_subnet : {'subnet': {'id': '6f7cf1cc-35ec-4d89-a141-c0b1033eb462', 'name': '', 'tenant_id': 'a7a8c85d44f946e99e56c340de4540e4', 'network_id': 'c4fab8f7-4766-428f-ab83-71bcb2401de2', 'ip_version': 4, 'cidr': '90.69.51.128/26', 'allocation_pools': [{'start': '90.69.51.130', 'end': '90.69.51.190'}], 'gateway_ip': '90.69.51.129', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '90.69.51.130', 'tags': [], 'project_id': 'a7a8c85d44f946e99e56c340de4540e4'}} 2025-12-22 03:59:36,958 - DEBUG - Response for create_subnet : {'subnet': {'id': 'dd4e487b-8089-455c-b1e9-aa1246f784cc', 'name': '', 'tenant_id': 'a7a8c85d44f946e99e56c340de4540e4', 'network_id': 'c4fab8f7-4766-428f-ab83-71bcb2401de2', 'ip_version': 6, 'cidr': '3ecb:e24c:fb5e:11dd:5961:4547::/96', 'allocation_pools': [{'start': '3ecb:e24c:fb5e:11dd:5961:4547:0:2', 'end': '3ecb:e24c:fb5e:11dd:5961:4547:ffff:fffe'}], 'gateway_ip': '3ecb:e24c:fb5e:11dd:5961:4547:0:1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '3ecb:e24c:fb5e:11dd:5961:4547:0:2', 'tags': [], 'project_id': 'a7a8c85d44f946e99e56c340de4540e4'}} 2025-12-22 03:59:36,987 - INFO - Created VN ctest-vn-56076709 2025-12-22 03:59:36,999 - DEBUG - VN ctest-vn-56076709 UUID is c4fab8f7-4766-428f-ab83-71bcb2401de2 2025-12-22 03:59:37,098 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-12-22 03:59:37,394 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-12-22 03:59:37,422 - DEBUG - Requesting: http://10.0.0.23:8082/domain/cc89f09a-44c2-47d5-97ff-1101c2b1f822 2025-12-22 03:59:37,439 - DEBUG - Requesting: http://10.0.0.23:8082/project/a7a8c85d-44f9-46e9-9e56-c340de4540e4 2025-12-22 03:59:37,535 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/c4fab8f7-4766-428f-ab83-71bcb2401de2 2025-12-22 03:59:37,546 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/c4fab8f7-4766-428f-ab83-71bcb2401de2 2025-12-22 03:59:37,554 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/e183222f-7830-47d3-9bc8-f3c0f77610d1 2025-12-22 03:59:37,561 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/e183222f-7830-47d3-9bc8-f3c0f77610d1 2025-12-22 03:59:37,567 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/d7de339e-82c7-4214-9936-680cb4a64100 2025-12-22 03:59:37,573 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-12-22 03:59:37,573 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/c4fab8f7-4766-428f-ab83-71bcb2401de2 2025-12-22 03:59:37,583 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/e183222f-7830-47d3-9bc8-f3c0f77610d1 2025-12-22 03:59:37,599 - INFO - Verified VN network id 11 for VN c4fab8f7-4766-428f-ab83-71bcb2401de2 2025-12-22 03:59:37,599 - INFO - Verifications in API Server for VN ctest-vn-56076709 passed 2025-12-22 03:59:37,599 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/c4fab8f7-4766-428f-ab83-71bcb2401de2 2025-12-22 03:59:37,608 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/e183222f-7830-47d3-9bc8-f3c0f77610d1 2025-12-22 03:59:37,614 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/e183222f-7830-47d3-9bc8-f3c0f77610d1 2025-12-22 03:59:37,620 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/d7de339e-82c7-4214-9936-680cb4a64100 2025-12-22 03:59:37,633 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-FatFlowSerialIpv6-50712872:ctest-vn-56076709', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-FatFlowSerialIpv6-50712872', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '14193860547805856399', 'uuid-lslong': '12358846857693109730'}, 'enable': 'true', 'created': '2025-12-22T03:59:36', 'last-modified': '2025-12-22T03:59:36', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.753974'} 2025-12-22 03:59:37,638 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-12-22 03:59:37,643 - DEBUG - Control-node 10.0.0.22 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-FatFlowSerialIpv6-50712872:ctest-vn-56076709', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-FatFlowSerialIpv6-50712872', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '14193860547805856399', 'uuid-lslong': '12358846857693109730'}, 'enable': 'true', 'created': '2025-12-22T03:59:36', 'last-modified': '2025-12-22T03:59:36', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.760930'} 2025-12-22 03:59:37,650 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-12-22 03:59:37,655 - DEBUG - Control-node 10.0.0.65 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-FatFlowSerialIpv6-50712872:ctest-vn-56076709', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-FatFlowSerialIpv6-50712872', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '14193860547805856399', 'uuid-lslong': '12358846857693109730'}, 'enable': 'true', 'created': '2025-12-22T03:59:36', 'last-modified': '2025-12-22T03:59:36', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.772575'} 2025-12-22 03:59:37,661 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-12-22 03:59:37,661 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn-56076709 passed 2025-12-22 03:59:37,661 - DEBUG - ====Verifying policy data for ctest-vn-56076709 in API_Server ====== 2025-12-22 03:59:37,661 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-12-22 03:59:37,667 - DEBUG - Requesting: http://10.0.0.23:8082/domain/cc89f09a-44c2-47d5-97ff-1101c2b1f822 2025-12-22 03:59:37,680 - DEBUG - Requesting: http://10.0.0.23:8082/project/a7a8c85d-44f9-46e9-9e56-c340de4540e4 2025-12-22 03:59:37,770 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/c4fab8f7-4766-428f-ab83-71bcb2401de2 2025-12-22 03:59:37,781 - DEBUG - =>VN ctest-vn-56076709 has no policy to be verified 2025-12-22 03:59:37,781 - DEBUG - Verifying the vn in opserver 2025-12-22 03:59:37,781 - DEBUG - Verifying the default-domain:ctest-FatFlowSerialIpv6-50712872:ctest-vn-56076709 virtual network link through opserver 10.0.0.23 2025-12-22 03:59:37,782 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-12-22 03:59:38,080 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-12-22 03:59:38,141 - DEBUG - vn link and name as {'name': 'default-domain:ctest-FatFlowSerialIpv6-50712872:ctest-vn-56076709', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-FatFlowSerialIpv6-50712872:ctest-vn-56076709?flat'} 2025-12-22 03:59:38,141 - DEBUG - Verifying the default-domain:ctest-FatFlowSerialIpv6-50712872:ctest-vn-56076709 virtual network link through opserver 10.0.0.22 2025-12-22 03:59:38,141 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-networks 2025-12-22 03:59:38,439 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-networks 2025-12-22 03:59:38,498 - DEBUG - vn link and name as {'name': 'default-domain:ctest-FatFlowSerialIpv6-50712872:ctest-vn-56076709', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-network/default-domain:ctest-FatFlowSerialIpv6-50712872:ctest-vn-56076709?flat'} 2025-12-22 03:59:38,499 - DEBUG - Verifying the default-domain:ctest-FatFlowSerialIpv6-50712872:ctest-vn-56076709 virtual network link through opserver 10.0.0.65 2025-12-22 03:59:38,499 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-networks 2025-12-22 03:59:38,808 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-networks 2025-12-22 03:59:38,866 - DEBUG - vn link and name as {'name': 'default-domain:ctest-FatFlowSerialIpv6-50712872:ctest-vn-56076709', 'href': 'http://10.0.0.65:8081/analytics/uves/virtual-network/default-domain:ctest-FatFlowSerialIpv6-50712872:ctest-vn-56076709?flat'} 2025-12-22 03:59:38,866 - INFO - Validated that VN default-domain:ctest-FatFlowSerialIpv6-50712872:ctest-vn-56076709 is found in opserver 2025-12-22 03:59:38,888 - DEBUG - Do not have enough data to verify VN in agent 2025-12-22 03:59:38,900 - DEBUG - VRF ids for VN ctest-vn-56076709: {} 2025-12-22 03:59:38,923 - DEBUG - Services list from nova: [, , , ] 2025-12-22 03:59:38,923 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-4918-1', 'an-jenkins-deploy-platform-ansible-os-4918-2']} 2025-12-22 03:59:38,923 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-4918-1', 'an-jenkins-deploy-platform-ansible-os-4918-2']} 2025-12-22 03:59:40,251 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4918-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4918-1) 2025-12-22 03:59:41,466 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4918-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4918-1) 2025-12-22 03:59:41,466 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-4918-1', 'an-jenkins-deploy-platform-ansible-os-4918-2']} 2025-12-22 03:59:42,730 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4918-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4918-2) 2025-12-22 03:59:42,730 - INFO - Waiting for VM ctest-FatFlowSerialIpv6-50712872-20360735 to be up.. 2025-12-22 03:59:42,789 - DEBUG - VM is in ERROR state now 2025-12-22 03:59:42,789 - ERROR - VM in error state. 2025-12-22 03:59:42,789 - ERROR - VM is not in ACTIVE state 2025-12-22 03:59:42,872 - ERROR - Fatal Nova Exception while getting VM detail 2025-12-22 03:59:42,872 - DEBUG - None 2025-12-22 03:59:42,875 - INFO - Skip interface_detach for VM ctest-FatFlowSerialIpv6-50712872-99822102 in state BUILD 2025-12-22 03:59:42,875 - INFO - Deleting VM ctest-FatFlowSerialIpv6-50712872-99822102 2025-12-22 03:59:42,900 - ERROR - VM ctest-FatFlowSerialIpv6-50712872-99822102 has failed to come up 2025-12-22 03:59:42,900 - ERROR - Nova failed to get host of the VM 2025-12-22 03:59:42,901 - ERROR - VM ctest-FatFlowSerialIpv6-50712872-99822102 has failed to come up 2025-12-22 03:59:42,901 - ERROR - Nova failed to get host of the VM 2025-12-22 03:59:42,961 - INFO - Skip interface_detach for VM ctest-FatFlowSerialIpv6-50712872-86765884 in state BUILD 2025-12-22 03:59:42,961 - INFO - Deleting VM ctest-FatFlowSerialIpv6-50712872-86765884 2025-12-22 03:59:42,961 - ERROR - VM ctest-FatFlowSerialIpv6-50712872-86765884 has failed to come up 2025-12-22 03:59:42,962 - ERROR - Nova failed to get host of the VM 2025-12-22 03:59:42,962 - ERROR - VM ctest-FatFlowSerialIpv6-50712872-86765884 has failed to come up 2025-12-22 03:59:42,962 - ERROR - Nova failed to get host of the VM 2025-12-22 03:59:43,188 - INFO - Skip interface_detach for VM ctest-FatFlowSerialIpv6-50712872-20360735 in state ERROR 2025-12-22 03:59:43,188 - INFO - Deleting VM ctest-FatFlowSerialIpv6-50712872-20360735 2025-12-22 03:59:43,188 - ERROR - VM ctest-FatFlowSerialIpv6-50712872-20360735 has failed to come up 2025-12-22 03:59:43,188 - ERROR - Fault seen in nova show is: {'code': 500, 'created': '2025-12-22T03:59:40Z', 'message': 'No valid host was found. ', 'details': 'Traceback (most recent call last):\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/conductor/manager.py", line 1581, in schedule_and_build_instances\n instance_uuids, return_alternates=True)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/conductor/manager.py", line 942, in _schedule_instances\n return_alternates=return_alternates)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations\n instance_uuids, return_objects, return_alternates)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations\n return cctxt.call(ctxt, \'select_destinations\', **msg_args)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 192, in call\n retry=self.retry, transport_options=self.transport_options)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/transport.py", line 128, in _send\n transport_options=transport_options)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 691, in send\n transport_options=transport_options)\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send\n raise result\nnova.exception_Remote.NoValidHost_Remote: No valid host was found. \nTraceback (most recent call last):\n\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner\n return func(*args, **kwargs)\n\n File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/scheduler/manager.py", line 209, in select_destinations\n raise exception.NoValidHost(reason="")\n\nnova.exception.NoValidHost: No valid host was found. \n\n'} 2025-12-22 03:59:43,189 - INFO - Deleting VN ctest-vn-56076709 2025-12-22 03:59:43,361 - DEBUG - Response for deleting network () 2025-12-22 03:59:43,362 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/e183222f-7830-47d3-9bc8-f3c0f77610d1 2025-12-22 03:59:43,375 - DEBUG - Response Code: 404 2025-12-22 03:59:43,375 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-12-22 03:59:43,385 - DEBUG - Requesting: http://10.0.0.23:8082/domain/cc89f09a-44c2-47d5-97ff-1101c2b1f822 2025-12-22 03:59:43,399 - DEBUG - Requesting: http://10.0.0.23:8082/project/a7a8c85d-44f9-46e9-9e56-c340de4540e4 2025-12-22 03:59:43,483 - INFO - Validated that VN ctest-vn-56076709 is not found in API Server 2025-12-22 03:59:43,494 - DEBUG - VN ctest-vn-56076709 is not present in Agent 10.0.0.130 2025-12-22 03:59:43,508 - DEBUG - VN ctest-vn-56076709 is not present in Agent 10.0.0.19 2025-12-22 03:59:43,508 - INFO - Validated that VN ctest-vn-56076709 is not in any agent 2025-12-22 03:59:43,540 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn-56076709 info 2025-12-22 03:59:44,808 - ERROR - AssertionError Python 3.9.21: /usr/bin/python3 Mon Dec 22 03:59:42 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={}) 78 log.info('Initial checks done. Running the testcase now') 79 log.info('') 80 result = function(self, *args, **kwargs) 81 if self.inputs.upgrade: 82 pid = os.getpid() result = None function = self = args = () kwargs = {} /contrail-test/serial_scripts/vrouter/test_fat_flow_serial.py in test_fat_flow_intra_vn_inter_node(self=) 42 server_fixtures = self.create_vms(vn_fixture= vn1_fixture,count=1, 43 node_name=compute_hosts[1]) 44 self.verify_vms(client_fixtures) 45 self.verify_vms(server_fixtures) 46 self = self.verify_vms = > client_fixtures = [, ] /contrail-test/common/vrouter/base.py in verify_vms(self=, vm_fixtures=[, ]) 198 def verify_vms(self, vm_fixtures): 199 for vm_fixture in vm_fixtures: 200 assert vm_fixture.wait_till_vm_is_up() 201 202 def add_static_routes_on_vms(self,prefix, vm_fixtures, ip=None): vm_fixture = vm_fixture.wait_till_vm_is_up = > AssertionError: __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {} __dir__ = __doc__ = 'Assertion failed.' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = args = () with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper result = function(self, *args, **kwargs) File "/contrail-test/serial_scripts/vrouter/test_fat_flow_serial.py", line 44, in test_fat_flow_intra_vn_inter_node self.verify_vms(client_fixtures) File "/contrail-test/common/vrouter/base.py", line 200, in verify_vms assert vm_fixture.wait_till_vm_is_up() AssertionError 2025-12-22 03:59:44,810 - DEBUG - Skipping xmpp flap check 2025-12-22 03:59:44,810 - INFO -
2025-12-22 03:59:44,810 - INFO - END TEST : test_fat_flow_intra_vn_inter_node : FAILED[0:00:10]
2025-12-22 03:59:44,810 - INFO - -------------------------------------------------------------------------------- 2025-12-22 03:59:45,601 - INFO - Deleted project: ctest-FatFlowSerialIpv6-50712872, ID : a7a8c85d-44f9-46e9-9e56-c340de4540e4