2025-08-27 14:16:35,708 - INFO - Domain Default found not creating
2025-08-27 14:16:35,842 - INFO - Project ctest-TestSvcRegr-30847890 not found, creating it
2025-08-27 14:16:36,266 - INFO - Created Project:ctest-TestSvcRegr-30847890, ID : 9656c20d-04f0-40d9-b9ae-3cdb95830e35 
2025-08-27 14:16:37,617 - INFO - ================================================================================
2025-08-27 14:16:37,617 - INFO - STARTING TEST    : test_svc_in_network_datapath
2025-08-27 14:16:37,741 - DEBUG - Nothing to compare xmpp stats {'10.0.0.38': {'10.20.0.254': '0'}} with
2025-08-27 14:16:37,741 - INFO - Initial checks done. Running the testcase now
2025-08-27 14:16:37,741 - INFO - 
2025-08-27 14:16:38,267 - DEBUG - Response for create_network : {'network': {'id': '8857dd63-b2ac-4e94-b5ac-b7cb9f1d052f', 'name': 'ctest-mgmt_vn-06711312', 'tenant_id': '9656c20d04f040d9b9ae3cdb95830e35', 'project_id': '9656c20d04f040d9b9ae3cdb95830e35', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestSvcRegr-30847890', 'ctest-mgmt_vn-06711312'], 'port_security_enabled': True, 'description': ''}}
2025-08-27 14:16:38,453 - ERROR - Neutron Exception while creating subnet for vn with id 8857dd63-b2ac-4e94-b5ac-b7cb9f1d052f
Traceback (most recent call last):
  File "/contrail-test/fixtures/quantum_test.py", line 128, in create_subnet
    subnet_rsp = self.obj.create_subnet({'subnet': subnet_req})
  File "/usr/local/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 848, in create_subnet
    return self.post(self.subnets_path, body=body)
  File "/usr/local/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 358, in post
    return self.do_request("POST", action, body=body,
  File "/usr/local/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
    self._handle_fault_response(status_code, replybody, resp)
  File "/usr/local/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
    exception_handler_v20(status_code, error_body)
  File "/usr/local/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
    raise client_exc(message=error_message,
neutronclient.common.exceptions.InternalServerError: The server has either erred or is incapable of performing the requested operation.<br /><br />



Neutron server returns request_ids: ['req-c0667c4d-c173-4ca1-bd55-8fb8b3d10606']
2025-08-27 14:16:38,469 - INFO - Created VN ctest-mgmt_vn-06711312
2025-08-27 14:16:38,519 - DEBUG - VN ctest-mgmt_vn-06711312 UUID is 8857dd63-b2ac-4e94-b5ac-b7cb9f1d052f
2025-08-27 14:16:38,598 - DEBUG - Requesting: http://10.0.0.38:8082/domains
2025-08-27 14:16:38,887 - DEBUG - Requesting: http://10.0.0.38:8082/domains
2025-08-27 14:16:38,913 - DEBUG - Requesting: http://10.0.0.38:8082/domain/9130d1c2-5fd5-4a37-8697-96bb219a4e4c
2025-08-27 14:16:38,923 - DEBUG - Requesting: http://10.0.0.38:8082/project/9656c20d-04f0-40d9-b9ae-3cdb95830e35
2025-08-27 14:16:38,993 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/8857dd63-b2ac-4e94-b5ac-b7cb9f1d052f
2025-08-27 14:16:39,007 - INFO - Deleting VN ctest-mgmt_vn-06711312 
2025-08-27 14:16:39,073 - DEBUG - Response for deleting network ()
2025-08-27 14:16:39,191 - ERROR - KeyError
Python 3.9.21: /usr/bin/python3
Wed Aug 27 14:16:39 2025

A problem occurred in a Python script.  Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /contrail-test/tcutils/wrappers.py in wrapper(self=<scripts.svc_firewall.test_svc_fw.TestSvcRegr.te...in_network_datapath[ci_sanity] id=0x7fb4d436f910>, *args=(), **kwargs={})
   78             log.info('Initial checks done. Running the testcase now')
   79             log.info('')
   80             result = function(self, *args, **kwargs)
   81             if self.inputs.upgrade:
   82                 pid = os.getpid()
result = None
function = <function TestSvcRegr.test_svc_in_network_datapath>
self = <scripts.svc_firewall.test_svc_fw.TestSvcRegr.te...in_network_datapath[ci_sanity] id=0x7fb4d436f910>
args = ()
kwargs = {}

 /contrail-test/scripts/svc_firewall/test_svc_fw.py in test_svc_in_network_datapath(self=<scripts.svc_firewall.test_svc_fw.TestSvcRegr.te...in_network_datapath[ci_sanity] id=0x7fb4d436f910>)
   23     @preposttest_wrapper
   24     def test_svc_in_network_datapath(self):
   25         return self.verify_svc_chain(svc_img_name='cirros_in_net', service_mode='in-network',
   26                                      create_svms=True)
   27 
self = <scripts.svc_firewall.test_svc_fw.TestSvcRegr.te...in_network_datapath[ci_sanity] id=0x7fb4d436f910>
self.verify_svc_chain = <bound method VerifySvcChain.verify_svc_chain of...n_network_datapath[ci_sanity] id=0x7fb4d436f910>>
svc_img_name undefined
service_mode undefined
create_svms undefined

 /contrail-test/common/servicechain/verify.py in verify_svc_chain(self=<scripts.svc_firewall.test_svc_fw.TestSvcRegr.te...in_network_datapath[ci_sanity] id=0x7fb4d436f910>, *args=(), **kwargs={'create_svms': True, 'service_mode': 'in-network', 'svc_img_name': 'cirros_in_net'})
  318     def verify_svc_chain(self, *args, **kwargs):
  319         svc_chain_info = kwargs.get('svc_chain_info')
  320         ret_dict = svc_chain_info or self.config_svc_chain(*args, **kwargs)
  321         proto = kwargs.get('proto', 'any')
  322         left_vn_fq_name = ret_dict.get('left_vn_fixture').vn_fq_name
ret_dict undefined
svc_chain_info = None
self = <scripts.svc_firewall.test_svc_fw.TestSvcRegr.te...in_network_datapath[ci_sanity] id=0x7fb4d436f910>
self.config_svc_chain = <bound method ConfigSvcChain.config_svc_chain of...n_network_datapath[ci_sanity] id=0x7fb4d436f910>>
args = ()
kwargs = {'create_svms': True, 'service_mode': 'in-network', 'svc_img_name': 'cirros_in_net'}

 /contrail-test/common/servicechain/config.py in config_svc_chain(self=<scripts.svc_firewall.test_svc_fw.TestSvcRegr.te...in_network_datapath[ci_sanity] id=0x7fb4d436f910>, service_mode='in-network', service_type='firewall', max_inst=1, proto='any', svc_chain_type='parallel', src_ports=[0, 65535], dst_ports=[0, 65535], svc_img_name='cirros_in_net', st_version=2, mgmt_vn_name=None, mgmt_vn_subnets=[], mgmt_vn_fixture=None, left_vn_name=None, left_vn_subnets=[], left_vn_fixture=None, right_vn_name=None, right_vn_subnets=[], right_vn_fixture=None, left_lr_child_vn_fixture=None, right_lr_child_vn_fixture=None, left_vm_name=None, left_vm_fixture=None, right_vm_name=None, right_vm_fixture=None, image_name='cirros', policy_fixture=None, st_fixture=None, si_fixture=None, port_tuples_props=None, static_route=None, svm_fixtures=[], create_svms=True, evpn=False, modify_rules=False, policy_action='pass', hosts=[], **kwargs={})
  440          mgmt_vn_subnets,
  441          mgmt_vn_fixture,
  442          mgmt_vn_fq_name) = self._get_vn_for_config(mgmt_vn_name,
  443                                                     mgmt_vn_subnets,
  444                                                     mgmt_vn_fixture,
mgmt_vn_fq_name undefined
self = <scripts.svc_firewall.test_svc_fw.TestSvcRegr.te...in_network_datapath[ci_sanity] id=0x7fb4d436f910>
self._get_vn_for_config = <bound method ConfigSvcChain._get_vn_for_config ...n_network_datapath[ci_sanity] id=0x7fb4d436f910>>
mgmt_vn_name = None

 /contrail-test/common/servicechain/config.py in _get_vn_for_config(self=<scripts.svc_firewall.test_svc_fw.TestSvcRegr.te...in_network_datapath[ci_sanity] id=0x7fb4d436f910>, vn_name='ctest-mgmt_vn-06711312', vn_subnets=['106.12.108.0/26'], vn_fixture=None, vn_name_prefix='mgmt_vn', **kwargs={})
  355             vn_subnets = vn_subnets or \
  356                                   [get_random_cidr(af=self.inputs.get_af())]
  357             vn_fixture = vn_fixture or self.config_vn(vn_name, vn_subnets, **kwargs)
  358         vn_fq_name = vn_fixture.vn_fq_name
  359         return (vn_name, vn_subnets, vn_fixture, vn_fq_name)
vn_fixture = None
self = <scripts.svc_firewall.test_svc_fw.TestSvcRegr.te...in_network_datapath[ci_sanity] id=0x7fb4d436f910>
self.config_vn = <bound method ConfigSvcChain.config_vn of <scrip...n_network_datapath[ci_sanity] id=0x7fb4d436f910>>
vn_name = 'ctest-mgmt_vn-06711312'
vn_subnets = ['106.12.108.0/26']
kwargs = {}

 /contrail-test/common/servicechain/config.py in config_vn(self=<scripts.svc_firewall.test_svc_fw.TestSvcRegr.te...in_network_datapath[ci_sanity] id=0x7fb4d436f910>, vn_name='ctest-mgmt_vn-06711312', vn_net=['106.12.108.0/26'], **kwargs={})
  137             project_name=self.inputs.project_name, connections=self.connections,
  138             vn_name=vn_name, inputs=self.inputs, subnets=vn_net, **kwargs))
  139         assert vn_fixture.verify_on_setup()
  140         return vn_fixture
  141 
vn_fixture = <vn_test.VNFixture object>
vn_fixture.verify_on_setup = <bound method VNFixture.verify_on_setup of <vn_test.VNFixture object>>

 /contrail-test/fixtures/vn_test.py in verify_on_setup(self=<vn_test.VNFixture object>)
  574     def verify_on_setup(self):
  575         result = True
  576         if not self.verify_vn_in_api_server():
  577             result = result and False
  578             self.logger.error(
self = <vn_test.VNFixture object>
self.verify_vn_in_api_server = <bound method retry.<locals>.deco_retry.<locals>.f_retry of <vn_test.VNFixture object>>

 /contrail-test/tcutils/util.py in f_retry(*args=(<vn_test.VNFixture object>,), **kwargs={})
   82             mtries, mdelay = tries, delay  # make mutable
   83 
   84             result = f(*args, **kwargs)  # first attempt
   85             rv = result
   86             final = False
result undefined
f = <function VNFixture.verify_vn_in_api_server>
args = (<vn_test.VNFixture object>,)
kwargs = {}

 /contrail-test/fixtures/vn_test.py in verify_vn_in_api_server(self=<vn_test.VNFixture object>)
  638 
  639         subnets = list()
  640         for ipam in self.api_s_vn_obj['virtual-network']['network_ipam_refs']:
  641             subnets.extend(ipam['attr']['ipam_subnets'])
  642         for vn_subnet in self.vn_subnets:
ipam undefined
self = <vn_test.VNFixture object>
self.api_s_vn_obj = {'virtual-network': {'uuid': '8857dd63-b2ac-4e94...2/project/9656c20d-04f0-40d9-b9ae-3cdb95830e35'}}
KeyError: 'network_ipam_refs'
    __cause__ = None
    __class__ = <class 'KeyError'>
    __context__ = None
    __delattr__ = <method-wrapper '__delattr__' of KeyError object>
    __dict__ = {}
    __dir__ = <built-in method __dir__ of KeyError object>
    __doc__ = 'Mapping key not found.'
    __eq__ = <method-wrapper '__eq__' of KeyError object>
    __format__ = <built-in method __format__ of KeyError object>
    __ge__ = <method-wrapper '__ge__' of KeyError object>
    __getattribute__ = <method-wrapper '__getattribute__' of KeyError object>
    __gt__ = <method-wrapper '__gt__' of KeyError object>
    __hash__ = <method-wrapper '__hash__' of KeyError object>
    __init__ = <method-wrapper '__init__' of KeyError object>
    __init_subclass__ = <built-in method __init_subclass__ of type object>
    __le__ = <method-wrapper '__le__' of KeyError object>
    __lt__ = <method-wrapper '__lt__' of KeyError object>
    __ne__ = <method-wrapper '__ne__' of KeyError object>
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of KeyError object>
    __reduce_ex__ = <built-in method __reduce_ex__ of KeyError object>
    __repr__ = <method-wrapper '__repr__' of KeyError object>
    __setattr__ = <method-wrapper '__setattr__' of KeyError object>
    __setstate__ = <built-in method __setstate__ of KeyError object>
    __sizeof__ = <built-in method __sizeof__ of KeyError object>
    __str__ = <method-wrapper '__str__' of KeyError object>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __suppress_context__ = False
    __traceback__ = <traceback object>
    args = ('network_ipam_refs',)
    with_traceback = <built-in method with_traceback of KeyError object>

The above is a description of an error in a Python program.  Here is
the original traceback:

Traceback (most recent call last):
  File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper
    result = function(self, *args, **kwargs)
  File "/contrail-test/scripts/svc_firewall/test_svc_fw.py", line 25, in test_svc_in_network_datapath
    return self.verify_svc_chain(svc_img_name='cirros_in_net', service_mode='in-network',
  File "/contrail-test/common/servicechain/verify.py", line 320, in verify_svc_chain
    ret_dict = svc_chain_info or self.config_svc_chain(*args, **kwargs)
  File "/contrail-test/common/servicechain/config.py", line 442, in config_svc_chain
    mgmt_vn_fq_name) = self._get_vn_for_config(mgmt_vn_name,
  File "/contrail-test/common/servicechain/config.py", line 357, in _get_vn_for_config
    vn_fixture = vn_fixture or self.config_vn(vn_name, vn_subnets, **kwargs)
  File "/contrail-test/common/servicechain/config.py", line 139, in config_vn
    assert vn_fixture.verify_on_setup()
  File "/contrail-test/fixtures/vn_test.py", line 576, in verify_on_setup
    if not self.verify_vn_in_api_server():
  File "/contrail-test/tcutils/util.py", line 84, in f_retry
    result = f(*args, **kwargs)  # first attempt
  File "/contrail-test/fixtures/vn_test.py", line 640, in verify_vn_in_api_server
    for ipam in self.api_s_vn_obj['virtual-network']['network_ipam_refs']:
KeyError: 'network_ipam_refs'



2025-08-27 14:16:39,197 - DEBUG - No XMPP flaps were noticed during the test
2025-08-27 14:16:39,197 - INFO - 
2025-08-27 14:16:39,197 - INFO - END TEST : test_svc_in_network_datapath : FAILED[0:00:02]
2025-08-27 14:16:39,197 - INFO - --------------------------------------------------------------------------------
2025-08-27 14:16:39,866 - INFO - Deleted project: ctest-TestSvcRegr-30847890, ID : 9656c20d-04f0-40d9-b9ae-3cdb95830e35