2025-08-25 20:23:36,838 - INFO - Domain Default found not creating
2025-08-25 20:23:37,003 - INFO - Project ctest-TestBasicPolicyIpv4v6-10007468 not found, creating it
2025-08-25 20:23:37,524 - INFO - Created Project:ctest-TestBasicPolicyIpv4v6-10007468, ID : 0b77b05f-cbd6-4839-9483-185d289a34fb 
2025-08-25 20:23:39,051 - INFO - ================================================================================
2025-08-25 20:23:39,051 - INFO - STARTING TEST    : test_basic_policy_allow_deny
2025-08-25 20:23:39,051 - INFO - TEST DESCRIPTION : 
        Create 2 Vns and allow icmp traffic between them and validate with pings
        Update the policy to deny the same traffic
        Check that pings fail
        
2025-08-25 20:23:39,179 - DEBUG - Nothing to compare xmpp stats {'10.0.0.242': {'10.20.0.17': '0'}} with
2025-08-25 20:23:39,179 - INFO - Initial checks done. Running the testcase now
2025-08-25 20:23:39,179 - INFO - 
2025-08-25 20:23:39,762 - DEBUG - Response for create_network : {'network': {'id': 'dbfa7958-6de4-4102-9c2d-e146ec6dc816', 'name': 'ctest-vn-39078640', 'tenant_id': '0b77b05fcbd648399483185d289a34fb', 'project_id': '0b77b05fcbd648399483185d289a34fb', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestBasicPolicyIpv4v6-10007468', 'ctest-vn-39078640'], 'port_security_enabled': True, 'description': ''}}
2025-08-25 20:23:39,902 - ERROR - Neutron Exception while creating subnet for vn with id dbfa7958-6de4-4102-9c2d-e146ec6dc816
Traceback (most recent call last):
  File "/contrail-test/fixtures/quantum_test.py", line 128, in create_subnet
    subnet_rsp = self.obj.create_subnet({'subnet': subnet_req})
  File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 848, in create_subnet
    return self.post(self.subnets_path, body=body)
  File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 359, in post
    headers=headers, params=params)
  File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
    self._handle_fault_response(status_code, replybody, resp)
  File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
    exception_handler_v20(status_code, error_body)
  File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
    request_ids=request_ids)
neutronclient.common.exceptions.InternalServerError: The server has either erred or is incapable of performing the requested operation.<br /><br />



Neutron server returns request_ids: ['req-1949d372-53a1-4e16-8818-49e31dcb08d0']
2025-08-25 20:23:40,015 - ERROR - Neutron Exception while creating subnet for vn with id dbfa7958-6de4-4102-9c2d-e146ec6dc816
Traceback (most recent call last):
  File "/contrail-test/fixtures/quantum_test.py", line 128, in create_subnet
    subnet_rsp = self.obj.create_subnet({'subnet': subnet_req})
  File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 848, in create_subnet
    return self.post(self.subnets_path, body=body)
  File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 359, in post
    headers=headers, params=params)
  File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
    self._handle_fault_response(status_code, replybody, resp)
  File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
    exception_handler_v20(status_code, error_body)
  File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
    request_ids=request_ids)
neutronclient.common.exceptions.InternalServerError: The server has either erred or is incapable of performing the requested operation.<br /><br />



Neutron server returns request_ids: ['req-ad628b9f-71c7-49ba-b9ca-2c73e13ff0ca']
2025-08-25 20:23:40,031 - INFO - Created VN ctest-vn-39078640
2025-08-25 20:23:40,084 - DEBUG - VN ctest-vn-39078640 UUID is dbfa7958-6de4-4102-9c2d-e146ec6dc816
2025-08-25 20:23:40,399 - DEBUG - Response for create_network : {'network': {'id': '4473fb03-64eb-423e-b40a-f05c369c0fa0', 'name': 'ctest-vn-90175757', 'tenant_id': '0b77b05fcbd648399483185d289a34fb', 'project_id': '0b77b05fcbd648399483185d289a34fb', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestBasicPolicyIpv4v6-10007468', 'ctest-vn-90175757'], 'port_security_enabled': True, 'description': ''}}
2025-08-25 20:23:40,533 - ERROR - Neutron Exception while creating subnet for vn with id 4473fb03-64eb-423e-b40a-f05c369c0fa0
Traceback (most recent call last):
  File "/contrail-test/fixtures/quantum_test.py", line 128, in create_subnet
    subnet_rsp = self.obj.create_subnet({'subnet': subnet_req})
  File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 848, in create_subnet
    return self.post(self.subnets_path, body=body)
  File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 359, in post
    headers=headers, params=params)
  File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
    self._handle_fault_response(status_code, replybody, resp)
  File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
    exception_handler_v20(status_code, error_body)
  File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
    request_ids=request_ids)
neutronclient.common.exceptions.InternalServerError: The server has either erred or is incapable of performing the requested operation.<br /><br />



Neutron server returns request_ids: ['req-ae808281-fd1f-4924-84b8-d01529455db1']
2025-08-25 20:23:40,689 - ERROR - Neutron Exception while creating subnet for vn with id 4473fb03-64eb-423e-b40a-f05c369c0fa0
Traceback (most recent call last):
  File "/contrail-test/fixtures/quantum_test.py", line 128, in create_subnet
    subnet_rsp = self.obj.create_subnet({'subnet': subnet_req})
  File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 848, in create_subnet
    return self.post(self.subnets_path, body=body)
  File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 359, in post
    headers=headers, params=params)
  File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
    self._handle_fault_response(status_code, replybody, resp)
  File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
    exception_handler_v20(status_code, error_body)
  File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
    request_ids=request_ids)
neutronclient.common.exceptions.InternalServerError: The server has either erred or is incapable of performing the requested operation.<br /><br />



Neutron server returns request_ids: ['req-ecc0a190-5f6c-449c-bbc9-1cc9f30deeec']
2025-08-25 20:23:40,707 - INFO - Created VN ctest-vn-90175757
2025-08-25 20:23:40,764 - DEBUG - VN ctest-vn-90175757 UUID is 4473fb03-64eb-423e-b40a-f05c369c0fa0
2025-08-25 20:23:40,901 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicPolicyIpv4v6-10007468:ctest-vn-39078640, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicPolicyIpv4v6-10007468:ctest-vn-90175757, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None, rule_sequence = None, rule_uuid = None, direction = <>, protocol = 58, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicPolicyIpv4v6-10007468:ctest-vn-39078640, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicPolicyIpv4v6-10007468:ctest-vn-90175757, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None]
2025-08-25 20:23:41,260 - DEBUG - Response for mapping policy(s) with vn {'network': {'id': 'dbfa7958-6de4-4102-9c2d-e146ec6dc816', 'name': 'ctest-vn-39078640', 'tenant_id': '0b77b05fcbd648399483185d289a34fb', 'project_id': '0b77b05fcbd648399483185d289a34fb', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestBasicPolicyIpv4v6-10007468', 'ctest-vn-39078640'], 'policys': [['default-domain', 'ctest-TestBasicPolicyIpv4v6-10007468', 'ctest-policy-allow-all-48519866']], 'port_security_enabled': True, 'description': ''}}
2025-08-25 20:23:41,511 - DEBUG - Response for mapping policy(s) with vn {'network': {'id': '4473fb03-64eb-423e-b40a-f05c369c0fa0', 'name': 'ctest-vn-90175757', 'tenant_id': '0b77b05fcbd648399483185d289a34fb', 'project_id': '0b77b05fcbd648399483185d289a34fb', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestBasicPolicyIpv4v6-10007468', 'ctest-vn-90175757'], 'policys': [['default-domain', 'ctest-TestBasicPolicyIpv4v6-10007468', 'ctest-policy-allow-all-48519866']], 'port_security_enabled': True, 'description': ''}}
2025-08-25 20:23:41,688 - DEBUG - Requesting: http://10.0.0.242:8082/domains
2025-08-25 20:23:41,992 - DEBUG - Requesting: http://10.0.0.242:8082/domains
2025-08-25 20:23:42,022 - DEBUG - Requesting: http://10.0.0.242:8082/domain/5d5f8548-17f5-4173-99c5-bd5c5a182438
2025-08-25 20:23:42,037 - DEBUG - Requesting: http://10.0.0.242:8082/project/0b77b05f-cbd6-4839-9483-185d289a34fb
2025-08-25 20:23:42,117 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-network/dbfa7958-6de4-4102-9c2d-e146ec6dc816
2025-08-25 20:23:42,482 - INFO - Deleted policy ctest-policy-allow-all-48519866
2025-08-25 20:23:42,482 - INFO - Deleting VN ctest-vn-90175757 
2025-08-25 20:23:42,572 - DEBUG - Response for deleting network ()
2025-08-25 20:23:42,572 - INFO - Deleting VN ctest-vn-39078640 
2025-08-25 20:23:42,655 - DEBUG - Response for deleting network ()
2025-08-25 20:23:42,780 - ERROR - KeyError
Python 3.6.8: /usr/bin/python3
Mon Aug 25 20:23:42 2025

A problem occurred in a Python script.  Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /contrail-test/tcutils/wrappers.py in wrapper(self=<scripts.policy.test_policy.TestBasicPolicyIpv4v...ty,quick_sanity,sanity,suite1] id=0x7f8e13038f60>, *args=(), **kwargs={})
   78             log.info('Initial checks done. Running the testcase now')
   79             log.info('')
   80             result = function(self, *args, **kwargs)
   81             if self.inputs.upgrade:
   82                 pid = os.getpid()
result = None
function = <function TestBasicPolicy.test_basic_policy_allow_deny>
self = <scripts.policy.test_policy.TestBasicPolicyIpv4v...ty,quick_sanity,sanity,suite1] id=0x7f8e13038f60>
args = ()
kwargs = {}

 /contrail-test/scripts/policy/test_policy_basic.py in test_basic_policy_allow_deny(self=<scripts.policy.test_policy.TestBasicPolicyIpv4v...ty,quick_sanity,sanity,suite1] id=0x7f8e13038f60>)
  109         policy_fixture = self.setup_policy_between_vns(vn1_fixture,
  110             vn2_fixture, rules)
  111         assert vn1_fixture.verify_on_setup()
  112         assert vn2_fixture.verify_on_setup()
  113 
vn1_fixture = <vn_test.VNFixture object>
vn1_fixture.verify_on_setup = <bound method VNFixture.verify_on_setup of <vn_test.VNFixture object>>

 /contrail-test/fixtures/vn_test.py in verify_on_setup(self=<vn_test.VNFixture object>)
  574     def verify_on_setup(self):
  575         result = True
  576         if not self.verify_vn_in_api_server():
  577             result = result and False
  578             self.logger.error(
self = <vn_test.VNFixture object>
self.verify_vn_in_api_server = <bound method retry.<locals>.deco_retry.<locals>.f_retry of <vn_test.VNFixture object>>

 /contrail-test/tcutils/util.py in f_retry(*args=(<vn_test.VNFixture object>,), **kwargs={})
   82             mtries, mdelay = tries, delay  # make mutable
   83 
   84             result = f(*args, **kwargs)  # first attempt
   85             rv = result
   86             final = False
result undefined
f = <function VNFixture.verify_vn_in_api_server>
args = (<vn_test.VNFixture object>,)
kwargs = {}

 /contrail-test/fixtures/vn_test.py in verify_vn_in_api_server(self=<vn_test.VNFixture object>)
  638 
  639         subnets = list()
  640         for ipam in self.api_s_vn_obj['virtual-network']['network_ipam_refs']:
  641             subnets.extend(ipam['attr']['ipam_subnets'])
  642         for vn_subnet in self.vn_subnets:
ipam undefined
self = <vn_test.VNFixture object>
self.api_s_vn_obj = {'virtual-network': {'uuid': 'dbfa7958-6de4-4102...2/project/0b77b05f-cbd6-4839-9483-185d289a34fb'}}
KeyError: 'network_ipam_refs'
    __cause__ = None
    __class__ = <class 'KeyError'>
    __context__ = None
    __delattr__ = <method-wrapper '__delattr__' of KeyError object>
    __dict__ = {}
    __dir__ = <built-in method __dir__ of KeyError object>
    __doc__ = 'Mapping key not found.'
    __eq__ = <method-wrapper '__eq__' of KeyError object>
    __format__ = <built-in method __format__ of KeyError object>
    __ge__ = <method-wrapper '__ge__' of KeyError object>
    __getattribute__ = <method-wrapper '__getattribute__' of KeyError object>
    __gt__ = <method-wrapper '__gt__' of KeyError object>
    __hash__ = <method-wrapper '__hash__' of KeyError object>
    __init__ = <method-wrapper '__init__' of KeyError object>
    __init_subclass__ = <built-in method __init_subclass__ of type object>
    __le__ = <method-wrapper '__le__' of KeyError object>
    __lt__ = <method-wrapper '__lt__' of KeyError object>
    __ne__ = <method-wrapper '__ne__' of KeyError object>
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of KeyError object>
    __reduce_ex__ = <built-in method __reduce_ex__ of KeyError object>
    __repr__ = <method-wrapper '__repr__' of KeyError object>
    __setattr__ = <method-wrapper '__setattr__' of KeyError object>
    __setstate__ = <built-in method __setstate__ of KeyError object>
    __sizeof__ = <built-in method __sizeof__ of KeyError object>
    __str__ = <method-wrapper '__str__' of KeyError object>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __suppress_context__ = False
    __traceback__ = <traceback object>
    args = ('network_ipam_refs',)
    with_traceback = <built-in method with_traceback of KeyError object>

The above is a description of an error in a Python program.  Here is
the original traceback:

Traceback (most recent call last):
  File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper
    result = function(self, *args, **kwargs)
  File "/contrail-test/scripts/policy/test_policy_basic.py", line 111, in test_basic_policy_allow_deny
    assert vn1_fixture.verify_on_setup()
  File "/contrail-test/fixtures/vn_test.py", line 576, in verify_on_setup
    if not self.verify_vn_in_api_server():
  File "/contrail-test/tcutils/util.py", line 84, in f_retry
    result = f(*args, **kwargs)  # first attempt
  File "/contrail-test/fixtures/vn_test.py", line 640, in verify_vn_in_api_server
    for ipam in self.api_s_vn_obj['virtual-network']['network_ipam_refs']:
KeyError: 'network_ipam_refs'



2025-08-25 20:23:42,787 - DEBUG - No XMPP flaps were noticed during the test
2025-08-25 20:23:42,787 - INFO - 
2025-08-25 20:23:42,787 - INFO - END TEST : test_basic_policy_allow_deny : FAILED[0:00:03]
2025-08-25 20:23:42,787 - INFO - --------------------------------------------------------------------------------
2025-08-25 20:23:43,467 - INFO - Deleted project: ctest-TestBasicPolicyIpv4v6-10007468, ID : 0b77b05f-cbd6-4839-9483-185d289a34fb