2025-10-09 10:29:17,783 - DEBUG - [10.0.0.33]: Running cmd : hostname 2025-10-09 10:29:18,037 - DEBUG - Output : cn-jenkins-deploy-platform-juju-os-587-1 2025-10-09 10:29:18,037 - DEBUG - [10.0.0.33]: Running cmd : hostname -f 2025-10-09 10:29:18,137 - DEBUG - Output : cn-jenkins-deploy-platform-juju-os-587-1 2025-10-09 10:29:18,137 - DEBUG - [10.0.0.33]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-10-09 10:29:18,260 - DEBUG - Output : NAMES contrail_test_ObuSs2dL7 vrouter-vrouter-agent-1 vrouter-nodemgr-1 webui-job-1 webui-web-1 control-named-1 control-control-1 control-nodemgr-1 control-dns-1 config_database-zookeeper-1 config_database-rabbitmq-1 config_database-nodemgr-1 config_database-cassandra-1 config_api-devicemgr-1 config_api-dnsmasq-1 config_api-schema-1 config_api-api-1 config_api-nodemgr-1 config_api-svcmonitor-1 analytics_database-cassandra-1 analytics_database-nodemgr-1 analytics_database-query-engine-1 redis-redis-1 redis-stunnel-1 analytics_snmp-nodemgr-1 analytics_snmp-snmp-collector-1 analytics_snmp-topology-1 analytics_alarm-alarm-gen-1 analytics_alarm-nodemgr-1 analytics_alarm-kafka-1 analytics-nodemgr-1 analytics-api-1 analytics-collector-1 2025-10-09 10:29:18,260 - DEBUG - [10.0.0.33]: Running cmd : getent hosts 10.0.0.33 | head -n 1 | awk '{print $2}' 2025-10-09 10:29:18,353 - DEBUG - Output : cn-jenkins-deploy-platform-juju-os-587-1 2025-10-09 10:29:18,353 - DEBUG - [10.0.0.33]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-10-09 10:29:18,445 - DEBUG - Output : 127.0.0.1/8 252.33.0.1/8 10.174.214.1/24 172.17.0.1/16 10.0.0.33/24 2025-10-09 10:29:18,445 - DEBUG - [10.0.0.33]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2025-10-09 10:29:18,536 - DEBUG - Output : 10.0.0.33/24 2025-10-09 10:29:29,112 - INFO - Using existing project ['admin_domain', 'admin'](4ffa32a1-1c41-42f7-a8b6-f5d80e4454ba) 2025-10-09 10:29:29,257 - INFO - ================================================================================ 2025-10-09 10:29:29,257 - INFO - STARTING TEST : test_contrail_status 2025-10-09 10:29:29,257 - INFO - TEST DESCRIPTION : Test to verify that all services are running and active 2025-10-09 10:29:29,462 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:29:29,512 - DEBUG - Nothing to compare xmpp stats {'10.0.0.33': {'10.0.0.33': '0'}} with 2025-10-09 10:29:29,512 - INFO - Initial checks done. Running the testcase now 2025-10-09 10:29:29,512 - INFO - 2025-10-09 10:29:29,513 - DEBUG - [10.0.0.33]: Running cmd : docker ps -f status=running --format {{.Names}} 2>/dev/null 2025-10-09 10:29:29,601 - DEBUG - Output : nifty_borg contrail_test_ObuSs2dL7 vrouter-vrouter-agent-1 vrouter-provisioner-1 vrouter-nodemgr-1 webui-job-1 webui-web-1 control-named-1 control-control-1 control-provisioner-1 control-nodemgr-1 control-dns-1 config_database-zookeeper-1 config_database-rabbitmq-1 config_database-provisioner-1 config_database-nodemgr-1 config_database-cassandra-1 config_api-devicemgr-1 config_api-dnsmasq-1 config_api-schema-1 config_api-api-1 config_api-nodemgr-1 config_api-provisioner-1 config_api-svcmonitor-1 analytics_database-cassandra-1 analytics_database-nodemgr-1 analytics_database-provisioner-1 analytics_database-query-engine-1 redis-redis-1 redis-stunnel-1 analytics_snmp-nodemgr-1 analytics_snmp-snmp-collector-1 analytics_snmp-topology-1 analytics_snmp-provisioner-1 analytics_alarm-alarm-gen-1 analytics_alarm-nodemgr-1 analytics_alarm-kafka-1 analytics_alarm-provisioner-1 analytics-nodemgr-1 analytics-provisioner-1 analytics-api-1 analytics-collector-1 2025-10-09 10:29:29,601 - INFO - 10.0.0.33 2025-10-09 10:29:29,601 - INFO - config 2025-10-09 10:29:29,660 - INFO - config-nodemgr:active 2025-10-09 10:29:29,719 - INFO - api-server:active 2025-10-09 10:29:29,779 - INFO - schema:active 2025-10-09 10:29:29,839 - INFO - svc-monitor:active 2025-10-09 10:29:29,895 - INFO - device-manager:active 2025-10-09 10:29:29,895 - INFO - config-database 2025-10-09 10:29:29,895 - INFO - config-cassandra:active 2025-10-09 10:29:29,895 - INFO - config-zookeeper:active 2025-10-09 10:29:29,895 - INFO - config-rabbitmq:active 2025-10-09 10:29:29,896 - INFO - vrouter 2025-10-09 10:29:29,947 - INFO - vrouter-nodemgr:active 2025-10-09 10:29:29,999 - INFO - agent:active 2025-10-09 10:29:29,999 - INFO - control 2025-10-09 10:29:30,055 - INFO - control-nodemgr:active 2025-10-09 10:29:30,114 - INFO - control:active 2025-10-09 10:29:30,114 - INFO - named:active 2025-10-09 10:29:30,167 - INFO - dns:active 2025-10-09 10:29:30,167 - INFO - analytics 2025-10-09 10:29:30,223 - INFO - analytics-nodemgr:active 2025-10-09 10:29:30,279 - INFO - analytics-api:active 2025-10-09 10:29:30,335 - INFO - collector:active 2025-10-09 10:29:30,336 - INFO - analytics-database 2025-10-09 10:29:30,336 - INFO - analytics-cassandra:active 2025-10-09 10:29:30,392 - INFO - analyticsdb-nodemgr:active 2025-10-09 10:29:30,455 - INFO - query-engine:active 2025-10-09 10:29:30,455 - INFO - webui 2025-10-09 10:29:30,455 - INFO - webui:active 2025-10-09 10:29:30,456 - INFO - webui-middleware:active 2025-10-09 10:29:30,456 - INFO - redis:active 2025-10-09 10:29:30,456 - INFO - analytics_snmp 2025-10-09 10:29:30,515 - INFO - snmp-collector:active 2025-10-09 10:29:30,515 - INFO - snmp-topology:active 2025-10-09 10:29:30,515 - INFO - analytics_alarm 2025-10-09 10:29:30,515 - INFO - alarmgen:active 2025-10-09 10:29:30,515 - INFO - All the contrail services are up on all nodes 2025-10-09 10:29:30,515 - INFO - contrail-status passed 2025-10-09 10:29:30,719 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:29:30,772 - DEBUG - No XMPP flaps were noticed during the test 2025-10-09 10:29:30,772 - INFO - END TEST : test_contrail_status : PASSED[0:00:01] 2025-10-09 10:29:30,772 - INFO - -------------------------------------------------------------------------------- 2025-10-09 10:29:40,426 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:29:44,543 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-AnalyticsBasicTestSanity-58014060:ctest-vn22-96956903:ctest-vn22-96956903 2025-10-09 10:29:44,591 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-AnalyticsBasicTestSanity-58014060:ctest-vn22-96956903:ctest-vn22-96956903 2025-10-09 10:29:44,639 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-AnalyticsBasicTestSanity-58014060:ctest-vn22-96956903:ctest-vn22-96956903 2025-10-09 10:30:07,152 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VnListReq?name=admin_domain:ctest-AnalyticsBasicTestSanity-58014060:ctest-vn22-96956903 2025-10-09 10:30:07,203 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-AnalyticsBasicTestSanity-58014060:ctest-vn22-96956903:ctest-vn22-96956903 2025-10-09 10:30:07,255 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-AnalyticsBasicTestSanity-58014060:ctest-vn22-96956903:ctest-vn22-96956903 2025-10-09 10:30:07,628 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:30:09,070 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:30:09,121 - DEBUG - [10.0.0.33]: Running cmd : docker exec --privileged -it config_api-api-1 /bin/bash -c 'db-manage check' 2025-10-09 10:30:11,862 - DEBUG - Output : 2025-10-09 10:30:11,701 INFO: (v1.41) Checker check_zk_mode_and_node_count: Success 2025-10-09 10:30:11,751 ERROR: (v1.41) Checker check_cassandra_keyspace_replication: Exception, NoHostAvailable Python 3.9.21: /usr/bin/python3 Thu Oct 9 10:30:11 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py in wrapper(*args=(,), **kwargs={}) 1535 self = args[0] 1536 try: 1537 errors = func(*args, **kwargs) 1538 if not errors: 1539 self._logger.info('(v%s) Checker %s: Success', __version__, errors undefined func = args = (,) kwargs = {} /usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py in check_cassandra_keyspace_replication(self=) 1635 raise CassandraConnectionError("error, {}: {}".format( 1636 error, utils.detailed_traceback())) 1637 temp_cluster.connect() 1638 for ks_name, _ in self._db_info: 1639 if self._api_args.cluster_id: temp_cluster = temp_cluster.connect = > /cassandra/cluster.py in cassandra.cluster.Cluster.connect() /cassandra/cluster.py in cassandra.cluster.Cluster.connect() /cassandra/cluster.py in cassandra.cluster.Cluster.connect() /cassandra/cluster.py in cassandra.cluster.ControlConnection.connect() /cassandra/cluster.py in cassandra.cluster.ControlConnection._reconnect_internal() NoHostAvailable: ('Unable to connect to any servers', {'10.0.0.33:9041': ConnectionResetError(104, 'Connection reset by peer')}) __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {'errors': {'10.0.0.33:9041': ConnectionResetError(104, 'Connection reset by peer')}} __dir__ = __doc__ = '\n Raised when an operation is attempted but a...ct, closed, or resulted in errors when used.\n ' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __module__ = 'cassandra.cluster' __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = __weakref__ = None args = ('Unable to connect to any servers', {'10.0.0.33:9041': ConnectionResetError(104, 'Connection reset by peer')}) errors = {'10.0.0.33:9041': ConnectionResetError(104, 'Connection reset by peer')} with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py", line 1537, in wrapper errors = func(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py", line 1637, in check_cassandra_keyspace_replication temp_cluster.connect() File "cassandra/cluster.py", line 1717, in cassandra.cluster.Cluster.connect File "cassandra/cluster.py", line 1753, in cassandra.cluster.Cluster.connect File "cassandra/cluster.py", line 1740, in cassandra.cluster.Cluster.connect File "cassandra/cluster.py", line 3543, in cassandra.cluster.ControlConnection.connect File "cassandra/cluster.py", line 3588, in cassandra.cluster.ControlConnection._reconnect_internal cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers', {'10.0.0.33:9041': ConnectionResetError(104, 'Connection reset by peer')}) Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py", line 1537, in wrapper errors = func(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py", line 1637, in check_cassandra_keyspace_replication temp_cluster.connect() File "cassandra/cluster.py", line 1717, in cassandra.cluster.Cluster.connect File "cassandra/cluster.py", line 1753, in cassandra.cluster.Cluster.connect File "cassandra/cluster.py", line 1740, in cassandra.cluster.Cluster.connect File "cassandra/cluster.py", line 3543, in cassandra.cluster.ControlConnection.connect File "cassandra/cluster.py", line 3588, in cassandra.cluster.ControlConnection._reconnect_internal cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers', {'10.0.0.33:9041': ConnectionResetError(104, 'Connection reset by peer')}) NoHostAvailable Python 3.9.21: /usr/bin/python3 Thu Oct 9 10:30:11 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /usr/local/bin/db-manage in 3 from vnc_cfg_api_server.db_manage import main 4 if __name__ == '__main__': 5 if sys.argv[0].endswith('.exe'): 6 sys.argv[0] = sys.argv[0][:-4] 7 sys.exit(main()) sys = sys.exit = main = /usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py in main() 3451 verb = args.operation 3452 if 'db_%s' % (verb) in globals(): 3453 return globals()['db_%s' % (verb)](args, api_args) 3454 3455 if getattr(DatabaseChecker, verb, None): builtinglobals = verb = 'check' args = Namespace(operation='check', api_conf='/etc/cont...1024, log_file='/var/log/contrail/db_manage.log') api_args = Namespace(conf_file=['/etc/contrail/contrail-api...esh_base.SandeshConfig object at 0x7fa1fe9a00d0>) /usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py in db_check(args=Namespace(operation='check', api_conf='/etc/cont...1024, log_file='/var/log/contrail/db_manage.log'), api_args=Namespace(conf_file=['/etc/contrail/contrail-api...esh_base.SandeshConfig object at 0x7fa1fe9a00d0>)) 3358 # Mode and node count check across all nodes 3359 db_checker.check_zk_mode_and_node_count() 3360 db_checker.check_cassandra_keyspace_replication() 3361 # Obj UUID cassandra inconsistencies 3362 db_checker.check_obj_mandatory_fields() db_checker = db_checker.check_cassandra_keyspace_replication = > /usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py in wrapper(*args=(,), **kwargs={}) 1535 self = args[0] 1536 try: 1537 errors = func(*args, **kwargs) 1538 if not errors: 1539 self._logger.info('(v%s) Checker %s: Success', __version__, errors undefined func = args = (,) kwargs = {} /usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py in check_cassandra_keyspace_replication(self=) 1635 raise CassandraConnectionError("error, {}: {}".format( 1636 error, utils.detailed_traceback())) 1637 temp_cluster.connect() 1638 for ks_name, _ in self._db_info: 1639 if self._api_args.cluster_id: temp_cluster = temp_cluster.connect = > /cassandra/cluster.py in cassandra.cluster.Cluster.connect() /cassandra/cluster.py in cassandra.cluster.Cluster.connect() /cassandra/cluster.py in cassandra.cluster.Cluster.connect() /cassandra/cluster.py in cassandra.cluster.ControlConnection.connect() /cassandra/cluster.py in cassandra.cluster.ControlConnection._reconnect_internal() NoHostAvailable: ('Unable to connect to any servers', {'10.0.0.33:9041': ConnectionResetError(104, 'Connection reset by peer')}) __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {'errors': {'10.0.0.33:9041': ConnectionResetError(104, 'Connection reset by peer')}} __dir__ = __doc__ = '\n Raised when an operation is attempted but a...ct, closed, or resulted in errors when used.\n ' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __module__ = 'cassandra.cluster' __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = __weakref__ = None args = ('Unable to connect to any servers', {'10.0.0.33:9041': ConnectionResetError(104, 'Connection reset by peer')}) errors = {'10.0.0.33:9041': ConnectionResetError(104, 'Connection reset by peer')} with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/usr/local/bin/db-manage", line 7, in sys.exit(main()) File "/usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py", line 3453, in main return globals()['db_%s' % (verb)](args, api_args) File "/usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py", line 3360, in db_check db_checker.check_cassandra_keyspace_replication() File "/usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py", line 1537, in wrapper errors = func(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py", line 1637, in check_cassandra_keyspace_replication temp_cluster.connect() File "cassandra/cluster.py", line 1717, in cassandra.cluster.Cluster.connect File "cassandra/cluster.py", line 1753, in cassandra.cluster.Cluster.connect File "cassandra/cluster.py", line 1740, in cassandra.cluster.Cluster.connect File "cassandra/cluster.py", line 3543, in cassandra.cluster.ControlConnection.connect File "cassandra/cluster.py", line 3588, in cassandra.cluster.ControlConnection._reconnect_internal cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers', {'10.0.0.33:9041': ConnectionResetError(104, 'Connection reset by peer')}) 2025-10-09 10:30:12,106 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:30:12,439 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:30:12,723 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:30:12,979 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:30:13,251 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:30:16,527 - INFO - admin user already present 2025-10-09 10:30:17,655 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:30:21,151 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-26969105:ctest-fvn-30453982:ctest-fvn-30453982 2025-10-09 10:30:21,203 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-26969105:ctest-fvn-30453982:ctest-fvn-30453982 2025-10-09 10:30:21,252 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-26969105:ctest-fvn-30453982:ctest-fvn-30453982 2025-10-09 10:30:23,487 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-26969105:ctest-vn1-01719722:ctest-vn1-01719722 2025-10-09 10:30:23,535 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-26969105:ctest-vn1-01719722:ctest-vn1-01719722 2025-10-09 10:30:23,591 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-26969105:ctest-vn1-01719722:ctest-vn1-01719722 2025-10-09 10:30:34,617 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e4c9c6aa-922b-478d-8009-dd56064297d2 2025-10-09 10:30:34,668 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-26969105:ctest-vn1-01719722:ctest-vn1-01719722 2025-10-09 10:30:34,720 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e4c9c6aa-922b-478d-8009-dd56064297d2 2025-10-09 10:30:46,480 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:4904a174-c0bb-4c83-928d-8353a064dcf1 2025-10-09 10:30:46,533 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-26969105:ctest-fvn-30453982:ctest-fvn-30453982 2025-10-09 10:30:46,583 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:4904a174-c0bb-4c83-928d-8353a064dcf1 2025-10-09 10:30:48,352 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e4c9c6aa-922b-478d-8009-dd56064297d2 2025-10-09 10:30:48,404 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VnListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-26969105:ctest-fvn-30453982 2025-10-09 10:30:48,456 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-26969105:ctest-fvn-30453982:ctest-fvn-30453982 2025-10-09 10:30:48,512 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_Inet4UcRouteReq?x=3 2025-10-09 10:30:54,968 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VnListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-26969105:ctest-vn1-01719722 2025-10-09 10:30:55,023 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-26969105:ctest-vn1-01719722:ctest-vn1-01719722 2025-10-09 10:30:55,076 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-26969105:ctest-vn1-01719722:ctest-vn1-01719722 2025-10-09 10:30:55,644 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VnListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-26969105:ctest-fvn-30453982 2025-10-09 10:30:55,695 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-26969105:ctest-fvn-30453982:ctest-fvn-30453982 2025-10-09 10:30:55,747 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-26969105:ctest-fvn-30453982:ctest-fvn-30453982 2025-10-09 10:30:56,221 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:30:59,961 - INFO - ctest-TestHeat-96061239 user already added to project 2025-10-09 10:31:00,299 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:31:01,221 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:31:09,437 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:31:17,625 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:31:21,709 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:31:24,738 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:31:25,121 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:31:34,765 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:3d5b598a-4ec8-41d3-8bca-3ef94ef08beb 2025-10-09 10:31:34,816 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestPorts-03109103:ctest-vn1-36116190:ctest-vn1-36116190 2025-10-09 10:31:34,868 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:3d5b598a-4ec8-41d3-8bca-3ef94ef08beb 2025-10-09 10:31:43,208 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:9c228783-d79e-4249-888c-bd71515058a4 2025-10-09 10:31:43,260 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestPorts-03109103:ctest-vn1-36116190:ctest-vn1-36116190 2025-10-09 10:31:43,312 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:9c228783-d79e-4249-888c-bd71515058a4 2025-10-09 10:31:50,297 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:31:54,329 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:32:06,249 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:7fa1a189-a1b6-4d7a-81f3-15dcae64acf7 2025-10-09 10:32:06,300 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestRouters-31676881:ctest-vn1-79796669:ctest-vn1-79796669 2025-10-09 10:32:06,352 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:7fa1a189-a1b6-4d7a-81f3-15dcae64acf7 2025-10-09 10:32:12,716 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:68e46482-a99f-479d-b30a-34c25755f3c4 2025-10-09 10:32:12,768 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestRouters-31676881:ctest-vn2-05925071:ctest-vn2-05925071 2025-10-09 10:32:12,820 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:68e46482-a99f-479d-b30a-34c25755f3c4 2025-10-09 10:32:31,922 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:32:35,891 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:32:48,676 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:60790fb6-c7d3-4791-ae11-9e13d36ef157 2025-10-09 10:32:48,728 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestRoutersBasic-03648398:ctest-vn-14960974:ctest-vn-14960974 2025-10-09 10:32:48,776 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:60790fb6-c7d3-4791-ae11-9e13d36ef157 2025-10-09 10:32:55,144 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:a9444cb6-f0da-4631-b76e-9d407b2139e6 2025-10-09 10:32:55,196 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestRoutersBasic-03648398:ctest-ext_vn-34155382:ctest-ext_vn-34155382 2025-10-09 10:32:55,247 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:a9444cb6-f0da-4631-b76e-9d407b2139e6 2025-10-09 10:33:04,070 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:33:08,093 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:33:14,200 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-39235319:ctest-vn-54778176:ctest-vn-54778176 2025-10-09 10:33:14,256 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-39235319:ctest-vn-54778176:ctest-vn-54778176 2025-10-09 10:33:14,307 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-39235319:ctest-vn-54778176:ctest-vn-54778176 2025-10-09 10:33:15,575 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-39235319:ctest-vn-57789610:ctest-vn-57789610 2025-10-09 10:33:15,623 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-39235319:ctest-vn-57789610:ctest-vn-57789610 2025-10-09 10:33:15,671 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-39235319:ctest-vn-57789610:ctest-vn-57789610 2025-10-09 10:33:16,732 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VnListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-39235319:ctest-vn-57789610 2025-10-09 10:33:16,779 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-39235319:ctest-vn-57789610:ctest-vn-57789610 2025-10-09 10:33:16,828 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-39235319:ctest-vn-57789610:ctest-vn-57789610 2025-10-09 10:33:17,396 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VnListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-39235319:ctest-vn-54778176 2025-10-09 10:33:17,447 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-39235319:ctest-vn-54778176:ctest-vn-54778176 2025-10-09 10:33:17,495 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-39235319:ctest-vn-54778176:ctest-vn-54778176 2025-10-09 10:33:17,986 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:33:22,065 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:33:22,458 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:33:26,940 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyNegativeIpv4v6-52261713:ctest-vn4-00830792:ctest-vn4-00830792 2025-10-09 10:33:26,991 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyNegativeIpv4v6-52261713:ctest-vn4-00830792:ctest-vn4-00830792 2025-10-09 10:33:27,039 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyNegativeIpv4v6-52261713:ctest-vn4-00830792:ctest-vn4-00830792 2025-10-09 10:33:27,768 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VnListReq?name=admin_domain:ctest-TestBasicPolicyNegativeIpv4v6-52261713:ctest-vn4-00830792 2025-10-09 10:33:27,819 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyNegativeIpv4v6-52261713:ctest-vn4-00830792:ctest-vn4-00830792 2025-10-09 10:33:27,872 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyNegativeIpv4v6-52261713:ctest-vn4-00830792:ctest-vn4-00830792 2025-10-09 10:33:28,557 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:33:28,941 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:33:32,812 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:33:38,000 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-64599488:ctest-vn-10803880:ctest-vn-10803880 2025-10-09 10:33:38,052 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-64599488:ctest-vn-10803880:ctest-vn-10803880 2025-10-09 10:33:38,103 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-64599488:ctest-vn-10803880:ctest-vn-10803880 2025-10-09 10:33:39,375 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-64599488:ctest-vn-92527430:ctest-vn-92527430 2025-10-09 10:33:39,427 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-64599488:ctest-vn-92527430:ctest-vn-92527430 2025-10-09 10:33:39,479 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-64599488:ctest-vn-92527430:ctest-vn-92527430 2025-10-09 10:33:47,721 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:41aecbeb-0493-4ddd-854e-26c2f0e39f95 2025-10-09 10:33:47,776 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-64599488:ctest-vn-10803880:ctest-vn-10803880 2025-10-09 10:33:47,828 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:41aecbeb-0493-4ddd-854e-26c2f0e39f95 2025-10-09 10:33:54,197 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:30327621-9ad5-41da-97aa-6681539a1c16 2025-10-09 10:33:54,248 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-64599488:ctest-vn-92527430:ctest-vn-92527430 2025-10-09 10:33:54,300 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:30327621-9ad5-41da-97aa-6681539a1c16 2025-10-09 10:34:12,356 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VnListReq?name=admin_domain:ctest-TestBasicPolicy-64599488:ctest-vn-92527430 2025-10-09 10:34:12,408 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-64599488:ctest-vn-92527430:ctest-vn-92527430 2025-10-09 10:34:12,460 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-64599488:ctest-vn-92527430:ctest-vn-92527430 2025-10-09 10:34:13,068 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VnListReq?name=admin_domain:ctest-TestBasicPolicy-64599488:ctest-vn-10803880 2025-10-09 10:34:13,119 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-64599488:ctest-vn-10803880:ctest-vn-10803880 2025-10-09 10:34:13,171 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-64599488:ctest-vn-10803880:ctest-vn-10803880 2025-10-09 10:34:13,661 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:34:17,849 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:34:29,346 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:bfbeb234-3635-40d1-a564-9b4f6aaefadb 2025-10-09 10:34:29,396 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestRP-92060887:ctest-bgpaas_vn-76198126:ctest-bgpaas_vn-76198126 2025-10-09 10:34:29,448 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:bfbeb234-3635-40d1-a564-9b4f6aaefadb 2025-10-09 10:34:41,621 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:bd431fb4-5d40-4e3c-93e9-cca5948e2ea8 2025-10-09 10:34:41,677 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestRP-92060887:ctest-bgpaas_vn-27801384:ctest-bgpaas_vn-27801384 2025-10-09 10:34:41,729 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:bd431fb4-5d40-4e3c-93e9-cca5948e2ea8 2025-10-09 10:34:43,679 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:34:50,776 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:34:54,828 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:34:58,263 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-mgmt_vn-25465840:ctest-mgmt_vn-25465840 2025-10-09 10:34:58,312 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-mgmt_vn-25465840:ctest-mgmt_vn-25465840 2025-10-09 10:34:58,367 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-mgmt_vn-25465840:ctest-mgmt_vn-25465840 2025-10-09 10:35:00,523 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-left_vn-76055038:ctest-left_vn-76055038 2025-10-09 10:35:00,571 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-left_vn-76055038:ctest-left_vn-76055038 2025-10-09 10:35:00,619 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-left_vn-76055038:ctest-left_vn-76055038 2025-10-09 10:35:02,740 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-right_vn-60750500:ctest-right_vn-60750500 2025-10-09 10:35:02,787 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-right_vn-60750500:ctest-right_vn-60750500 2025-10-09 10:35:02,835 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-right_vn-60750500:ctest-right_vn-60750500 2025-10-09 10:35:20,152 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:02ff4430-55d8-486a-81c3-2255d7fa2e68 2025-10-09 10:35:20,205 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-mgmt_vn-25465840:ctest-mgmt_vn-25465840 2025-10-09 10:35:20,255 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:52998dbc-f95f-4be2-9557-fd4d6b6c5f09 2025-10-09 10:35:20,308 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-left_vn-76055038:ctest-left_vn-76055038 2025-10-09 10:35:20,361 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:2f0462fe-9ffa-4e8c-bc90-fd50899da403 2025-10-09 10:35:20,412 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-right_vn-60750500:ctest-right_vn-60750500 2025-10-09 10:35:20,464 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:02ff4430-55d8-486a-81c3-2255d7fa2e68 2025-10-09 10:35:20,516 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:2f0462fe-9ffa-4e8c-bc90-fd50899da403 2025-10-09 10:35:20,568 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:52998dbc-f95f-4be2-9557-fd4d6b6c5f09 2025-10-09 10:35:57,668 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:1ee9c2c2-1956-4f74-ad9d-042645c3d9e0 2025-10-09 10:35:57,720 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-left_vn-76055038:ctest-left_vn-76055038 2025-10-09 10:35:57,772 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:1ee9c2c2-1956-4f74-ad9d-042645c3d9e0 2025-10-09 10:35:59,953 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:8d945512-cbce-4554-9fc1-633124db5809 2025-10-09 10:36:00,004 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-right_vn-60750500:ctest-right_vn-60750500 2025-10-09 10:36:00,055 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:8d945512-cbce-4554-9fc1-633124db5809 2025-10-09 10:36:06,565 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:02ff4430-55d8-486a-81c3-2255d7fa2e68 2025-10-09 10:36:06,616 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:2f0462fe-9ffa-4e8c-bc90-fd50899da403 2025-10-09 10:36:06,664 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:52998dbc-f95f-4be2-9557-fd4d6b6c5f09 2025-10-09 10:36:07,975 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VnListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-left_vn-76055038 2025-10-09 10:36:08,035 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:52998dbc-f95f-4be2-9557-fd4d6b6c5f09 2025-10-09 10:36:08,088 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-09 10:36:08,100 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-left_vn-76055038:ctest-left_vn-76055038 2025-10-09 10:36:08,307 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_Inet4UcRouteReq?x=2 2025-10-09 10:36:08,515 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_Layer2RouteReq?x=2 2025-10-09 10:36:08,574 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VnListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-left_vn-76055038 2025-10-09 10:36:08,623 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-left_vn-76055038:ctest-left_vn-76055038 2025-10-09 10:36:08,831 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_Inet4UcRouteReq?x=2 2025-10-09 10:36:09,043 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_Layer2RouteReq?x=2 2025-10-09 10:36:09,058 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VnListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-mgmt_vn-25465840 2025-10-09 10:36:09,107 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:02ff4430-55d8-486a-81c3-2255d7fa2e68 2025-10-09 10:36:09,156 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-09 10:36:09,167 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-mgmt_vn-25465840:ctest-mgmt_vn-25465840 2025-10-09 10:36:09,371 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_Inet4UcRouteReq?x=4 2025-10-09 10:36:09,583 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_Layer2RouteReq?x=4 2025-10-09 10:36:09,598 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VnListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-mgmt_vn-25465840 2025-10-09 10:36:09,652 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-mgmt_vn-25465840:ctest-mgmt_vn-25465840 2025-10-09 10:36:09,855 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_Inet4UcRouteReq?x=4 2025-10-09 10:36:10,063 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_Layer2RouteReq?x=4 2025-10-09 10:36:10,077 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VnListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-right_vn-60750500 2025-10-09 10:36:10,127 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:2f0462fe-9ffa-4e8c-bc90-fd50899da403 2025-10-09 10:36:10,180 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-09 10:36:10,195 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-right_vn-60750500:ctest-right_vn-60750500 2025-10-09 10:36:10,403 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_Inet4UcRouteReq?x=3 2025-10-09 10:36:10,623 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_Layer2RouteReq?x=3 2025-10-09 10:36:10,637 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VnListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-right_vn-60750500 2025-10-09 10:36:10,691 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-right_vn-60750500:ctest-right_vn-60750500 2025-10-09 10:36:10,895 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_Inet4UcRouteReq?x=3 2025-10-09 10:36:11,103 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_Layer2RouteReq?x=3 2025-10-09 10:36:11,815 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name= 2025-10-09 10:36:11,868 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_Inet4UcRouteReq?x=5 2025-10-09 10:36:11,918 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:52998dbc-f95f-4be2-9557-fd4d6b6c5f09 2025-10-09 10:36:12,075 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name= 2025-10-09 10:36:12,124 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_Inet4UcRouteReq?x=6 2025-10-09 10:36:12,176 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:2f0462fe-9ffa-4e8c-bc90-fd50899da403 2025-10-09 10:36:20,177 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VnListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-right_vn-60750500 2025-10-09 10:36:20,227 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-right_vn-60750500:ctest-right_vn-60750500 2025-10-09 10:36:20,275 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-right_vn-60750500:ctest-right_vn-60750500 2025-10-09 10:36:20,840 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VnListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-left_vn-76055038 2025-10-09 10:36:20,891 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-left_vn-76055038:ctest-left_vn-76055038 2025-10-09 10:36:20,940 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-left_vn-76055038:ctest-left_vn-76055038 2025-10-09 10:36:21,520 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VnListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-mgmt_vn-25465840 2025-10-09 10:36:21,571 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-mgmt_vn-25465840:ctest-mgmt_vn-25465840 2025-10-09 10:36:21,624 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-88834741:ctest-mgmt_vn-25465840:ctest-mgmt_vn-25465840 2025-10-09 10:36:22,107 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:36:26,237 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:36:36,250 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:c4199c15-6f43-41fd-beeb-5bf785e435ed 2025-10-09 10:36:36,304 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestvDNSBasic0-99526940:ctest-vn1-vdns-31930474:ctest-vn1-vdns-31930474 2025-10-09 10:36:36,353 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:c4199c15-6f43-41fd-beeb-5bf785e435ed 2025-10-09 10:36:53,652 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_DnsInfo? 2025-10-09 10:37:00,953 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:5025cec6-8b5d-4c6b-b088-db61fd83b997 2025-10-09 10:37:01,005 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestvDNSBasic0-99526940:ctest-vn1-vdns-31930474:ctest-vn1-vdns-31930474 2025-10-09 10:37:01,060 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:5025cec6-8b5d-4c6b-b088-db61fd83b997 2025-10-09 10:37:13,066 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_DnsInfo? 2025-10-09 10:37:25,175 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:37:29,091 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:37:39,221 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:6086c881-68db-4250-b2f6-8e37acc3fb1a 2025-10-09 10:37:39,272 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicVMVNx-89609442:ctest-vn-11807250:ctest-vn-11807250 2025-10-09 10:37:39,320 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:6086c881-68db-4250-b2f6-8e37acc3fb1a 2025-10-09 10:37:45,713 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e49e734c-cde6-4c15-af7a-30bcf5f50588 2025-10-09 10:37:45,764 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicVMVNx-89609442:ctest-vn-11807250:ctest-vn-11807250 2025-10-09 10:37:45,815 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e49e734c-cde6-4c15-af7a-30bcf5f50588 2025-10-09 10:37:55,126 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:37:59,235 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:38:07,202 - DEBUG - [10.0.0.33]: Running cmd : docker exec --privileged -it vrouter-vrouter-agent-1 /bin/bash -c 'echo "10.0.0.33 cn-jenkins-deploy-platform-juju-os-587-1-test" >> /etc/hosts' 2025-10-09 10:38:07,331 - DEBUG - Output : 2025-10-09 10:38:08,097 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:8f3fb206-3725-4d63-a434-ee884ad6e13b 2025-10-09 10:38:08,152 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicVMVN-49544884:ctest-vn2_metadata-91439191:ctest-vn2_metadata-91439191 2025-10-09 10:38:08,207 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:8f3fb206-3725-4d63-a434-ee884ad6e13b 2025-10-09 10:38:20,897 - DEBUG - [10.0.0.33]: Running cmd : docker exec --privileged -it vrouter-vrouter-agent-1 /bin/bash -c 'sed -i '$ d' /etc/hosts' 2025-10-09 10:38:21,093 - DEBUG - Output : sed: -e expression #1, char 1: missing command 2025-10-09 10:38:23,988 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:38:24,390 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:38:32,150 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:3b98098d-3d68-4024-bc0c-b4b7873c1e53 2025-10-09 10:38:32,204 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicVMVN-49544884:ctest-vn-47795713:ctest-vn-47795713 2025-10-09 10:38:32,255 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:3b98098d-3d68-4024-bc0c-b4b7873c1e53 2025-10-09 10:38:44,573 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:38:44,957 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:38:52,344 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicVMVN-49544884:ctest-vn-45437348:ctest-vn-45437348 2025-10-09 10:38:52,957 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:bef8d761-5f72-4ead-ada8-731d6c5d0313 2025-10-09 10:38:53,017 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicVMVN-49544884:ctest-vn-45437348:ctest-vn-45437348 2025-10-09 10:38:53,072 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:bef8d761-5f72-4ead-ada8-731d6c5d0313 2025-10-09 10:39:03,616 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:f1af7061-3a28-443b-a48d-0e8bc8fff317 2025-10-09 10:39:03,672 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicVMVN-49544884:ctest-vn-45437348:ctest-vn-45437348 2025-10-09 10:39:03,719 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:f1af7061-3a28-443b-a48d-0e8bc8fff317 2025-10-09 10:39:05,916 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:11ef07fc-d368-42b9-aa4e-94613c7400d0 2025-10-09 10:39:05,968 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicVMVN-49544884:ctest-vn-45437348:ctest-vn-45437348 2025-10-09 10:39:06,019 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:11ef07fc-d368-42b9-aa4e-94613c7400d0 2025-10-09 10:39:24,860 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VnListReq?name=admin_domain:ctest-TestBasicVMVN-49544884:ctest-vn-45437348 2025-10-09 10:39:24,913 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicVMVN-49544884:ctest-vn-45437348:ctest-vn-45437348 2025-10-09 10:39:24,965 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_VrfListReq?name= 2025-10-09 10:39:25,017 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_KRouteReq?x=2 2025-10-09 10:39:25,074 - DEBUG - Unable to fetch route details in vrouter for vrf index 2, Got :[] 2025-10-09 10:39:25,725 - DEBUG - Requesting: https://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 10:41:09,435 - INFO - Using existing project ['admin_domain', 'admin'](4ffa32a1-1c41-42f7-a8b6-f5d80e4454ba) 2025-10-09 10:41:09,447 - INFO - ================================================================================ 2025-10-09 10:41:09,447 - INFO - STARTING TEST : test_contrail_status 2025-10-09 10:41:09,447 - INFO - TEST DESCRIPTION : Test to verify that all services are running and active 2025-10-09 10:41:09,654 - DEBUG - Skipping xmpp flap check 2025-10-09 10:41:09,654 - INFO - Initial checks done. Running the testcase now 2025-10-09 10:41:09,654 - INFO - 2025-10-09 10:41:09,785 - INFO - 10.0.0.33 2025-10-09 10:41:09,785 - INFO - config 2025-10-09 10:41:09,840 - INFO - config-nodemgr:active 2025-10-09 10:41:09,900 - INFO - api-server:active 2025-10-09 10:41:09,955 - INFO - schema:active 2025-10-09 10:41:10,011 - INFO - svc-monitor:active 2025-10-09 10:41:10,067 - INFO - device-manager:active 2025-10-09 10:41:10,067 - INFO - config-database 2025-10-09 10:41:10,067 - INFO - config-cassandra:active 2025-10-09 10:41:10,067 - INFO - config-zookeeper:active 2025-10-09 10:41:10,068 - INFO - config-rabbitmq:active 2025-10-09 10:41:10,068 - INFO - vrouter 2025-10-09 10:41:10,123 - INFO - vrouter-nodemgr:active 2025-10-09 10:41:10,179 - INFO - agent:active 2025-10-09 10:41:10,180 - INFO - control 2025-10-09 10:41:10,236 - INFO - control-nodemgr:active 2025-10-09 10:41:10,291 - INFO - control:active 2025-10-09 10:41:10,291 - INFO - named:active 2025-10-09 10:41:10,344 - INFO - dns:active 2025-10-09 10:41:10,344 - INFO - analytics 2025-10-09 10:41:10,400 - INFO - analytics-nodemgr:active 2025-10-09 10:41:10,451 - INFO - analytics-api:active 2025-10-09 10:41:10,503 - INFO - collector:active 2025-10-09 10:41:10,504 - INFO - analytics-database 2025-10-09 10:41:10,504 - INFO - analytics-cassandra:active 2025-10-09 10:41:10,563 - INFO - analyticsdb-nodemgr:active 2025-10-09 10:41:10,623 - INFO - query-engine:active 2025-10-09 10:41:10,623 - INFO - webui 2025-10-09 10:41:10,623 - INFO - webui:active 2025-10-09 10:41:10,623 - INFO - webui-middleware:active 2025-10-09 10:41:10,624 - INFO - redis:active 2025-10-09 10:41:10,624 - INFO - analytics_snmp 2025-10-09 10:41:10,679 - INFO - snmp-collector:active 2025-10-09 10:41:10,679 - INFO - snmp-topology:active 2025-10-09 10:41:10,679 - INFO - analytics_alarm 2025-10-09 10:41:10,680 - INFO - alarmgen:active 2025-10-09 10:41:10,680 - INFO - All the contrail services are up on all nodes 2025-10-09 10:41:10,680 - INFO - contrail-status passed 2025-10-09 10:41:10,882 - DEBUG - Skipping xmpp flap check 2025-10-09 10:41:10,882 - INFO - END TEST : test_contrail_status : PASSED[0:00:01] 2025-10-09 10:41:10,882 - INFO - --------------------------------------------------------------------------------