09/18/2025 01:27:53.623 7f46fc9d1500 [contrail-config-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/18/2025 01:27:53.624 7f46fc9d1500 [contrail-config-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/18/2025 01:27:53.624 7f46fc9d1500 [contrail-config-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/18/2025 01:27:53.661 7f46f9eda4a0 [contrail-config-database-nodemgr] [ERROR]: Cannot write http_port 8112 to /tmp/contrail-config-database-nodemgr.0.http_port 09/18/2025 01:27:53.661 7f46f9eda4a0 [contrail-config-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8112 09/18/2025 01:27:53.661 7f46f9eda860 [contrail-config-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 09/18/2025 01:27:53.666 7f46f9eda860 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:27:53.671 7f46f9eda860 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/18/2025 01:27:53.673 7f46f9eda860 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/18/2025 01:27:53.673 7f46f9eda860 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:27:53.673 7f46f9eda860 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1485-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/18/2025 01:27:53.673 7f46f9eda860 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:27:53.673 7f46f9eda860 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:27:53.674 7f46f9eda860 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1485-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1758158873669023 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/18/2025 01:27:53.674 7f46f9eda860 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:27:53.877 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 18970271218382288060298170454182658725413934458306769947688820607766436657148, 'start': '1758158869000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'zookeeper', 'group': 'zookeeper', 'pid': 72934788801262346693767589556122072745239992199626968746843193037271882061152, 'start': '1758158869000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-config-database-nodemgr', 'group': 'contrail-config-database-nodemgr', 'pid': 54416256270270679629900833995636773623814392350788722919982942369672426305440, 'start': '1758158869000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 09/18/2025 01:27:54.033 7f46f9eda900 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:27:54.033 7f46f9eda900 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1485-1 build_info = {"build-info" : [{"build-version" : "v0+nightly", "build-time" : "2025-09-18 00:11:34.983097", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-11084-1.novalocal", "build-id" : "package contrail", "build-number" : "lib is not installed "}]} system_cpu_info = << num_socket = 1 num_cpu = 4 num_core_per_socket = 4 num_thread_per_core = 1 >> running_package_version = package contrail-lib is not installed installed_package_version = package contrail-lib is not installed >> 09/18/2025 01:27:54.033 7f46f9eda900 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:27:54.034 7f46fc9d1500 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-k8s-1485-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1758158874034083, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 09/18/2025 01:27:54.034 7f46fc9d1500 [contrail-config-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-k8s-1485-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1758158874034338, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1758158869000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='zookeeper', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1758158869000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-config-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1758158869000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "v0+nightly", "build-time" : "2025-09-18 00:11:34.983097", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-11084-1.novalocal", "build-id" : "package contrail", "build-number" : "lib is not installed\n"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 09/18/2025 01:27:54.038 7f46f9eda900 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:27:54.038 7f46f9eda900 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1485-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/18/2025 01:27:54.038 7f46f9eda900 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:27:54.038 7f46f9eda900 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:27:54.038 7f46f9eda900 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1485-1 deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1758158869000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = zookeeper process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1758158869000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-config-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1758158869000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "v0+nightly", "build-time" : "2025-09-18 00:11:34.983097", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-11084-1.novalocal", "build-id" : "package contrail", "build-number" : "lib is not installed "}]} >> 09/18/2025 01:27:54.038 7f46f9eda900 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:27:54.510 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 09/18/2025 01:27:55.209 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:DatabaseUsage] map 09/18/2025 01:27:55.210 7f46f9d9aae0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:27:55.211 7f46f9d9aae0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-k8s-1485-1 database_usage = [ << disk_space_used_1k = 12120944 disk_space_available_1k = 68969540 config_db_size_1k = 4 >>, ] >> 09/18/2025 01:27:55.211 7f46f9d9aae0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:27:55.406 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to connect to database by CQL and socket: Result: 1 09/18/2025 01:27:55.407 7f46fc9d1500 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-k8s-1485-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1758158875406935, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 09/18/2025 01:27:55.408 7f46f9edaf40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:27:55.408 7f46f9edaf40 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1485-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/18/2025 01:27:55.408 7f46f9edaf40 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:27:57.272 7f46f9edaf40 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:27:57.666 7f46f9edaf40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/18/2025 01:27:57.691 7f46f9edaf40 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/18/2025 01:27:57.691 7f46f9edaf40 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/18/2025 01:27:57.691 7f46f9edaf40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/18/2025 01:27:57.691 7f46f9edaf40 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1485-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/18/2025 01:27:57.692 7f46f9edaf40 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/18/2025 01:27:57.692 7f46f9edaf40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/18/2025 01:27:57.692 7f46f9edaf40 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/18/2025 01:27:57.692 7f46f9edaf40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:27:57.692 7f46f9edaf40 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1485-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/18/2025 01:27:57.692 7f46f9edaf40 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:27:57.692 7f46f9edaf40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:27:57.693 7f46f9edaf40 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1485-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1758158873669023 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/18/2025 01:27:57.693 7f46f9edaf40 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:27:58.169 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 09/18/2025 01:28:00.012 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr config_db_uuid'. Error is: Result: 1 09/18/2025 01:28:01.671 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr to_bgp_keyspace'. Error is: Result: 1 09/18/2025 01:28:01.691 7f46f9d9af40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/18/2025 01:28:01.693 7f46f9d9af40 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/18/2025 01:28:01.693 7f46f9d9af40 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/18/2025 01:28:01.693 7f46f9d9af40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/18/2025 01:28:01.693 7f46f9d9af40 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1485-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/18/2025 01:28:01.693 7f46f9d9af40 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/18/2025 01:28:01.693 7f46f9d9af40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/18/2025 01:28:01.693 7f46f9d9af40 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/18/2025 01:28:01.693 7f46f9d9af40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:28:01.694 7f46f9d9af40 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1485-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/18/2025 01:28:01.694 7f46f9d9af40 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:28:01.694 7f46f9d9af40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:28:01.694 7f46f9d9af40 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1485-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1758158873669023 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/18/2025 01:28:01.694 7f46f9d9af40 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:28:04.659 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr svc_monitor_keyspace'. Error is: Result: 1 09/18/2025 01:28:05.694 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/18/2025 01:28:05.695 7f46f9edac20 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/18/2025 01:28:05.695 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/18/2025 01:28:05.695 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/18/2025 01:28:05.696 7f46f9edac20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1485-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/18/2025 01:28:05.696 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/18/2025 01:28:05.696 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/18/2025 01:28:05.696 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/18/2025 01:28:05.696 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:28:05.697 7f46f9edac20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1485-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/18/2025 01:28:05.697 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:28:05.697 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:28:05.697 7f46f9edac20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1485-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1758158873669023 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/18/2025 01:28:05.697 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:28:05.892 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr DISCOVERY_SERVER'. Error is: Result: 1 09/18/2025 01:28:07.040 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr useragent'. Error is: Result: 1 09/18/2025 01:28:08.193 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr dm_keyspace'. Error is: Result: 1 09/18/2025 01:28:09.037 7f46f9b6b4a0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:28:09.696 7f46f9b6b400 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/18/2025 01:28:09.697 7f46f9b6b400 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/18/2025 01:28:09.697 7f46f9b6b400 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/18/2025 01:28:09.697 7f46f9b6b400 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/18/2025 01:28:09.697 7f46f9b6b400 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1485-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/18/2025 01:28:09.697 7f46f9b6b400 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/18/2025 01:28:09.697 7f46f9b6b400 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/18/2025 01:28:09.698 7f46f9b6b400 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/18/2025 01:28:09.698 7f46f9b6b400 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:28:09.698 7f46f9b6b400 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1485-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/18/2025 01:28:09.698 7f46f9b6b400 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:28:09.698 7f46f9b6b400 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:28:09.698 7f46f9b6b400 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1485-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1758158873669023 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/18/2025 01:28:09.698 7f46f9b6b400 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:28:13.253 7f46fc9d1500 [contrail-config-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-k8s-1485-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1758158893253054, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1758158869000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='zookeeper', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1758158869000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-config-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1758158869000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "v0+nightly", "build-time" : "2025-09-18 00:11:34.983097", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-11084-1.novalocal", "build-id" : "package contrail", "build-number" : "lib is not installed\n"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 09/18/2025 01:28:13.253 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12147196, partition_space_available_1k=68943288, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1272292, cpu_share=0.0, mem_res=1259752, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=57136, cpu_share=0.0, mem_res=52580, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=56232, cpu_share=0.0, mem_res=44812, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=4890800, free=311796, buffers=387968, cached=10593644, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.96, five_min_avg=2.18, fifteen_min_avg=1.06, cpu_share=0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 09/18/2025 01:28:13.253 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:28:13.253 7f46f9edac20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1485-1 deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1758158869000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = zookeeper process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1758158869000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-config-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1758158869000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "v0+nightly", "build-time" : "2025-09-18 00:11:34.983097", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-11084-1.novalocal", "build-id" : "package contrail", "build-number" : "lib is not installed "}]} >> 09/18/2025 01:28:13.253 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:28:13.253 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:28:13.254 7f46f9edac20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1485-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 12147196 partition_space_available_1k = 68943288 percentage_partition_space_used = 15 >> } all_core_file_list = [ ] process_mem_cpu_usage = { cassandra : << mem_virt = 1272292 cpu_share = 0.0 mem_res = 1259752 >>zookeeper : << mem_virt = 57136 cpu_share = 0.0 mem_res = 52580 >>contrail-config-database-nodemgr : << mem_virt = 56232 cpu_share = 0.0 mem_res = 44812 >> } system_mem_usage = << total = 16184208 used = 4890800 free = 311796 buffers = 387968 cached = 10593644 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 4.96 five_min_avg = 2.18 fifteen_min_avg = 1.06 cpu_share = 0 node_type = config-database-node >> >> 09/18/2025 01:28:13.254 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/18/2025 01:28:13.697 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/18/2025 01:28:13.698 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Session Event: TCP Connected 09/18/2025 01:28:13.698 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/18/2025 01:28:13.698 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/18/2025 01:28:13.699 7f46f9edac20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1485-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/18/2025 01:28:13.699 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/18/2025 01:28:13.699 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/18/2025 01:28:13.699 7f46f9edac20 [contrail-config-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 09/18/2025 01:28:13.699 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/18/2025 01:28:13.699 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/18/2025 01:28:13.699 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/18/2025 01:28:13.703 7f46f9b6bae0 [contrail-config-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001013 09/18/2025 01:28:13.704 7f46f9b6bae0 [contrail-config-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 09/18/2025 01:28:13.704 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 09/18/2025 01:28:13.704 7f46f9edac20 [contrail-config-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 09/18/2025 01:28:13.704 7f46f9edac20 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:DatabaseUsage] map 09/18/2025 01:28:13.705 7f46f9edac20 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:28:13.705 7f46f9edac20 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/18/2025 01:28:13.705 7f46f9edac20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 09/18/2025 01:28:13.705 7f46f9b6b720 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/18/2025 01:28:13.706 7f46f9b6b720 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/18/2025 01:28:27.289 7f46f9b6bae0 [contrail-config-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001434 09/18/2025 01:28:27.289 7f46f9b6bae0 [contrail-config-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 09/18/2025 01:28:27.290 7f46f9edaf40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 09/18/2025 01:28:27.290 7f46f9edaf40 [contrail-config-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 3 09/18/2025 01:28:27.290 7f46f9edaf40 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:DatabaseUsage] map 09/18/2025 01:28:27.290 7f46f9edaf40 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:28:27.290 7f46f9edaf40 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/18/2025 01:28:27.290 7f46f9b6b360 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/18/2025 01:28:27.291 7f46f9b6b360 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/18/2025 01:28:37.251 7f46f9edaf40 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:28:47.907 7f46f9d9af40 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:28:53.748 7f46f9d9af40 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:28:54.741 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 09/18/2025 01:28:55.470 7f46fc9d1500 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 09/18/2025 01:28:55.470 7f46fc9d1500 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-k8s-1485-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1758158935470469, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.38:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 09/18/2025 01:28:56.682 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 09/18/2025 01:28:57.428 7f46f9b6b7c0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:29:01.881 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=13012008, partition_space_available_1k=68078476, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1341540, cpu_share=0.36, mem_res=1324680, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=150596, cpu_share=0.11, mem_res=145076, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55476, cpu_share=0.01, mem_res=44928, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7533460, free=209344, buffers=361232, cached=8080172, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=13.36, five_min_avg=4.82, fifteen_min_avg=2.02, cpu_share=21.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 09/18/2025 01:29:05.043 7f46f9edac20 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:29:10.702 7f46f9d9a400 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:29:54.221 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 09/18/2025 01:29:54.597 7f46fc9d1500 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 09/18/2025 01:29:55.340 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 09/18/2025 01:30:00.894 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=13033532, partition_space_available_1k=68056952, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1350300, cpu_share=0.22, mem_res=1333736, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=162656, cpu_share=0.01, mem_res=157088, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55740, cpu_share=0.01, mem_res=45124, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7478956, free=291688, buffers=369240, cached=8044324, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=8.67, five_min_avg=5.3, fifteen_min_avg=2.37, cpu_share=12.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 09/18/2025 01:30:52.469 7f46f9edaf40 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:30:54.239 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 09/18/2025 01:30:54.659 7f46fc9d1500 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 09/18/2025 01:30:55.417 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 09/18/2025 01:31:00.895 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=13042036, partition_space_available_1k=68048448, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1350156, cpu_share=0.03, mem_res=1333800, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=163132, cpu_share=0.0, mem_res=157484, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55572, cpu_share=0.01, mem_res=45180, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7515540, free=319292, buffers=379200, cached=7970176, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.3, five_min_avg=4.92, fifteen_min_avg=2.42, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 09/18/2025 01:31:14.476 7f46f9b6b900 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:31:17.251 7f46f9b6b900 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:31:21.680 7f46f9b6b900 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:31:54.254 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 09/18/2025 01:31:54.653 7f46fc9d1500 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 09/18/2025 01:31:55.686 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 09/18/2025 01:32:00.912 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14766436, partition_space_available_1k=66324048, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1349868, cpu_share=0.03, mem_res=1333716, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=163592, cpu_share=0.0, mem_res=158012, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55364, cpu_share=0.01, mem_res=45184, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7581728, free=614716, buffers=375328, cached=7612436, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.35, five_min_avg=4.69, fifteen_min_avg=2.5, cpu_share=12.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 09/18/2025 01:32:54.268 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 09/18/2025 01:32:54.654 7f46fc9d1500 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 09/18/2025 01:32:55.409 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 09/18/2025 01:33:00.957 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14326960, partition_space_available_1k=66763524, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1350272, cpu_share=0.11, mem_res=1333872, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=164892, cpu_share=0.0, mem_res=159332, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55632, cpu_share=0.01, mem_res=45128, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7707328, free=445968, buffers=387464, cached=7643448, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=7.51, five_min_avg=5.72, fifteen_min_avg=3.0, cpu_share=11.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 09/18/2025 01:33:54.276 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 09/18/2025 01:33:54.662 7f46fc9d1500 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 09/18/2025 01:33:55.398 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 09/18/2025 01:34:00.938 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14346688, partition_space_available_1k=66743796, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1351820, cpu_share=0.11, mem_res=1334672, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=168404, cpu_share=0.01, mem_res=162632, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55668, cpu_share=0.01, mem_res=45148, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7738540, free=391820, buffers=387680, cached=7666168, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.02, five_min_avg=5.37, fifteen_min_avg=3.05, cpu_share=10.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 09/18/2025 01:34:54.279 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 09/18/2025 01:34:54.684 7f46fc9d1500 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 09/18/2025 01:34:55.406 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 09/18/2025 01:35:00.945 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14362164, partition_space_available_1k=66728320, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1351960, cpu_share=0.03, mem_res=1334588, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=168868, cpu_share=0.0, mem_res=163292, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55780, cpu_share=0.01, mem_res=45108, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7741408, free=380552, buffers=387736, cached=7674512, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.42, five_min_avg=4.82, fifteen_min_avg=3.01, cpu_share=8.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 09/18/2025 01:35:22.855 7f46f9edad60 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:35:25.500 7f46f9edad60 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:35:50.850 7f46f9b6bb80 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:35:54.304 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 09/18/2025 01:35:54.756 7f46fc9d1500 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 09/18/2025 01:35:55.694 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 09/18/2025 01:36:00.957 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14377012, partition_space_available_1k=66713472, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1352324, cpu_share=0.05, mem_res=1334984, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=171716, cpu_share=0.0, mem_res=165932, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55748, cpu_share=0.01, mem_res=45124, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7794104, free=393604, buffers=385044, cached=7611456, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.3, five_min_avg=4.55, fifteen_min_avg=3.02, cpu_share=11.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 09/18/2025 01:36:35.585 7f46f9d9a7c0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:36:54.329 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 09/18/2025 01:36:54.715 7f46fc9d1500 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 09/18/2025 01:36:55.437 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 09/18/2025 01:37:00.956 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14386008, partition_space_available_1k=66704476, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1352040, cpu_share=0.03, mem_res=1334760, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=172176, cpu_share=0.0, mem_res=166328, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55796, cpu_share=0.01, mem_res=45192, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7770032, free=358048, buffers=394716, cached=7661412, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.43, five_min_avg=4.09, fifteen_min_avg=2.96, cpu_share=9.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 09/18/2025 01:37:54.313 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 09/18/2025 01:37:54.705 7f46fc9d1500 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 09/18/2025 01:37:55.437 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 09/18/2025 01:38:00.985 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14397280, partition_space_available_1k=66693204, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1352280, cpu_share=0.05, mem_res=1334792, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=173260, cpu_share=0.0, mem_res=167516, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55812, cpu_share=0.01, mem_res=44972, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7783708, free=329776, buffers=394788, cached=7675936, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.9, five_min_avg=3.98, fifteen_min_avg=3.0, cpu_share=8.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 09/18/2025 01:38:54.314 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 09/18/2025 01:38:54.713 7f46fc9d1500 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 09/18/2025 01:38:55.457 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 09/18/2025 01:39:00.982 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14405276, partition_space_available_1k=66685208, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1352740, cpu_share=0.03, mem_res=1334860, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=173716, cpu_share=0.0, mem_res=168044, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55800, cpu_share=0.01, mem_res=45000, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7784472, free=319572, buffers=394812, cached=7685352, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.77, five_min_avg=3.7, fifteen_min_avg=2.96, cpu_share=7.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 09/18/2025 01:39:40.413 7f46f9d2ed60 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:39:42.981 7f46f9d2ed60 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:39:46.973 7f46f9edaf40 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:39:54.323 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 09/18/2025 01:39:54.719 7f46fc9d1500 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 09/18/2025 01:39:55.484 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 09/18/2025 01:40:00.985 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14414544, partition_space_available_1k=66675940, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1353080, cpu_share=0.05, mem_res=1335036, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=174760, cpu_share=0.0, mem_res=169100, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55944, cpu_share=0.01, mem_res=44976, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7790392, free=301872, buffers=394904, cached=7697040, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.86, five_min_avg=3.95, fifteen_min_avg=3.1, cpu_share=9.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 09/18/2025 01:40:54.349 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 09/18/2025 01:40:54.747 7f46fc9d1500 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 09/18/2025 01:40:55.523 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 09/18/2025 01:41:00.986 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14418004, partition_space_available_1k=66672480, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1353056, cpu_share=0.03, mem_res=1334980, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=175252, cpu_share=0.0, mem_res=169496, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=56000, cpu_share=0.01, mem_res=45216, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7802560, free=282012, buffers=395012, cached=7704624, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.64, five_min_avg=3.89, fifteen_min_avg=3.13, cpu_share=10.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 09/18/2025 01:41:17.157 7f46fa7db040 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:41:54.354 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 09/18/2025 01:41:54.789 7f46fc9d1500 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 09/18/2025 01:41:55.506 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 09/18/2025 01:42:00.998 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14424364, partition_space_available_1k=66666120, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1353068, cpu_share=0.05, mem_res=1335040, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=176272, cpu_share=0.0, mem_res=170552, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55948, cpu_share=0.01, mem_res=45324, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7800544, free=275652, buffers=395032, cached=7712980, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.44, five_min_avg=3.86, fifteen_min_avg=3.17, cpu_share=8.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 09/18/2025 01:42:54.362 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 09/18/2025 01:42:54.753 7f46fc9d1500 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 09/18/2025 01:42:55.510 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 09/18/2025 01:43:01.028 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14432664, partition_space_available_1k=66657820, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1353316, cpu_share=0.03, mem_res=1335128, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=176768, cpu_share=0.0, mem_res=171080, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55916, cpu_share=0.01, mem_res=45124, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7809116, free=256828, buffers=395040, cached=7723224, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.04, five_min_avg=3.39, fifteen_min_avg=3.05, cpu_share=8.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 09/18/2025 01:43:54.366 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 09/18/2025 01:43:54.764 7f46fc9d1500 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 09/18/2025 01:43:55.549 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 09/18/2025 01:44:01.018 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14438856, partition_space_available_1k=66651628, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1353940, cpu_share=0.06, mem_res=1335204, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=177008, cpu_share=0.01, mem_res=171184, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55876, cpu_share=0.01, mem_res=45052, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7819640, free=238968, buffers=395064, cached=7730536, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.25, five_min_avg=3.22, fifteen_min_avg=3.02, cpu_share=8.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 09/18/2025 01:44:54.364 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 09/18/2025 01:44:54.720 7f46fc9d1500 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 09/18/2025 01:44:55.423 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 09/18/2025 01:45:01.032 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14441172, partition_space_available_1k=66649312, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1353924, cpu_share=0.03, mem_res=1335256, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=177480, cpu_share=0.0, mem_res=171712, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55964, cpu_share=0.01, mem_res=45236, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7821856, free=233436, buffers=395068, cached=7733848, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.64, five_min_avg=3.1, fifteen_min_avg=2.98, cpu_share=7.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 09/18/2025 01:45:36.800 7f46fa7db5e0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:45:39.446 7f46fa7db5e0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:45:43.730 7f46fa7db5e0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:45:54.384 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 09/18/2025 01:45:54.781 7f46fc9d1500 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 09/18/2025 01:45:55.517 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 09/18/2025 01:46:01.034 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14446844, partition_space_available_1k=66643640, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1351600, cpu_share=0.05, mem_res=1332736, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=178520, cpu_share=0.0, mem_res=172768, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=56020, cpu_share=0.01, mem_res=45084, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7849856, free=286360, buffers=391600, cached=7656392, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.19, five_min_avg=3.57, fifteen_min_avg=3.16, cpu_share=12.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 09/18/2025 01:46:33.761 7f46fa7db5e0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 09/18/2025 01:46:54.399 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 09/18/2025 01:46:54.788 7f46fc9d1500 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 09/18/2025 01:46:55.505 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 09/18/2025 01:47:01.048 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14449808, partition_space_available_1k=66640676, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1351552, cpu_share=0.03, mem_res=1332712, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=178988, cpu_share=0.0, mem_res=173296, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=56020, cpu_share=0.01, mem_res=45096, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7852912, free=278632, buffers=391804, cached=7660860, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.49, five_min_avg=3.19, fifteen_min_avg=3.05, cpu_share=7.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 09/18/2025 01:47:54.403 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 09/18/2025 01:47:54.773 7f46fc9d1500 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 09/18/2025 01:47:55.501 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 09/18/2025 01:48:01.058 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14460832, partition_space_available_1k=66629652, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1351664, cpu_share=0.03, mem_res=1332588, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=179456, cpu_share=0.0, mem_res=173692, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=56236, cpu_share=0.01, mem_res=45216, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7857216, free=270012, buffers=391808, cached=7665172, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.89, five_min_avg=2.91, fifteen_min_avg=2.96, cpu_share=7.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 09/18/2025 01:48:54.418 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 09/18/2025 01:48:54.825 7f46fc9d1500 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 09/18/2025 01:48:55.549 7f46fc9d1500 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 09/18/2025 01:49:01.081 7f46fc9d1500 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1485-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14474104, partition_space_available_1k=66616380, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1353004, cpu_share=0.06, mem_res=1333264, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=181844, cpu_share=0.0, mem_res=176068, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=56224, cpu_share=0.01, mem_res=45052, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7861304, free=253176, buffers=391820, cached=7677908, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.12, five_min_avg=2.76, fifteen_min_avg=2.91, cpu_share=9.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 09/18/2025 01:49:45.036 7f46f9b6bc20 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map