10/13/2025 01:37:50.409 7fa205e5a240 [contrail-config-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 10/13/2025 01:37:50.409 7fa205e5a240 [contrail-config-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 10/13/2025 01:37:50.409 7fa205e5a240 [contrail-config-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 10/13/2025 01:37:50.431 7fa2032ef4a0 [contrail-config-database-nodemgr] [ERROR]: Cannot write http_port 8112 to /tmp/contrail-config-database-nodemgr.0.http_port 10/13/2025 01:37:50.432 7fa2032ef4a0 [contrail-config-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8112 10/13/2025 01:37:50.432 7fa2032ef860 [contrail-config-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 10/13/2025 01:37:50.433 7fa2032ef860 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:37:50.434 7fa2032ef860 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 10/13/2025 01:37:50.434 7fa2032ef860 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/13/2025 01:37:50.434 7fa2032ef860 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:37:50.435 7fa2032ef860 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-589-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 10/13/2025 01:37:50.435 7fa2032ef860 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:37:50.435 7fa2032ef860 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:37:50.435 7fa2032ef860 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-589-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1760319470434168 collector_name = collector_ip = collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/13/2025 01:37:50.435 7fa2032ef860 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:37:50.600 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 113121190669816956754330554639398966352126300545029991103950254337795660777268, 'start': '1760319467000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'zookeeper', 'group': 'zookeeper', 'pid': 64524846638507333597930798491842442333402677616550295313463688599517054884793, 'start': '1760319467000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-config-database-nodemgr', 'group': 'contrail-config-database-nodemgr', 'pid': 23342494389591193402960433448031418047784245978386497029052890046874500926673, 'start': '1760319467000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 10/13/2025 01:37:50.848 7fa202faa4a0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:37:50.849 7fa202faa4a0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-589-1 build_info = {"build-info" : [{"build-version" : "v0+nightly", "build-time" : "2025-10-13 00:48:09.442511", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-11624-1.novalocal", "build-id" : "package contrail", "build-number" : "lib is not installed "}]} system_cpu_info = << num_socket = 1 num_cpu = 8 num_core_per_socket = 8 num_thread_per_core = 1 >> running_package_version = package contrail-lib is not installed installed_package_version = package contrail-lib is not installed >> 10/13/2025 01:37:50.849 7fa202faa4a0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:37:50.850 7fa205e5a240 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-589-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1760319470850615, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 10/13/2025 01:37:50.851 7fa205e5a240 [contrail-config-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-589-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1760319470851256, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1760319467000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='zookeeper', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1760319467000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-config-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1760319467000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "v0+nightly", "build-time" : "2025-10-13 00:48:09.442511", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-11624-1.novalocal", "build-id" : "package contrail", "build-number" : "lib is not installed\n"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 10/13/2025 01:37:50.855 7fa202faa4a0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:37:50.856 7fa202faa4a0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-589-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 10/13/2025 01:37:50.856 7fa202faa4a0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:37:50.857 7fa202faa4a0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:37:50.857 7fa202faa4a0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-589-1 deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1760319467000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = zookeeper process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1760319467000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-config-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1760319467000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "v0+nightly", "build-time" : "2025-10-13 00:48:09.442511", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-11624-1.novalocal", "build-id" : "package contrail", "build-number" : "lib is not installed "}]} >> 10/13/2025 01:37:50.858 7fa202faa4a0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:37:51.417 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 01:37:52.048 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:DatabaseUsage] map 10/13/2025 01:37:52.050 7fa2032efcc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:37:52.051 7fa2032efcc0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-589-1 database_usage = [ << disk_space_used_1k = 54042412 disk_space_available_1k = 27048072 config_db_size_1k = 4 >>, ] >> 10/13/2025 01:37:52.051 7fa2032efcc0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:37:52.954 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to connect to database by CQL and socket: Result: 1 10/13/2025 01:37:52.954 7fa205e5a240 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-589-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1760319472954252, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 10/13/2025 01:37:52.955 7fa2032efcc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:37:52.955 7fa2032efcc0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-589-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 10/13/2025 01:37:52.955 7fa2032efcc0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:37:54.434 7fa2032efcc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/13/2025 01:37:54.939 7fa202faa9a0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:37:55.613 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 01:37:57.749 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr config_db_uuid'. Error is: Result: 1 10/13/2025 01:37:59.290 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr to_bgp_keyspace'. Error is: Result: 1 10/13/2025 01:37:59.439 7fa2032efcc0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/13/2025 01:37:59.439 7fa2032efcc0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/13/2025 01:37:59.439 7fa2032efcc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/13/2025 01:37:59.439 7fa2032efcc0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-589-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/13/2025 01:37:59.440 7fa2032efcc0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/13/2025 01:37:59.440 7fa2032efcc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/13/2025 01:37:59.441 7fa2032efcc0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/13/2025 01:37:59.441 7fa2032efcc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:37:59.441 7fa2032efcc0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-589-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/13/2025 01:37:59.442 7fa2032efcc0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:37:59.446 7fa2032efcc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:37:59.446 7fa2032efcc0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-589-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1760319470434168 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 10/13/2025 01:37:59.446 7fa2032efcc0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:38:00.775 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr svc_monitor_keyspace'. Error is: Result: 1 10/13/2025 01:38:02.358 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr DISCOVERY_SERVER'. Error is: Result: 1 10/13/2025 01:38:03.439 7fa2032eff40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/13/2025 01:38:03.524 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr useragent'. Error is: Result: 1 10/13/2025 01:38:04.660 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr dm_keyspace'. Error is: Result: 1 10/13/2025 01:38:05.066 7fa202faae00 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:38:05.697 7fa2032eff40 [contrail-config-database-nodemgr] [INFO]: Session Event: TCP Connected 10/13/2025 01:38:05.697 7fa2032eff40 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/13/2025 01:38:05.697 7fa2032eff40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/13/2025 01:38:05.698 7fa2032eff40 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-589-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/13/2025 01:38:05.698 7fa2032eff40 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/13/2025 01:38:05.698 7fa2032eff40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 10/13/2025 01:38:05.698 7fa2032eff40 [contrail-config-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 10/13/2025 01:38:05.698 7fa2032eff40 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 10/13/2025 01:38:05.699 7fa2032eff40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/13/2025 01:38:05.699 7fa2032eff40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/13/2025 01:38:05.702 7fa202faa360 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 10/13/2025 01:38:05.702 7fa2032eff40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 10/13/2025 01:38:05.703 7fa2032eff40 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 10/13/2025 01:38:05.703 7fa2032eff40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:38:05.703 7fa2032eff40 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-589-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Cassandra state detected DOWN. Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 10/13/2025 01:38:05.703 7fa2032eff40 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:38:05.703 7fa2032eff40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:38:05.703 7fa2032eff40 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-589-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1760319470434168 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 10/13/2025 01:38:05.704 7fa2032eff40 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:38:09.705 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/13/2025 01:38:09.714 7fa205e5a240 [contrail-config-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-589-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1760319489714860, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1760319467000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='zookeeper', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1760319467000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-config-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1760319467000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "v0+nightly", "build-time" : "2025-10-13 00:48:09.442511", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-11624-1.novalocal", "build-id" : "package contrail", "build-number" : "lib is not installed\n"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 10/13/2025 01:38:09.715 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=54258876, partition_space_available_1k=26831608, percentage_partition_space_used=67, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1297840, cpu_share=0.0, mem_res=1258796, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=172080, cpu_share=0.0, mem_res=117228, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52148, cpu_share=0.0, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=14722668, free=10385176, buffers=752784, cached=7319096, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=8.58, five_min_avg=5.0, fifteen_min_avg=2.98, cpu_share=0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 01:38:09.728 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Session Event: TCP Connected 10/13/2025 01:38:09.728 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/13/2025 01:38:09.728 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/13/2025 01:38:09.729 7fa202faaf40 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-589-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/13/2025 01:38:09.729 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/13/2025 01:38:09.729 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/13/2025 01:38:09.729 7fa202faaf40 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-589-1 deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1760319467000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = zookeeper process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1760319467000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-config-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1760319467000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "v0+nightly", "build-time" : "2025-10-13 00:48:09.442511", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-11624-1.novalocal", "build-id" : "package contrail", "build-number" : "lib is not installed "}]} >> 10/13/2025 01:38:09.730 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/13/2025 01:38:09.730 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/13/2025 01:38:09.730 7fa202faaf40 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-589-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 54258876 partition_space_available_1k = 26831608 percentage_partition_space_used = 67 >> } all_core_file_list = [ ] process_mem_cpu_usage = { cassandra : << mem_virt = 1297840 cpu_share = 0.0 mem_res = 1258796 >>zookeeper : << mem_virt = 172080 cpu_share = 0.0 mem_res = 117228 >>contrail-config-database-nodemgr : << mem_virt = 52148 cpu_share = 0.0 mem_res = 42704 >> } system_mem_usage = << total = 32665120 used = 14722668 free = 10385176 buffers = 752784 cached = 7319096 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 8.58 five_min_avg = 5.0 fifteen_min_avg = 2.98 cpu_share = 0 node_type = config-database-node >> >> 10/13/2025 01:38:09.730 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/13/2025 01:38:09.730 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 10/13/2025 01:38:09.731 7fa202faaf40 [contrail-config-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 10/13/2025 01:38:09.731 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 10/13/2025 01:38:09.732 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/13/2025 01:38:09.732 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/13/2025 01:38:09.734 7fa202faa360 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 10/13/2025 01:38:09.735 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 10/13/2025 01:38:09.736 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 10/13/2025 01:38:09.736 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:38:09.736 7fa202faaf40 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-589-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Cassandra state detected DOWN. Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 10/13/2025 01:38:09.736 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:38:09.737 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:38:09.737 7fa202faaf40 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-589-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 2 pid = 1 http_port = 8112 start_time = 1760319470434168 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/13/2025 01:38:09.737 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/13/2025 01:38:12.133 7fa202faaf40 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:38:13.738 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/13/2025 01:38:13.756 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Session Event: TCP Connected 10/13/2025 01:38:13.756 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/13/2025 01:38:13.756 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/13/2025 01:38:13.756 7fa202faaf40 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-589-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/13/2025 01:38:13.756 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/13/2025 01:38:13.756 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 10/13/2025 01:38:13.757 7fa202faaf40 [contrail-config-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 10/13/2025 01:38:13.757 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 10/13/2025 01:38:13.757 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/13/2025 01:38:13.758 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/13/2025 01:38:13.770 7fa202faa360 [contrail-config-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001012 10/13/2025 01:38:13.770 7fa202faa360 [contrail-config-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 10/13/2025 01:38:13.771 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 10/13/2025 01:38:13.771 7fa202faaf40 [contrail-config-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 10/13/2025 01:38:13.771 7fa202faaf40 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:DatabaseUsage] map 10/13/2025 01:38:13.772 7fa202faaf40 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:38:13.772 7fa202faaf40 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 10/13/2025 01:38:13.773 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 10/13/2025 01:38:13.773 7fa202faad60 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 10/13/2025 01:38:13.779 7fa202faad60 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 10/13/2025 01:38:15.662 7fa202faa360 [contrail-config-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001432 10/13/2025 01:38:15.663 7fa202faa360 [contrail-config-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 10/13/2025 01:38:15.663 7fa202faa7c0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 10/13/2025 01:38:15.663 7fa202faa7c0 [contrail-config-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 3 10/13/2025 01:38:15.663 7fa202faa7c0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:DatabaseUsage] map 10/13/2025 01:38:15.663 7fa202faa7c0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:38:15.664 7fa202faa7c0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 10/13/2025 01:38:15.664 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 10/13/2025 01:38:15.664 7fa202faaf40 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 10/13/2025 01:38:25.394 7fa202faa900 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:38:43.823 7fa202faa4a0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:38:51.030 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 01:38:51.370 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 01:38:51.370 7fa205e5a240 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-589-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1760319531370728, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.129:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 10/13/2025 01:38:52.101 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 01:38:58.113 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=54298188, partition_space_available_1k=26792296, percentage_partition_space_used=67, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1342308, cpu_share=0.54, mem_res=1329044, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=188124, cpu_share=0.03, mem_res=143296, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44764, cpu_share=0.01, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=16752548, free=8542052, buffers=772772, cached=7065576, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=7.21, five_min_avg=5.34, fifteen_min_avg=3.21, cpu_share=6.12, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 01:39:10.754 7fa202faae00 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:39:51.042 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 01:39:51.385 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 01:39:52.099 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 01:39:58.126 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=54307792, partition_space_available_1k=26782692, percentage_partition_space_used=67, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1338960, cpu_share=0.04, mem_res=1326548, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=188552, cpu_share=0.0, mem_res=143716, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44776, cpu_share=0.01, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=16557916, free=8714128, buffers=773104, cached=7088000, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.63, five_min_avg=4.65, fifteen_min_avg=3.11, cpu_share=2.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 01:40:51.078 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 01:40:51.473 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 01:40:52.401 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 01:40:58.144 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=54319836, partition_space_available_1k=26770648, percentage_partition_space_used=67, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1341744, cpu_share=0.04, mem_res=1329328, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=190216, cpu_share=0.01, mem_res=145372, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44776, cpu_share=0.01, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=16626832, free=7166056, buffers=809120, cached=8531060, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=7.8, five_min_avg=5.43, fifteen_min_avg=3.46, cpu_share=5.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 01:41:05.263 7fa203c4b040 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:41:09.981 7fa202faa900 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:41:15.070 7fa202faa900 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:41:51.065 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 01:41:51.403 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 01:41:52.087 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 01:41:57.150 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=54234880, partition_space_available_1k=26855604, percentage_partition_space_used=67, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1341760, cpu_share=0.03, mem_res=1329340, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=190692, cpu_share=0.0, mem_res=145848, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44784, cpu_share=0.01, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=16533428, free=7242128, buffers=809336, cached=8548076, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.16, five_min_avg=4.83, fifteen_min_avg=3.37, cpu_share=2.88, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 01:42:07.814 7fa202faa220 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:42:51.082 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 01:42:51.417 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 01:42:52.146 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 01:42:58.179 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=54254640, partition_space_available_1k=26835844, percentage_partition_space_used=67, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1341780, cpu_share=0.03, mem_res=1329352, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=191152, cpu_share=0.0, mem_res=146308, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44812, cpu_share=0.01, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=16658960, free=6968488, buffers=809420, cached=8696096, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.41, five_min_avg=4.51, fifteen_min_avg=3.35, cpu_share=4.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 01:43:51.144 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 01:43:51.482 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 01:43:52.192 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 01:43:58.179 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=54248184, partition_space_available_1k=26842300, percentage_partition_space_used=67, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1343504, cpu_share=0.04, mem_res=1330456, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=191600, cpu_share=0.0, mem_res=146752, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44820, cpu_share=0.01, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=16630396, free=6975480, buffers=809424, cached=8716700, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.45, five_min_avg=4.02, fifteen_min_avg=3.26, cpu_share=2.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 01:44:51.117 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 01:44:51.464 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 01:44:52.198 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 01:44:58.196 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=54254876, partition_space_available_1k=26835608, percentage_partition_space_used=67, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1343576, cpu_share=0.03, mem_res=1330528, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=192068, cpu_share=0.0, mem_res=147220, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44820, cpu_share=0.01, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=16635544, free=6962580, buffers=809424, cached=8724436, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.22, five_min_avg=3.66, fifteen_min_avg=3.18, cpu_share=2.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 01:45:44.966 7fa203c4b040 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:45:49.558 7fa203c4b040 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:45:51.109 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 01:45:51.511 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 01:45:52.301 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 01:45:54.782 7fa202fc0220 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:45:58.197 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=54271152, partition_space_available_1k=26819332, percentage_partition_space_used=67, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1343664, cpu_share=0.04, mem_res=1330616, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=194456, cpu_share=0.0, mem_res=149576, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44828, cpu_share=0.01, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=16693632, free=6873720, buffers=809444, cached=8755204, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.39, five_min_avg=4.34, fifteen_min_avg=3.45, cpu_share=4.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 01:46:51.112 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 01:46:51.429 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 01:46:52.144 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 01:46:56.746 7fa2032efea0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:46:57.206 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=54280288, partition_space_available_1k=26810196, percentage_partition_space_used=67, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1343704, cpu_share=0.03, mem_res=1330652, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=194872, cpu_share=0.0, mem_res=149996, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44836, cpu_share=0.01, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=16772928, free=6780304, buffers=809452, cached=8769184, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.42, five_min_avg=3.96, fifteen_min_avg=3.37, cpu_share=2.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 01:47:51.311 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 01:47:51.668 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 01:47:52.502 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 01:47:58.229 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=55497520, partition_space_available_1k=25592964, percentage_partition_space_used=68, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1343692, cpu_share=0.03, mem_res=1330664, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=195344, cpu_share=0.0, mem_res=150472, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44836, cpu_share=0.01, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=16934908, free=5320304, buffers=840584, cached=10035900, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.71, five_min_avg=3.89, fifteen_min_avg=3.38, cpu_share=3.38, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 01:48:51.139 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 01:48:51.545 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 01:48:52.314 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 01:48:58.234 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=55508552, partition_space_available_1k=25581932, percentage_partition_space_used=68, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1345356, cpu_share=0.07, mem_res=1331680, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=196244, cpu_share=0.0, mem_res=151364, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44840, cpu_share=0.01, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=17708304, free=4470932, buffers=844152, cached=10107220, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.44, five_min_avg=3.78, fifteen_min_avg=3.37, cpu_share=3.88, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 01:49:51.255 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 01:49:51.694 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 01:49:52.828 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 01:49:58.263 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=55564720, partition_space_available_1k=25525764, percentage_partition_space_used=69, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1347692, cpu_share=0.13, mem_res=1333956, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=198860, cpu_share=0.0, mem_res=153900, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44848, cpu_share=0.01, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=18646808, free=3443284, buffers=847484, cached=10192988, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=11.27, five_min_avg=5.7, fifteen_min_avg=4.04, cpu_share=6.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 01:50:00.658 7fa203c4b040 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:50:05.581 7fa203c4b040 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:50:11.924 7fa2032efcc0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:50:51.176 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 01:50:51.579 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 01:50:52.462 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 01:50:58.262 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=55592328, partition_space_available_1k=25498156, percentage_partition_space_used=69, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1347928, cpu_share=0.16, mem_res=1334020, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=217760, cpu_share=0.02, mem_res=172728, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44848, cpu_share=0.01, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=18721324, free=3343024, buffers=848272, cached=10217488, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.69, five_min_avg=5.41, fifteen_min_avg=4.05, cpu_share=4.88, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 01:51:51.193 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 01:51:51.617 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 01:51:52.463 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 01:51:58.282 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=55603676, partition_space_available_1k=25486808, percentage_partition_space_used=69, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1351552, cpu_share=0.1, mem_res=1337224, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=220004, cpu_share=0.0, mem_res=174948, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44860, cpu_share=0.01, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=18385308, free=3652536, buffers=848316, cached=10243380, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.9, five_min_avg=5.47, fifteen_min_avg=4.15, cpu_share=4.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 01:52:51.182 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 01:52:51.561 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 01:52:52.421 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 01:52:58.282 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=55625296, partition_space_available_1k=25465188, percentage_partition_space_used=69, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1351680, cpu_share=0.14, mem_res=1337200, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=252680, cpu_share=0.01, mem_res=207520, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44864, cpu_share=0.01, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=18455240, free=3550164, buffers=848332, cached=10275412, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.39, five_min_avg=5.9, fifteen_min_avg=4.4, cpu_share=4.88, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 01:53:51.211 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 01:53:51.645 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 01:53:52.440 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 01:53:58.277 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=55640244, partition_space_available_1k=25450240, percentage_partition_space_used=69, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1355692, cpu_share=0.11, mem_res=1340512, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=238212, cpu_share=0.0, mem_res=193040, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44860, cpu_share=0.01, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=18968160, free=3018024, buffers=848360, cached=10293532, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.38, five_min_avg=5.68, fifteen_min_avg=4.42, cpu_share=4.88, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 01:54:51.378 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 01:54:52.882 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 01:54:57.843 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 01:55:03.389 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=55717392, partition_space_available_1k=25373092, percentage_partition_space_used=69, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1355852, cpu_share=0.09, mem_res=1340536, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=240276, cpu_share=0.0, mem_res=195088, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44868, cpu_share=0.01, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=19660980, free=2276908, buffers=848388, cached=10342296, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=11.01, five_min_avg=7.08, fifteen_min_avg=4.98, cpu_share=6.38, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 01:55:17.651 7fa2032efd60 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:55:22.922 7fa2032efd60 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:55:28.425 7fa2032efd60 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:55:51.200 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 01:55:51.576 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 01:55:52.376 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 01:55:58.291 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=55705456, partition_space_available_1k=25385028, percentage_partition_space_used=69, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1356000, cpu_share=0.12, mem_res=1340468, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=242796, cpu_share=0.01, mem_res=197584, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44872, cpu_share=0.01, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=18770812, free=3146496, buffers=848808, cached=10362348, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.5, five_min_avg=6.5, fifteen_min_avg=4.9, cpu_share=5.38, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 01:56:51.225 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 01:56:51.791 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 01:56:52.716 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 01:56:58.305 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=55721172, partition_space_available_1k=25369312, percentage_partition_space_used=69, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1356144, cpu_share=0.07, mem_res=1340540, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=244000, cpu_share=0.0, mem_res=198776, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44876, cpu_share=0.01, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=18918200, free=2979440, buffers=848812, cached=10381968, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.13, five_min_avg=6.09, fifteen_min_avg=4.86, cpu_share=4.62, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 01:57:13.710 7fa203c4b040 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 01:57:51.200 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 01:57:51.535 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 01:57:52.197 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 01:57:57.303 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=55731460, partition_space_available_1k=25359024, percentage_partition_space_used=69, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1373460, cpu_share=0.09, mem_res=1357796, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=245776, cpu_share=0.0, mem_res=200540, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44884, cpu_share=0.01, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=18893964, free=2990020, buffers=848864, cached=10394464, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.55, five_min_avg=5.78, fifteen_min_avg=4.83, cpu_share=4.38, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 01:58:51.190 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 01:58:51.524 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 01:58:52.253 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 01:58:58.301 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=55741368, partition_space_available_1k=25349116, percentage_partition_space_used=69, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1374844, cpu_share=0.06, mem_res=1358580, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=247356, cpu_share=0.01, mem_res=202104, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44892, cpu_share=0.01, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=18594460, free=3275064, buffers=848868, cached=10408028, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.54, five_min_avg=5.86, fifteen_min_avg=4.92, cpu_share=4.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 01:59:51.189 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 01:59:51.522 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 01:59:52.209 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 01:59:57.304 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=55751128, partition_space_available_1k=25339356, percentage_partition_space_used=69, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1374932, cpu_share=0.07, mem_res=1358588, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=248508, cpu_share=0.0, mem_res=203248, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44892, cpu_share=0.01, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=18609660, free=3248072, buffers=848876, cached=10420120, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.01, five_min_avg=5.37, fifteen_min_avg=4.8, cpu_share=3.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 02:00:08.110 7fa203b670e0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 02:00:13.630 7fa203b670e0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 02:00:19.120 7fa202fc0220 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 02:00:51.206 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 02:00:51.548 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 02:00:52.321 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 10/13/2025 02:00:58.311 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-589-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=55780884, partition_space_available_1k=25309600, percentage_partition_space_used=69, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1375136, cpu_share=0.09, mem_res=1358592, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=250636, cpu_share=0.0, mem_res=205356, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44900, cpu_share=0.01, mem_res=42704, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=18415148, free=3407096, buffers=848884, cached=10455680, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=9.74, five_min_avg=6.77, fifteen_min_avg=5.32, cpu_share=6.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 10/13/2025 02:01:51.373 7fa205e5a240 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 10/13/2025 02:01:51.684 7fa202faa2c0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 10/13/2025 02:01:51.944 7fa205e5a240 [contrail-config-database-nodemgr] [WARNING]: CQLSH failed, but Cassandra native transport port is open 10/13/2025 02:01:52.993 7fa205e5a240 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1