05/19/2025 01:42:56.924 7f6f4a00b748 [contrail-config-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 05/19/2025 01:42:56.929 7f6f4a00b748 [contrail-config-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 05/19/2025 01:42:56.929 7f6f4a00b748 [contrail-config-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 05/19/2025 01:42:56.954 7f6f47db66d8 [contrail-config-database-nodemgr] [ERROR]: Cannot write http_port 8112 to /tmp/contrail-config-database-nodemgr.0.http_port 05/19/2025 01:42:56.955 7f6f47db66d8 [contrail-config-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8112 05/19/2025 01:42:56.955 7f6f47dc2048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 05/19/2025 01:42:56.956 7f6f47dc2048 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:42:56.956 7f6f47dc2048 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 05/19/2025 01:42:56.957 7f6f47dc2048 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 05/19/2025 01:42:56.957 7f6f47dc2048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:42:56.957 7f6f47dc2048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 05/19/2025 01:42:56.957 7f6f47dc2048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:42:56.957 7f6f47dc2048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:42:56.957 7f6f47dc2048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1747618976956706 collector_name = collector_ip = collector_list = [ 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 05/19/2025 01:42:56.957 7f6f47dc2048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:42:57.213 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 44531685136361065603867099075920712615850349139585910716993949796104074518574, 'start': '1747618973000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'zookeeper', 'group': 'zookeeper', 'pid': 5978480724145936500072112193726199559913858291674162479530881124323294874881, 'start': '1747618973000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-config-database-nodemgr', 'group': 'contrail-config-database-nodemgr', 'pid': 6337519599715244489258673464906938522264121828223694091371304191562847955742, 'start': '1747618973000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 05/19/2025 01:42:57.434 7f6f502dfa20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:42:57.434 7f6f502dfa20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-05-19 00:43:16.949815", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9196-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 4 num_core_per_socket = 4 num_thread_per_core = 1 >> running_package_version = nightly-nightly.el7 installed_package_version = nightly-nightly.el7 >> 05/19/2025 01:42:57.434 7f6f502dfa20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:42:57.436 7f6f4a00b748 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-k8s-1230-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1747618977435963, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 05/19/2025 01:42:57.471 7f6f4a00b748 [contrail-config-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-k8s-1230-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1747618977471773, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1747618973000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='zookeeper', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1747618973000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-config-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1747618973000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-05-19 00:43:16.949815", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9196-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 05/19/2025 01:42:57.474 7f6f4787c438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:42:57.474 7f6f4787c438 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 05/19/2025 01:42:57.474 7f6f4787c438 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:42:57.474 7f6f4787c438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:42:57.474 7f6f4787c438 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1747618973000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = zookeeper process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1747618973000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-config-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1747618973000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-05-19 00:43:16.949815", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9196-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 05/19/2025 01:42:57.474 7f6f4787c438 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:42:57.773 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 01:42:58.564 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:DatabaseUsage] map 05/19/2025 01:42:58.566 7f6f499dc2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:42:58.566 7f6f499dc2e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 database_usage = [ << disk_space_used_1k = 10925420 disk_space_available_1k = 70165064 config_db_size_1k = 4 >>, ] >> 05/19/2025 01:42:58.566 7f6f499dc2e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:00.148 7f6f4a00b748 [contrail-config-database-nodemgr] [ERROR]: Failed to connect to database by CQL: Result: 1 05/19/2025 01:43:00.148 7f6f4a00b748 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-k8s-1230-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1747618980148901, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 05/19/2025 01:43:00.150 7f6f479b9f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:00.150 7f6f479b9f60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 05/19/2025 01:43:00.150 7f6f479b9f60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:00.957 7f6f47dcf048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 05/19/2025 01:43:00.959 7f6f47dcf048 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 05/19/2025 01:43:00.959 7f6f47dcf048 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 05/19/2025 01:43:00.959 7f6f47dcf048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 05/19/2025 01:43:00.959 7f6f47dcf048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 05/19/2025 01:43:00.959 7f6f47dcf048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 05/19/2025 01:43:00.959 7f6f47dcf048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 05/19/2025 01:43:00.960 7f6f47dcf048 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 05/19/2025 01:43:00.960 7f6f47dcf048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:00.960 7f6f47dcf048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 05/19/2025 01:43:00.960 7f6f47dcf048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:00.960 7f6f47dcf048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:00.960 7f6f47dcf048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1747618976956706 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 05/19/2025 01:43:00.961 7f6f47dcf048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:01.279 7f6f47dcf048 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:43:03.204 7f6f4a00b748 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 2 05/19/2025 01:43:04.887 7f6f4a00b748 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr config_db_uuid'. Error is: Result: 1 05/19/2025 01:43:04.959 7f6f47dcf048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 05/19/2025 01:43:04.960 7f6f47dcf048 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 05/19/2025 01:43:04.962 7f6f47dcf048 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 05/19/2025 01:43:04.963 7f6f47dcf048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 05/19/2025 01:43:04.963 7f6f47dcf048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 05/19/2025 01:43:04.963 7f6f47dcf048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 05/19/2025 01:43:04.963 7f6f47dcf048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 05/19/2025 01:43:04.963 7f6f47dcf048 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 05/19/2025 01:43:04.963 7f6f47dcf048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:04.964 7f6f47dcf048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 05/19/2025 01:43:04.964 7f6f47dcf048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:04.964 7f6f47dcf048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:04.964 7f6f47dcf048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1747618976956706 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 05/19/2025 01:43:04.964 7f6f47dcf048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:06.289 7f6f4a00b748 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr to_bgp_keyspace'. Error is: Result: 1 05/19/2025 01:43:07.412 7f6f4a00b748 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr svc_monitor_keyspace'. Error is: Result: 1 05/19/2025 01:43:08.480 7f6f4a00b748 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr DISCOVERY_SERVER'. Error is: Result: 1 05/19/2025 01:43:08.961 7f6f4657ce10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 05/19/2025 01:43:08.963 7f6f4657ce10 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 05/19/2025 01:43:08.963 7f6f4657ce10 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 05/19/2025 01:43:08.963 7f6f4657ce10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 05/19/2025 01:43:08.963 7f6f4657ce10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 05/19/2025 01:43:08.963 7f6f4657ce10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 05/19/2025 01:43:08.963 7f6f4657ce10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 05/19/2025 01:43:08.964 7f6f4657ce10 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 05/19/2025 01:43:08.964 7f6f4657ce10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:08.964 7f6f4657ce10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 05/19/2025 01:43:08.964 7f6f4657ce10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:08.964 7f6f4657ce10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:08.965 7f6f4657ce10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1747618976956706 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 05/19/2025 01:43:08.965 7f6f4657ce10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:09.391 7f6f4a00b748 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr useragent'. Error is: Result: 1 05/19/2025 01:43:10.271 7f6f4a00b748 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr dm_keyspace'. Error is: Result: 1 05/19/2025 01:43:12.964 7f6f4656ba20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 05/19/2025 01:43:12.965 7f6f4656ba20 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 05/19/2025 01:43:12.966 7f6f4656ba20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 05/19/2025 01:43:12.966 7f6f4656ba20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 05/19/2025 01:43:12.966 7f6f4656ba20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 05/19/2025 01:43:12.966 7f6f4656ba20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 05/19/2025 01:43:12.966 7f6f4656ba20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 05/19/2025 01:43:12.966 7f6f4656ba20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 05/19/2025 01:43:12.966 7f6f4656ba20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:12.967 7f6f4656ba20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 05/19/2025 01:43:12.967 7f6f4656ba20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:12.967 7f6f4656ba20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:12.967 7f6f4656ba20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1747618976956706 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 05/19/2025 01:43:12.967 7f6f4656ba20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:15.300 7f6f4a00b748 [contrail-config-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-k8s-1230-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1747618995300000, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1747618973000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='zookeeper', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1747618973000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-config-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1747618973000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-05-19 00:43:16.949815", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9196-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 05/19/2025 01:43:15.300 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10939028, partition_space_available_1k=70151456, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1225660, cpu_share=0, mem_res=1211460, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=50608, cpu_share=0, mem_res=46456, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53300, cpu_share=0, mem_res=39184, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=3368840, free=709384, buffers=486108, cached=11619876, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.71, five_min_avg=1.34, fifteen_min_avg=0.66, cpu_share=0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 01:43:15.300 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:15.300 7f6f4787cd68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1747618973000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = zookeeper process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1747618973000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-config-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1747618973000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-05-19 00:43:16.949815", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9196-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 05/19/2025 01:43:15.300 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:15.301 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:15.301 7f6f4787cd68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 10939028 partition_space_available_1k = 70151456 percentage_partition_space_used = 13 >> } all_core_file_list = [ ] process_mem_cpu_usage = { cassandra : << mem_virt = 1225660 cpu_share = 0 mem_res = 1211460 >>zookeeper : << mem_virt = 50608 cpu_share = 0 mem_res = 46456 >>contrail-config-database-nodemgr : << mem_virt = 53300 cpu_share = 0 mem_res = 39184 >> } system_mem_usage = << total = 16184208 used = 3368840 free = 709384 buffers = 486108 cached = 11619876 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 2.71 five_min_avg = 1.34 fifteen_min_avg = 0.66 cpu_share = 0 node_type = config-database-node >> >> 05/19/2025 01:43:15.301 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:16.966 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 05/19/2025 01:43:16.968 7f6f4787cd68 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 05/19/2025 01:43:16.968 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 05/19/2025 01:43:16.968 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 05/19/2025 01:43:16.968 7f6f4787cd68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 05/19/2025 01:43:16.968 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 05/19/2025 01:43:16.968 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 05/19/2025 01:43:16.969 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 05/19/2025 01:43:16.969 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:16.969 7f6f4787cd68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 05/19/2025 01:43:16.969 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:16.969 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:16.969 7f6f4787cd68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1747618976956706 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 05/19/2025 01:43:16.969 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:20.972 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 05/19/2025 01:43:20.973 7f6f4787cd68 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 05/19/2025 01:43:20.973 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 05/19/2025 01:43:20.973 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 05/19/2025 01:43:20.974 7f6f4787cd68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 05/19/2025 01:43:20.974 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 05/19/2025 01:43:20.974 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 05/19/2025 01:43:20.974 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 05/19/2025 01:43:20.974 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:20.974 7f6f4787cd68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 05/19/2025 01:43:20.974 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:20.974 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:20.975 7f6f4787cd68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1747618976956706 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 05/19/2025 01:43:20.975 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:24.975 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 05/19/2025 01:43:24.976 7f6f4787cd68 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 05/19/2025 01:43:24.977 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 05/19/2025 01:43:24.977 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 05/19/2025 01:43:24.977 7f6f4787cd68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 05/19/2025 01:43:24.977 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 05/19/2025 01:43:24.977 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 05/19/2025 01:43:24.977 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 05/19/2025 01:43:24.978 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:24.978 7f6f4787cd68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 05/19/2025 01:43:24.978 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:24.978 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:24.978 7f6f4787cd68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1747618976956706 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 05/19/2025 01:43:24.978 7f6f4787cd68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:28.978 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 05/19/2025 01:43:28.979 7f6f479b9390 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 05/19/2025 01:43:28.979 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 05/19/2025 01:43:28.979 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 05/19/2025 01:43:28.979 7f6f479b9390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 05/19/2025 01:43:28.979 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 05/19/2025 01:43:28.980 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 05/19/2025 01:43:28.980 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 05/19/2025 01:43:28.980 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:28.980 7f6f479b9390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 05/19/2025 01:43:28.980 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:28.981 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:28.981 7f6f479b9390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1747618976956706 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 05/19/2025 01:43:28.981 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:31.343 7f6f479b9390 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:43:32.981 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 05/19/2025 01:43:32.982 7f6f479b9390 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 05/19/2025 01:43:32.983 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 05/19/2025 01:43:32.983 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 05/19/2025 01:43:32.983 7f6f479b9390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 05/19/2025 01:43:32.983 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 05/19/2025 01:43:32.983 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 05/19/2025 01:43:32.984 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 05/19/2025 01:43:32.984 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:32.985 7f6f479b9390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 05/19/2025 01:43:32.985 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:32.985 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:32.985 7f6f479b9390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1747618976956706 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 05/19/2025 01:43:32.985 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/19/2025 01:43:36.983 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 05/19/2025 01:43:36.984 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Session Event: TCP Connected 05/19/2025 01:43:36.985 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 05/19/2025 01:43:36.985 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 05/19/2025 01:43:36.985 7f6f479b9390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1230-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 05/19/2025 01:43:36.985 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 05/19/2025 01:43:36.985 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 05/19/2025 01:43:36.985 7f6f479b9390 [contrail-config-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 05/19/2025 01:43:36.986 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 05/19/2025 01:43:36.986 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 05/19/2025 01:43:36.986 7f6f479b9390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 05/19/2025 01:43:37.935 7f6f45dc54e0 [contrail-config-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001013 05/19/2025 01:43:37.936 7f6f45dc54e0 [contrail-config-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 05/19/2025 01:43:37.936 7f6f46572b70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 05/19/2025 01:43:37.936 7f6f46572b70 [contrail-config-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 05/19/2025 01:43:37.936 7f6f46572b70 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:DatabaseUsage] map 05/19/2025 01:43:37.936 7f6f46572b70 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:43:37.936 7f6f46572b70 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 05/19/2025 01:43:37.937 7f6f46572b70 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 05/19/2025 01:43:37.937 7f6f45dc5a20 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 05/19/2025 01:43:37.938 7f6f45dc5a20 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 05/19/2025 01:43:49.161 7f6f479d8780 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:43:58.342 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 01:44:01.051 7f6f4a00b748 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-k8s-1230-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1747619041051472, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.57:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 05/19/2025 01:44:01.529 7f6f45dc54e0 [contrail-config-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001435 05/19/2025 01:44:01.530 7f6f45dc54e0 [contrail-config-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 05/19/2025 01:44:01.531 7f6f45dd38d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 05/19/2025 01:44:01.537 7f6f45dd38d0 [contrail-config-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 3 05/19/2025 01:44:01.538 7f6f45dd38d0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:DatabaseUsage] map 05/19/2025 01:44:01.540 7f6f45dd38d0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:44:01.552 7f6f45dd38d0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 05/19/2025 01:44:01.554 7f6f45dd3198 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 05/19/2025 01:44:01.558 7f6f45dd3198 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 05/19/2025 01:44:02.298 7f6f45dd3198 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:44:16.603 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-k8s-1230-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1747619056603456, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectConfigDatabaseInfo'), _seqnum=0) 05/19/2025 01:44:16.603 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:CassandraStatusUVE] map 05/19/2025 01:44:17.136 7f6f49cbc588 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:44:22.438 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11494776, partition_space_available_1k=69595708, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1310584, cpu_share=9.0, mem_res=1293452, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=163892, cpu_share=2.0, mem_res=158632, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52368, cpu_share=0.25, mem_res=39332, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=6424040, free=406524, buffers=416924, cached=8936720, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=12.07, five_min_avg=4.23, fifteen_min_avg=1.7, cpu_share=19.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 01:44:57.746 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 01:45:07.531 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11517888, partition_space_available_1k=69572596, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1305224, cpu_share=5.5, mem_res=1288060, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=164100, cpu_share=0.0, mem_res=158764, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52588, cpu_share=0.25, mem_res=39376, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=6521576, free=343912, buffers=417464, cached=8901256, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.45, five_min_avg=3.85, fifteen_min_avg=1.69, cpu_share=11.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 01:45:57.788 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 01:46:07.557 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11525420, partition_space_available_1k=69565064, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1301156, cpu_share=1.25, mem_res=1284388, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=164100, cpu_share=0.0, mem_res=158764, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52884, cpu_share=0.25, mem_res=39824, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=6516100, free=343368, buffers=426400, cached=8898340, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.5, five_min_avg=3.47, fifteen_min_avg=1.69, cpu_share=11.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 01:46:28.963 7f6f4aa35ac8 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:46:55.469 7f6f46570d68 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:46:57.787 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 01:47:02.139 7f6f46570eb8 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:47:05.158 7f6f45dd3780 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:47:08.095 7f6f47d474e0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:47:08.557 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11652140, partition_space_available_1k=69438344, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1302112, cpu_share=3.0, mem_res=1284944, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=164228, cpu_share=0.0, mem_res=159028, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52888, cpu_share=0.25, mem_res=39788, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=6662536, free=348192, buffers=419108, cached=8754372, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=8.63, five_min_avg=4.87, fifteen_min_avg=2.28, cpu_share=13.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 01:47:46.047 7f6f4aa42630 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:47:48.890 7f6f4aa42630 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:47:53.354 7f6f4aa42630 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:47:57.750 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 01:48:06.558 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11561996, partition_space_available_1k=69528488, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1336804, cpu_share=1.5, mem_res=1319280, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=164228, cpu_share=0.0, mem_res=159028, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53120, cpu_share=0.25, mem_res=39876, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=6638636, free=458152, buffers=419556, cached=8667864, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.73, five_min_avg=4.39, fifteen_min_avg=2.27, cpu_share=9.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 01:48:57.759 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 01:49:07.562 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11569764, partition_space_available_1k=69520720, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1305184, cpu_share=1.25, mem_res=1287256, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=164232, cpu_share=0.0, mem_res=159028, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53208, cpu_share=0.25, mem_res=39888, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=6647940, free=439684, buffers=419672, cached=8676912, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.51, five_min_avg=3.81, fifteen_min_avg=2.21, cpu_share=8.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 01:49:57.775 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 01:50:07.570 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11576656, partition_space_available_1k=69513828, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1305052, cpu_share=1.25, mem_res=1287204, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=167076, cpu_share=0.0, mem_res=161800, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53176, cpu_share=0.25, mem_res=39876, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=6655848, free=423232, buffers=419740, cached=8685388, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.88, five_min_avg=3.38, fifteen_min_avg=2.16, cpu_share=8.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 01:50:57.841 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 01:51:08.592 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11585320, partition_space_available_1k=69505164, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1305076, cpu_share=1.5, mem_res=1286996, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=167120, cpu_share=0.0, mem_res=161800, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53168, cpu_share=0.25, mem_res=39812, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=6697552, free=364384, buffers=421252, cached=8701020, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.33, five_min_avg=3.19, fifteen_min_avg=2.17, cpu_share=10.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 01:51:20.897 7f6f45984630 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:51:57.862 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 01:52:07.593 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11594188, partition_space_available_1k=69496296, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1324956, cpu_share=1.5, mem_res=1306876, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=167160, cpu_share=0.0, mem_res=161800, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53324, cpu_share=0.25, mem_res=39900, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=6709852, free=390584, buffers=419256, cached=8664516, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.97, five_min_avg=2.98, fifteen_min_avg=2.16, cpu_share=10.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 01:52:13.422 7f6f47dcfa20 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:52:16.236 7f6f47dcfa20 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:52:20.601 7f6f49d7b4e0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:52:57.810 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 01:53:06.613 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11605604, partition_space_available_1k=69484880, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1322720, cpu_share=1.5, mem_res=1304656, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=167200, cpu_share=0.0, mem_res=161932, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53388, cpu_share=0.25, mem_res=40248, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=6742528, free=336224, buffers=420760, cached=8684696, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.9, five_min_avg=2.84, fifteen_min_avg=2.16, cpu_share=9.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 01:53:57.831 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 01:54:06.608 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11612828, partition_space_available_1k=69477656, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1307620, cpu_share=1.25, mem_res=1289020, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=167200, cpu_share=0.0, mem_res=161932, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53304, cpu_share=0.25, mem_res=39940, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=6717720, free=351836, buffers=420808, cached=8693844, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.69, five_min_avg=2.6, fifteen_min_avg=2.12, cpu_share=8.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 01:54:57.824 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 01:55:06.625 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11615092, partition_space_available_1k=69475392, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1306884, cpu_share=1.25, mem_res=1288088, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=167200, cpu_share=0.0, mem_res=161932, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53304, cpu_share=0.25, mem_res=39976, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=6715960, free=349668, buffers=420892, cached=8697688, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.48, five_min_avg=2.37, fifteen_min_avg=2.07, cpu_share=8.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 01:55:57.874 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 01:56:07.618 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11618768, partition_space_available_1k=69471716, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1308340, cpu_share=1.5, mem_res=1289696, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=167200, cpu_share=0.0, mem_res=161932, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53244, cpu_share=0.25, mem_res=40032, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=6731688, free=321584, buffers=421520, cached=8709416, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.58, five_min_avg=2.54, fifteen_min_avg=2.15, cpu_share=10.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 01:56:21.348 7f6f4f26ef60 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:56:24.233 7f6f4f26ef60 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:56:28.715 7f6f4f26ef60 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:56:41.978 7f6f47a53390 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 01:56:57.833 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 01:57:07.636 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11625860, partition_space_available_1k=69464624, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1308584, cpu_share=1.25, mem_res=1289860, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=167208, cpu_share=0.0, mem_res=161932, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53272, cpu_share=0.25, mem_res=40100, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=6747032, free=359904, buffers=418176, cached=8659096, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.19, five_min_avg=2.45, fifteen_min_avg=2.14, cpu_share=10.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 01:57:57.890 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 01:58:08.659 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12730168, partition_space_available_1k=68360316, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1306196, cpu_share=1.25, mem_res=1288200, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=167208, cpu_share=0.0, mem_res=161932, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52732, cpu_share=0.25, mem_res=40324, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=6761212, free=555628, buffers=427588, cached=8439780, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.6, five_min_avg=2.48, fifteen_min_avg=2.17, cpu_share=11.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 01:58:57.853 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 01:59:07.631 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12787220, partition_space_available_1k=68303264, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1309692, cpu_share=2.5, mem_res=1290516, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=167216, cpu_share=0.0, mem_res=161932, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52724, cpu_share=0.25, mem_res=40156, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=6942872, free=322708, buffers=436768, cached=8481860, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.49, five_min_avg=2.72, fifteen_min_avg=2.27, cpu_share=12.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 01:59:57.876 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 02:00:07.656 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12800016, partition_space_available_1k=68290468, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1310088, cpu_share=3.75, mem_res=1290436, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=167284, cpu_share=0.0, mem_res=161932, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52796, cpu_share=0.25, mem_res=39980, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=6955280, free=340484, buffers=435456, cached=8452988, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.87, five_min_avg=3.78, fifteen_min_avg=2.66, cpu_share=11.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 02:00:57.919 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 02:01:07.679 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12806224, partition_space_available_1k=68284260, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1309872, cpu_share=1.5, mem_res=1290380, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=167292, cpu_share=0.0, mem_res=161932, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52956, cpu_share=0.25, mem_res=40052, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7013052, free=364280, buffers=440160, cached=8366716, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.23, five_min_avg=3.96, fifteen_min_avg=2.8, cpu_share=9.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 02:01:37.543 7f6f479fb240 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 02:01:57.866 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 02:02:04.987 7f6f49d19780 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 02:02:07.665 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12810176, partition_space_available_1k=68280308, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1310468, cpu_share=1.25, mem_res=1290876, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=167292, cpu_share=0.0, mem_res=161932, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52852, cpu_share=0.25, mem_res=40064, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7072768, free=275696, buffers=453180, cached=8382564, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.77, five_min_avg=4.51, fifteen_min_avg=3.07, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 02:02:07.970 7f6f49e4d780 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 02:02:12.725 7f6f49b4f978 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 02:02:57.866 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 02:03:07.671 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12817452, partition_space_available_1k=68273032, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1314052, cpu_share=1.5, mem_res=1294284, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=167292, cpu_share=0.0, mem_res=161932, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52884, cpu_share=0.25, mem_res=40016, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=6999960, free=340692, buffers=453900, cached=8389656, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.41, five_min_avg=4.08, fifteen_min_avg=3.01, cpu_share=9.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 02:03:57.909 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 02:04:10.662 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12825440, partition_space_available_1k=68265044, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1311656, cpu_share=2.25, mem_res=1291280, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=167300, cpu_share=0.0, mem_res=161932, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52600, cpu_share=0.25, mem_res=40052, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=6889572, free=439932, buffers=454004, cached=8400700, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.85, five_min_avg=3.99, fifteen_min_avg=3.05, cpu_share=11.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 02:04:57.887 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 02:05:08.676 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=13198768, partition_space_available_1k=67891716, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1315572, cpu_share=1.5, mem_res=1295528, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=167300, cpu_share=0.0, mem_res=161932, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52664, cpu_share=0.25, mem_res=40100, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7035840, free=344092, buffers=456720, cached=8347556, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.7, five_min_avg=4.14, fifteen_min_avg=3.16, cpu_share=11.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 02:05:12.688 7f6f47db6d68 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 02:05:39.123 7f6f47dc2438 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 02:05:57.886 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 02:06:05.640 7f6f49b4f978 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 02:06:09.690 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=13201752, partition_space_available_1k=67888732, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1311568, cpu_share=2.0, mem_res=1291388, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=167300, cpu_share=0.0, mem_res=161932, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52664, cpu_share=0.25, mem_res=39940, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=6990260, free=336020, buffers=461884, cached=8396044, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.85, five_min_avg=4.15, fifteen_min_avg=3.21, cpu_share=12.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 02:06:10.681 7f6f47dcfd68 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 02:06:13.887 7f6f47ddf438 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 02:06:40.929 7f6f47dc2438 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 02:06:44.481 7f6f4657b978 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 05/19/2025 02:06:58.022 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 05/19/2025 02:07:08.687 7f6f4a00b748 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1230-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=13220816, partition_space_available_1k=67869668, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1313576, cpu_share=1.5, mem_res=1293400, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=167304, cpu_share=0.0, mem_res=161932, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52668, cpu_share=0.25, mem_res=39940, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=6952736, free=440352, buffers=460528, cached=8330592, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.06, five_min_avg=4.29, fifteen_min_avg=3.32, cpu_share=12.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 05/19/2025 02:07:53.672 7f6f478c3048 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map