03/20/2025 00:48:36.127 7f0928182df0 [contrail-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 03/20/2025 00:48:36.127 7f0928182df0 [contrail-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 03/20/2025 00:48:36.128 7f0928182df0 [contrail-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 03/20/2025 00:48:36.135 7f092638f148 [contrail-database-nodemgr] [ERROR]: Cannot write http_port 8103 to /tmp/contrail-database-nodemgr.0.http_port 03/20/2025 00:48:36.135 7f092638f148 [contrail-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8103 03/20/2025 00:48:36.135 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 03/20/2025 00:48:36.136 7f092297c648 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/20/2025 00:48:36.136 7f092297c648 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/20/2025 00:48:36.137 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 03/20/2025 00:48:36.137 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:36.137 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/20/2025 00:48:36.137 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:36.137 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:36.137 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1742431716136789 collector_name = collector_ip = collector_list = [ 10.0.0.42:8086, 10.0.0.12:8086, 10.0.0.47:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 03/20/2025 00:48:36.137 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:36.221 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 21549857983011406121701510459059716657055180939097715056832767013973607837582, 'start': '1742431714000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-query-engine', 'group': 'contrail-query-engine', 'pid': 29653354915007651319541940775083343926056115146295114853271820085147009555723, 'start': '1742431714000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-database-nodemgr', 'group': 'contrail-database-nodemgr', 'pid': 53670892759447302133743699190302748841256872271881017739411426250531946226262, 'start': '1742431714000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 03/20/2025 00:48:36.352 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:36.353 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. build_info = {"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-17 01:12:14.132392", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8139-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 4 num_core_per_socket = 4 num_thread_per_core = 1 >> running_package_version = nightly.r24.1.centos-nightly_r24.1_centos.el7 installed_package_version = nightly.r24.1.centos-nightly_r24.1_centos.el7 >> 03/20/2025 00:48:36.353 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:36.353 7f0928182df0 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742431716353887, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 03/20/2025 00:48:36.354 7f0928182df0 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742431716354188, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742431714000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742431714000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742431714000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-17 01:12:14.132392", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8139-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 03/20/2025 00:48:36.356 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:36.356 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/20/2025 00:48:36.356 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:36.356 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:36.356 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742431714000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742431714000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742431714000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-17 01:12:14.132392", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8139-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]} >> 03/20/2025 00:48:36.356 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:36.618 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 00:48:36.884 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:DatabaseUsage] map 03/20/2025 00:48:36.886 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:36.886 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. database_usage = [ << disk_space_used_1k = 6677124 disk_space_available_1k = 74413360 analytics_db_size_1k = 4 >>, ] >> 03/20/2025 00:48:36.886 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:37.393 7f0928182df0 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL: Result: 1 03/20/2025 00:48:37.393 7f0928182df0 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742431717393203, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 03/20/2025 00:48:37.394 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:37.394 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/20/2025 00:48:37.394 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:40.136 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:48:40.139 7f092297c648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:48:40.139 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:48:40.139 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:40.139 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:48:40.140 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:40.140 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:48:40.141 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:48:40.141 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:40.141 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:48:40.141 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:40.141 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:40.144 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1742431716136789 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.12:8086, 10.0.0.47:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:48:40.145 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:40.232 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742431720232157, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 03/20/2025 00:48:40.232 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 03/20/2025 00:48:40.234 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:40.235 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] CassandraStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. cassandra_compaction_task = << pending_compaction_tasks = 0 >> thread_pool_stats = [ << pool_name = ReadStage active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = CompactionExecutor active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = MutationStage active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = MemtableFlushWriter active = 0 pending = 0 all_time_blocked = 0 >>, ] >> 03/20/2025 00:48:40.235 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:41.352 7f0928182df0 [contrail-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7200 repair -pr ContrailAnalyticsCql'. Error is: Result: 1 03/20/2025 00:48:44.140 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:48:44.141 7f092297c648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:48:44.141 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:48:44.142 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:44.142 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:48:44.142 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:44.142 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:48:44.142 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:48:44.142 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:44.143 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:48:44.143 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:44.143 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:44.143 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1742431716136789 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.12:8086, 10.0.0.47:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:48:44.143 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:46.946 7f0928182df0 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742431726945973, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742431714000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742431714000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742431714000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-17 01:12:14.132392", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8139-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 03/20/2025 00:48:46.946 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=6678260, partition_space_available_1k=74412224, percentage_partition_space_used=8, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1216132, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=7212, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51252, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=4380728, free=5423892, buffers=237656, cached=6133352, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.59, five_min_avg=0.98, fifteen_min_avg=0.48, cpu_share=0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 00:48:46.946 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:46.947 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742431714000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742431714000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742431714000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-17 01:12:14.132392", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8139-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]} >> 03/20/2025 00:48:46.947 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:46.947 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:46.947 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 6678260 partition_space_available_1k = 74412224 percentage_partition_space_used = 8 >> } all_core_file_list = [ ] process_mem_cpu_usage = { cassandra : << mem_virt = 1216132 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 7212 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 51252 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 16175628 used = 4380728 free = 5423892 buffers = 237656 cached = 6133352 node_type = database-node >> system_cpu_usage = << one_min_avg = 1.59 five_min_avg = 0.98 fifteen_min_avg = 0.48 cpu_share = 0 node_type = database-node >> >> 03/20/2025 00:48:46.947 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:48.142 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:48:48.144 7f092297c648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:48:48.144 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:48:48.144 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:48.144 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:48:48.144 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:48.144 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:48:48.145 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:48:48.145 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:48.145 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:48:48.145 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:48.145 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:48.146 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1742431716136789 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.12:8086, 10.0.0.47:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:48:48.146 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:52.144 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:48:52.146 7f092297c648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:48:52.146 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:48:52.146 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:52.146 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:48:52.147 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:52.147 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:48:52.147 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:48:52.147 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:52.147 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:48:52.147 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:52.147 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:52.148 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1742431716136789 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.12:8086, 10.0.0.47:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:48:52.148 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:56.150 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:48:56.151 7f092297c648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:48:56.151 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:48:56.151 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:56.152 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:48:56.152 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:56.152 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:48:56.152 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:48:56.152 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:56.152 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:48:56.152 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:56.153 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:56.153 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1742431716136789 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.12:8086, 10.0.0.47:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:48:56.153 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:00.155 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:49:00.157 7f092297c648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:49:00.157 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:49:00.157 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:00.157 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:49:00.157 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:00.158 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:49:00.158 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:49:00.158 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:00.158 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:49:00.158 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:00.158 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:00.158 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1742431716136789 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.12:8086, 10.0.0.47:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:49:00.159 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:04.161 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:49:04.163 7f092297c648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:49:04.163 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:49:04.163 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:04.163 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:49:04.163 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:04.163 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:49:04.164 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:49:04.164 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:04.165 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:49:04.165 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:04.165 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:04.165 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1742431716136789 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.12:8086, 10.0.0.47:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:49:04.165 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:08.163 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:49:08.165 7f092297c648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:49:08.165 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:49:08.165 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:08.165 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:49:08.166 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:08.166 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:49:08.166 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:49:08.166 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:08.166 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:49:08.166 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:08.166 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:08.167 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1742431716136789 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.12:8086, 10.0.0.47:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:49:08.167 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:12.166 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:49:12.168 7f092297c648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:49:12.168 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:49:12.168 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:12.168 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:49:12.169 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:12.169 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:49:12.169 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:49:12.169 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:12.169 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:49:12.169 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:12.169 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:12.169 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1742431716136789 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.12:8086, 10.0.0.47:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:49:12.170 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:16.171 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:49:16.172 7f092297c648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:49:16.172 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:49:16.173 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:16.173 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:49:16.173 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:16.173 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:49:16.173 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:49:16.173 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:16.174 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:49:16.174 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:16.174 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:16.174 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1742431716136789 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.12:8086, 10.0.0.47:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:49:16.174 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:20.176 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:49:20.178 7f092297c648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:49:20.178 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:49:20.178 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:20.178 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:49:20.178 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:20.178 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:49:20.179 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:49:20.179 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:20.179 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:49:20.179 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:20.179 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:20.179 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1742431716136789 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.12:8086, 10.0.0.47:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:49:20.179 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:24.180 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:49:24.182 7f092297c648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:49:24.182 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:49:24.182 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:24.182 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:49:24.182 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:24.182 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:49:24.183 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:49:24.183 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:24.183 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:49:24.183 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:24.183 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:24.184 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1742431716136789 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.12:8086, 10.0.0.47:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:49:24.184 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:28.184 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:49:28.186 7f092297c648 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 03/20/2025 00:49:28.186 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:49:28.186 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:28.186 7f092297c648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:49:28.186 7f092297c648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:28.186 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 03/20/2025 00:49:28.186 7f092297c648 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 03/20/2025 00:49:28.187 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 03/20/2025 00:49:28.187 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 00:49:28.187 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 00:49:36.608 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 00:49:36.935 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 00:49:37.459 7f0928182df0 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742431777459281, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.42:8086'], status='Initializing', description='Connect to ClientInit on EvTcpConnected')]), flag_infos=None, description='Collector:Collector[Connect to ClientInit on EvTcpConnected] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 03/20/2025 00:49:37.461 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 00:49:38.064 7f0922804148 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001016 03/20/2025 00:49:38.065 7f0922804148 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 03/20/2025 00:49:38.065 7f092297c648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 03/20/2025 00:49:38.066 7f092297c648 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 03/20/2025 00:49:38.066 7f092297c648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 03/20/2025 00:49:38.066 7f092297c648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 03/20/2025 00:49:38.066 7f092297c648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/20/2025 00:49:38.066 7f092297c648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/20/2025 00:49:38.067 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 03/20/2025 00:49:38.067 7f092297ce48 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/20/2025 00:49:38.068 7f092297ce48 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/20/2025 00:49:40.102 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742431780101983, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 03/20/2025 00:49:45.950 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=6990660, partition_space_available_1k=74099824, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1326888, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=9904, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51796, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=4668060, free=4790260, buffers=251072, cached=6466236, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.46, five_min_avg=1.0, fifteen_min_avg=0.52, cpu_share=5.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 00:49:50.644 7f0922804148 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001582 03/20/2025 00:49:50.644 7f0922804148 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 03/20/2025 00:49:50.645 7f092297ce48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 03/20/2025 00:49:50.645 7f092297ce48 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 4 03/20/2025 00:49:50.645 7f092297ce48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 03/20/2025 00:49:50.645 7f092297ce48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 03/20/2025 00:49:50.645 7f092297ce48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/20/2025 00:49:50.646 7f092297ce48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/20/2025 00:49:50.646 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/20/2025 00:49:50.647 7f092297c648 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/20/2025 00:50:36.600 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 00:50:44.962 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7126040, partition_space_available_1k=73964444, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1320716, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10036, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51872, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5618000, free=3662404, buffers=261688, cached=6633536, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.47, five_min_avg=1.38, fifteen_min_avg=0.68, cpu_share=11.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 00:51:36.613 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 00:51:44.981 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7184984, partition_space_available_1k=73905500, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1328832, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10096, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51876, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5621136, free=3607280, buffers=262656, cached=6684556, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.52, five_min_avg=1.56, fifteen_min_avg=0.79, cpu_share=10.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 00:52:16.149 7f092297c648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/20/2025 00:52:36.634 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 00:52:45.995 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7652684, partition_space_available_1k=73437800, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1326920, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10276, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51892, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5661352, free=3062904, buffers=269252, cached=7182120, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.79, five_min_avg=1.79, fifteen_min_avg=0.92, cpu_share=12.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 00:52:59.129 7f092297c648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/20/2025 00:53:36.754 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 00:53:45.996 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7666364, partition_space_available_1k=73424120, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1333048, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10328, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51924, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5711836, free=3001480, buffers=271012, cached=7191300, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.53, five_min_avg=1.88, fifteen_min_avg=1.0, cpu_share=11.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 00:54:36.669 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 00:54:46.004 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7673956, partition_space_available_1k=73416528, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1339812, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10344, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51940, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5783512, free=2918788, buffers=272792, cached=7200536, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.75, five_min_avg=2.33, fifteen_min_avg=1.21, cpu_share=11.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 00:55:36.684 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 00:55:46.014 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7678612, partition_space_available_1k=73411872, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1345464, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10348, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51936, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5828480, free=2877620, buffers=273964, cached=7195564, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.36, five_min_avg=2.49, fifteen_min_avg=1.34, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 00:56:36.689 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 00:56:46.025 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7687040, partition_space_available_1k=73403444, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1349964, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10256, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51960, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5866216, free=2815512, buffers=275064, cached=7218836, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.48, five_min_avg=2.35, fifteen_min_avg=1.36, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 00:57:36.752 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 00:57:46.045 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7695516, partition_space_available_1k=73394968, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1354460, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10308, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51992, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5898512, free=2777712, buffers=275628, cached=7223776, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.02, five_min_avg=2.54, fifteen_min_avg=1.49, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 00:58:36.723 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 00:58:46.047 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7700524, partition_space_available_1k=73389960, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1364200, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10388, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52008, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5906508, free=2758152, buffers=276096, cached=7234872, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.83, five_min_avg=2.29, fifteen_min_avg=1.47, cpu_share=10.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 00:59:36.728 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 00:59:46.046 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7716276, partition_space_available_1k=73374208, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1373808, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10512, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52012, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5959664, free=2698824, buffers=276388, cached=7240752, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.6, five_min_avg=2.6, fifteen_min_avg=1.62, cpu_share=10.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:00:36.795 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:00:46.069 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7721660, partition_space_available_1k=73368824, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1374216, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10516, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52016, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5990580, free=2652404, buffers=276600, cached=7256044, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.95, five_min_avg=2.59, fifteen_min_avg=1.68, cpu_share=11.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:01:36.799 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:01:46.075 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7730768, partition_space_available_1k=73359716, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1379132, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10536, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52024, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6030712, free=2607124, buffers=276840, cached=7260952, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.09, five_min_avg=2.72, fifteen_min_avg=1.79, cpu_share=11.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:02:36.789 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:02:46.093 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7739976, partition_space_available_1k=73350508, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1384760, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10552, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52020, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6079100, free=2543456, buffers=276956, cached=7276116, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.53, five_min_avg=2.61, fifteen_min_avg=1.81, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:03:36.783 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:03:46.077 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7749592, partition_space_available_1k=73340892, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1398392, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10600, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52032, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6100572, free=2517064, buffers=277076, cached=7280916, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.68, five_min_avg=2.62, fifteen_min_avg=1.87, cpu_share=11.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:04:36.749 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:04:46.083 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7758824, partition_space_available_1k=73331660, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1394740, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10644, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52044, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6154284, free=2453296, buffers=277108, cached=7290940, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.64, five_min_avg=2.61, fifteen_min_avg=1.91, cpu_share=11.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:05:36.884 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:05:46.092 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7768448, partition_space_available_1k=73322036, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1398316, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10660, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52044, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6179808, free=2417528, buffers=277144, cached=7301148, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.81, five_min_avg=2.65, fifteen_min_avg=1.97, cpu_share=11.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:06:36.780 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:06:46.120 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7775892, partition_space_available_1k=73314592, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1404428, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10668, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52044, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6199012, free=2395276, buffers=277164, cached=7304176, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.33, five_min_avg=2.56, fifteen_min_avg=1.98, cpu_share=11.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:07:36.785 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:07:46.102 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7781876, partition_space_available_1k=73308608, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1413312, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10732, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52048, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6180392, free=2403236, buffers=277176, cached=7314824, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.29, five_min_avg=2.5, fifteen_min_avg=2.0, cpu_share=11.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:08:36.837 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:08:46.096 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7785804, partition_space_available_1k=73304680, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1421496, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10684, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52048, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6202480, free=2371908, buffers=277196, cached=7324044, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.44, five_min_avg=2.51, fifteen_min_avg=2.04, cpu_share=11.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:09:36.771 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:09:46.101 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7802064, partition_space_available_1k=73288420, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1426520, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10732, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52052, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6215836, free=2356128, buffers=277228, cached=7326436, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.88, five_min_avg=2.58, fifteen_min_avg=2.09, cpu_share=11.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:10:36.785 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:10:46.108 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7783348, partition_space_available_1k=73307136, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1427160, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10776, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52056, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6226004, free=2350616, buffers=277228, cached=7321780, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.4, five_min_avg=2.52, fifteen_min_avg=2.1, cpu_share=11.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:11:36.798 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:11:46.105 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7791544, partition_space_available_1k=73298940, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1431296, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10784, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52056, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6245284, free=2328004, buffers=277252, cached=7325088, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.64, five_min_avg=2.84, fifteen_min_avg=2.23, cpu_share=11.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:12:36.796 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:12:46.119 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7801432, partition_space_available_1k=73289052, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1435800, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10792, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52064, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6243344, free=2324384, buffers=277260, cached=7330640, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.45, five_min_avg=2.63, fifteen_min_avg=2.2, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:13:36.749 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:13:46.109 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7804884, partition_space_available_1k=73285600, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1442868, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10808, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52064, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6229208, free=2324432, buffers=277264, cached=7344724, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.45, five_min_avg=2.6, fifteen_min_avg=2.21, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:14:36.809 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:14:46.104 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7811092, partition_space_available_1k=73279392, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1456292, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10716, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52068, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6258560, free=2288832, buffers=277280, cached=7350956, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.63, five_min_avg=2.62, fifteen_min_avg=2.24, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:15:36.773 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:15:46.107 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7820456, partition_space_available_1k=73270028, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1449280, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10736, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52072, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6257544, free=2284724, buffers=277284, cached=7356076, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.62, five_min_avg=2.61, fifteen_min_avg=2.26, cpu_share=10.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:16:36.764 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:16:46.115 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7825760, partition_space_available_1k=73264724, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1456584, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10756, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52072, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6263776, free=2274296, buffers=277288, cached=7360268, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.88, five_min_avg=2.66, fifteen_min_avg=2.3, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:17:36.823 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:17:46.135 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7828800, partition_space_available_1k=73261684, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1458360, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10780, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52448, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6270300, free=2263876, buffers=277304, cached=7364148, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.39, five_min_avg=2.55, fifteen_min_avg=2.29, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:18:36.815 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:18:46.132 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7837924, partition_space_available_1k=73252560, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1468160, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10792, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52456, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6285476, free=2240084, buffers=277308, cached=7372760, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.2, five_min_avg=2.43, fifteen_min_avg=2.26, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:19:36.806 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:19:46.144 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7849768, partition_space_available_1k=73240716, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1469980, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10796, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52480, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6287664, free=2230380, buffers=277312, cached=7380272, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.45, five_min_avg=2.45, fifteen_min_avg=2.28, cpu_share=11.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:20:36.833 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:20:46.148 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7852536, partition_space_available_1k=73237948, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1474680, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10732, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52476, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6275248, free=2234348, buffers=277316, cached=7388716, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.4, five_min_avg=2.73, fifteen_min_avg=2.38, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:21:36.897 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:21:46.161 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7856916, partition_space_available_1k=73233568, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1481680, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10756, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52484, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6284064, free=2217132, buffers=277324, cached=7397108, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.82, five_min_avg=2.66, fifteen_min_avg=2.38, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:22:36.819 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:22:46.187 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7861280, partition_space_available_1k=73229204, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1486224, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10788, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52480, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6287952, free=2208168, buffers=277324, cached=7402184, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.42, five_min_avg=2.82, fifteen_min_avg=2.45, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:23:36.812 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:23:46.187 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7870840, partition_space_available_1k=73219644, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1488932, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10808, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52492, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6286132, free=2205360, buffers=277324, cached=7406812, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.65, five_min_avg=2.71, fifteen_min_avg=2.43, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:24:36.921 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:24:46.186 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7876096, partition_space_available_1k=73214388, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1494864, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10820, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52500, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6302952, free=2182636, buffers=277324, cached=7412716, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.72, five_min_avg=2.97, fifteen_min_avg=2.54, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:25:36.874 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:25:46.202 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7886660, partition_space_available_1k=73203824, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1501088, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10820, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52500, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6302496, free=2177552, buffers=277328, cached=7418252, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.52, five_min_avg=2.75, fifteen_min_avg=2.49, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:26:36.870 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:26:46.200 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7891856, partition_space_available_1k=73198628, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1505316, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10752, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52520, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6309144, free=2165064, buffers=277328, cached=7424092, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.6, five_min_avg=2.73, fifteen_min_avg=2.5, cpu_share=10.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:27:36.921 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:27:46.241 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7897620, partition_space_available_1k=73192864, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1510268, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10768, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52516, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6309884, free=2158384, buffers=277328, cached=7430032, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.5, five_min_avg=2.66, fifteen_min_avg=2.49, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:28:36.885 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:28:46.229 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7902312, partition_space_available_1k=73188172, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1514072, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10768, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52516, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6308720, free=2149800, buffers=277332, cached=7439776, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.41, five_min_avg=2.58, fifteen_min_avg=2.47, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:29:36.923 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:29:46.224 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7920496, partition_space_available_1k=73169988, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1496996, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10780, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52524, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6281544, free=2158464, buffers=277336, cached=7458284, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.25, five_min_avg=2.48, fifteen_min_avg=2.44, cpu_share=11.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:30:36.937 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:30:46.235 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7925796, partition_space_available_1k=73164688, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1497848, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10780, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52524, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6287660, free=2141476, buffers=277340, cached=7469152, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.6, five_min_avg=2.53, fifteen_min_avg=2.46, cpu_share=10.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:31:36.907 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:31:46.232 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7932196, partition_space_available_1k=73158288, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1501060, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10784, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52528, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6270224, free=2152764, buffers=277340, cached=7475300, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.59, five_min_avg=2.54, fifteen_min_avg=2.47, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:32:36.993 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:32:46.262 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7937424, partition_space_available_1k=73153060, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1506884, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10764, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52532, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6295648, free=2121728, buffers=277340, cached=7480912, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.49, five_min_avg=2.52, fifteen_min_avg=2.46, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:33:36.978 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:33:46.259 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7947824, partition_space_available_1k=73142660, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1512336, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10768, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52532, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6306356, free=2105412, buffers=277344, cached=7486516, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.78, five_min_avg=2.31, fifteen_min_avg=2.39, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:34:36.978 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:34:46.252 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7957036, partition_space_available_1k=73133448, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1518036, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10788, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52536, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6299872, free=2107436, buffers=277348, cached=7490972, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.22, five_min_avg=2.33, fifteen_min_avg=2.39, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:35:36.930 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:35:46.264 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7961608, partition_space_available_1k=73128876, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1523072, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10800, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52540, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6311180, free=2091472, buffers=277348, cached=7495628, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.41, five_min_avg=2.38, fifteen_min_avg=2.4, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:36:36.932 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:36:46.281 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7966716, partition_space_available_1k=73123768, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1527812, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10800, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52540, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6313436, free=2079092, buffers=277348, cached=7505752, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.6, five_min_avg=2.42, fifteen_min_avg=2.42, cpu_share=10.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:37:36.987 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:37:46.278 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7969428, partition_space_available_1k=73121056, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1531284, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10816, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52548, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6319388, free=2064700, buffers=277352, cached=7514188, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.17, five_min_avg=2.34, fifteen_min_avg=2.39, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:38:37.007 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:38:46.267 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7979300, partition_space_available_1k=73111184, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1536668, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10760, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52548, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6313112, free=2066048, buffers=277352, cached=7519116, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.61, five_min_avg=2.42, fifteen_min_avg=2.41, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:39:36.929 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:39:46.259 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7989560, partition_space_available_1k=73100924, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1547388, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10800, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52552, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6341740, free=2031704, buffers=277352, cached=7524832, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.41, five_min_avg=2.4, fifteen_min_avg=2.4, cpu_share=10.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:40:36.949 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:40:46.265 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7994380, partition_space_available_1k=73096104, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1546912, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10800, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52560, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6330860, free=2037588, buffers=277352, cached=7529828, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.53, five_min_avg=2.43, fifteen_min_avg=2.41, cpu_share=10.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:41:36.986 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:41:46.261 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7999980, partition_space_available_1k=73090504, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1559840, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10808, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52556, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6327696, free=2030700, buffers=277352, cached=7539880, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.15, five_min_avg=2.34, fifteen_min_avg=2.38, cpu_share=10.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:42:37.013 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:42:46.286 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8003972, partition_space_available_1k=73086512, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1556048, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10828, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52560, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6331628, free=2021908, buffers=277352, cached=7544740, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.42, five_min_avg=2.36, fifteen_min_avg=2.38, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:43:36.945 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:43:46.275 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8012144, partition_space_available_1k=73078340, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1560656, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10880, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52564, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6336960, free=2008336, buffers=277352, cached=7552980, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.15, five_min_avg=2.31, fifteen_min_avg=2.36, cpu_share=10.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:44:37.018 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:44:46.289 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8018072, partition_space_available_1k=73072412, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1566740, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10760, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52564, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6316224, free=2022876, buffers=277352, cached=7559176, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.32, five_min_avg=2.34, fifteen_min_avg=2.37, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:45:37.052 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:45:46.284 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8029848, partition_space_available_1k=73060636, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1571624, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10768, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52696, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6312572, free=2020368, buffers=277372, cached=7565316, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.35, five_min_avg=2.34, fifteen_min_avg=2.37, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:46:37.006 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:46:46.304 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8034184, partition_space_available_1k=73056300, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1577428, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10812, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52672, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6322468, free=2005320, buffers=277372, cached=7570468, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.43, five_min_avg=2.39, fifteen_min_avg=2.38, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:47:37.020 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:47:46.331 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8051552, partition_space_available_1k=73038932, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1528004, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10812, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52672, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6254528, free=2050496, buffers=277376, cached=7593228, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.03, five_min_avg=2.28, fifteen_min_avg=2.34, cpu_share=11.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:48:37.074 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:48:46.328 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8054244, partition_space_available_1k=73036240, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1530368, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10832, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52676, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6265304, free=2036924, buffers=277392, cached=7596008, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.9, five_min_avg=2.19, fifteen_min_avg=2.31, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:49:37.156 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:49:40.867 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742435380867012, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=9, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 03/20/2025 01:49:46.341 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8061464, partition_space_available_1k=73029020, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1534604, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10840, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52684, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6275744, free=2019348, buffers=277396, cached=7603140, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.01, five_min_avg=2.17, fifteen_min_avg=2.29, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:50:37.137 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:50:40.840 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742435440840226, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 03/20/2025 01:50:46.346 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8069260, partition_space_available_1k=73021224, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1535644, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10804, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52684, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6266812, free=2020264, buffers=277400, cached=7611152, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.31, five_min_avg=2.22, fifteen_min_avg=2.3, cpu_share=11.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:51:37.126 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:51:40.808 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742435500808833, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=3, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 03/20/2025 01:51:46.354 7f0928182df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8075332, partition_space_available_1k=73015152, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1540828, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10816, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52684, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6265328, free=2015800, buffers=277400, cached=7617100, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.45, five_min_avg=2.26, fifteen_min_avg=2.31, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:52:57.070 7f86d9168f20 [contrail-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 03/20/2025 01:52:57.070 7f86d9168f20 [contrail-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 03/20/2025 01:52:57.070 7f86d9168f20 [contrail-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 03/20/2025 01:52:57.087 7f86d3356648 [contrail-database-nodemgr] [ERROR]: Cannot write http_port 8103 to /tmp/contrail-database-nodemgr.0.http_port 03/20/2025 01:52:57.088 7f86d3356648 [contrail-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8103 03/20/2025 01:52:57.094 7f86d3356e48 [contrail-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 03/20/2025 01:52:57.103 7f86d3356e48 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/20/2025 01:52:57.111 7f86d3356e48 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/20/2025 01:52:57.111 7f86d3356e48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 03/20/2025 01:52:57.112 7f86d3356e48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:57.116 7f86d3356e48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/20/2025 01:52:57.116 7f86d3356e48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:57.116 7f86d3356e48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:57.117 7f86d3356e48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 03/20/2025 01:52:57.117 7f86d3356e48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:57.877 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 21549857983011406121701510459059716657055180939097715056832767013973607837582, 'start': '1742435569000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-query-engine', 'group': 'contrail-query-engine', 'pid': 29653354915007651319541940775083343926056115146295114853271820085147009555723, 'start': '1742435569000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-database-nodemgr', 'group': 'contrail-database-nodemgr', 'pid': 53670892759447302133743699190302748841256872271881017739411426250531946226262, 'start': '1742435569000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 03/20/2025 01:52:58.293 7f86d21a7348 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:58.294 7f86d21a7348 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. build_info = {"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-17 01:12:14.132392", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8139-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 4 num_core_per_socket = 4 num_thread_per_core = 1 >> running_package_version = nightly.r24.1.centos-nightly_r24.1_centos.el7 installed_package_version = nightly.r24.1.centos-nightly_r24.1_centos.el7 >> 03/20/2025 01:52:58.294 7f86d21a7348 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:58.295 7f86d9168f20 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742435578294932, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 03/20/2025 01:52:58.295 7f86d9168f20 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742435578295364, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742435569000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742435569000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742435569000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-17 01:12:14.132392", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8139-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 03/20/2025 01:52:58.315 7f86d21a7348 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:58.315 7f86d21a7348 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/20/2025 01:52:58.315 7f86d21a7348 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:58.315 7f86d21a7348 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:58.315 7f86d21a7348 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742435569000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742435569000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742435569000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-17 01:12:14.132392", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8139-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]} >> 03/20/2025 01:52:58.315 7f86d21a7348 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:53:00.138 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:53:01.098 7f86d21a7348 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:53:01.114 7f86d21a7348 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 03/20/2025 01:53:01.115 7f86d21a7348 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:53:01.115 7f86d21a7348 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:53:01.115 7f86d21a7348 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:53:01.115 7f86d21a7348 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:53:01.115 7f86d21a7348 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 03/20/2025 01:53:01.115 7f86d21a7348 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 03/20/2025 01:53:01.115 7f86d21a7348 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 03/20/2025 01:53:01.116 7f86d21a7348 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:53:01.116 7f86d21a7348 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:53:01.401 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:DatabaseUsage] map 03/20/2025 01:53:01.403 7f86d21a7348 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:53:02.841 7f86d216b348 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001016 03/20/2025 01:53:02.842 7f86d216b348 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 03/20/2025 01:53:02.842 7f86d21a7348 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 03/20/2025 01:53:02.842 7f86d21a7348 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 03/20/2025 01:53:02.843 7f86d21a7348 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 03/20/2025 01:53:02.843 7f86d21a7348 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/20/2025 01:53:02.843 7f86d21a7348 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/20/2025 01:53:02.843 7f86d21a7348 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 03/20/2025 01:53:02.843 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/20/2025 01:53:02.844 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/20/2025 01:53:03.877 7f86d9168f20 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL: Result: 1 03/20/2025 01:53:03.922 7f86d9168f20 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742435583922384, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.12:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='Cassandra state detected DOWN. ')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 03/20/2025 01:53:13.148 7f86d9168f20 [contrail-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 2 03/20/2025 01:53:23.767 7f86d9168f20 [contrail-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7200 repair -pr ContrailAnalyticsCql'. Error is: Result: 1 03/20/2025 01:53:28.829 7f86d9168f20 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742435608829484, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742435569000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742435569000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742435569000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-17 01:12:14.132392", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8139-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 03/20/2025 01:53:28.830 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8305892, partition_space_available_1k=72784592, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1236644, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=9584, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44332, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=4870848, free=3097888, buffers=284996, cached=7921896, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=15.64, five_min_avg=5.92, fifteen_min_avg=3.55, cpu_share=0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:53:58.973 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:54:01.338 7f86d9168f20 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742435641338094, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.12:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 03/20/2025 01:54:11.998 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742435651998631, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 03/20/2025 01:54:11.999 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 03/20/2025 01:54:17.810 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=9018564, partition_space_available_1k=72071920, percentage_partition_space_used=11, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1317756, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=9920, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44388, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5387824, free=1695384, buffers=344704, cached=8747716, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=17.21, five_min_avg=7.98, fifteen_min_avg=4.38, cpu_share=23.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:54:58.778 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:55:11.934 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=9986288, partition_space_available_1k=71104196, percentage_partition_space_used=12, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1329736, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10076, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44400, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5786148, free=314940, buffers=387636, cached=9686904, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=10.76, five_min_avg=7.69, fifteen_min_avg=4.49, cpu_share=17.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:55:46.203 7f86d216b348 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 03/20/2025 01:55:46.203 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[Established] 03/20/2025 01:55:46.205 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 03/20/2025 01:55:46.205 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[Established] -> State[Connect] 03/20/2025 01:55:46.206 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:55:46.206 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Established to Connect on EvTcpClose >>, ] description = Collector:Collector[Established to Connect on EvTcpClose] connection down >>, ] >> 03/20/2025 01:55:46.206 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:55:46.206 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 03/20/2025 01:55:46.206 7f86d21a7048 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 03/20/2025 01:55:46.207 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 03/20/2025 01:55:46.207 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:55:46.207 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:55:46.251 7f86d216b348 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 03/20/2025 01:55:46.251 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 03/20/2025 01:55:46.251 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 03/20/2025 01:55:46.252 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:55:46.252 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 03/20/2025 01:55:46.252 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:55:46.252 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:55:46.252 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 2 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:55:46.252 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:55:50.253 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:55:50.255 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 03/20/2025 01:55:50.255 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:55:50.255 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:55:50.255 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:55:50.255 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:55:50.255 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 03/20/2025 01:55:50.255 7f86d21a7048 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 03/20/2025 01:55:50.256 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 03/20/2025 01:55:50.256 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:55:50.256 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:55:58.569 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:55:58.651 7f86d216b348 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 03/20/2025 01:55:58.651 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 03/20/2025 01:55:58.652 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 03/20/2025 01:55:58.652 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:55:58.652 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 03/20/2025 01:55:58.652 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:55:58.652 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:55:58.652 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:55:58.652 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:55:58.875 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:55:58.875 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. database_usage = [ << disk_space_used_1k = 10355204 disk_space_available_1k = 70735280 analytics_db_size_1k = 32040 >>, ] >> 03/20/2025 01:55:58.875 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:02.652 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:02.653 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:02.657 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:02.657 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:02.659 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:02.662 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:02.662 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:02.662 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:02.662 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:02.663 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:02.665 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:02.665 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:02.665 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:02.665 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:06.654 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:06.656 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:06.656 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:06.656 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:06.656 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:06.656 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:06.656 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:06.657 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:06.657 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:06.657 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:06.657 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:06.657 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:06.657 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:06.657 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:07.348 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10347244, partition_space_available_1k=70743240, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1333360, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11660, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44420, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=4947916, free=1045104, buffers=387392, cached=9795216, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.3, five_min_avg=7.04, fifteen_min_avg=4.45, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:56:07.348 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:07.349 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 10347244 partition_space_available_1k = 70743240 percentage_partition_space_used = 13 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1333360 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 11660 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 44420 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 16175628 used = 4947916 free = 1045104 buffers = 387392 cached = 9795216 node_type = database-node >> system_cpu_usage = << one_min_avg = 6.3 five_min_avg = 7.04 fifteen_min_avg = 4.45 cpu_share = 10.75 node_type = database-node >> >> 03/20/2025 01:56:07.349 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:10.656 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:10.658 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:10.659 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:10.659 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:10.659 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:10.659 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:10.659 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:10.659 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:10.660 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:10.660 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:10.660 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:10.660 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:10.660 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:10.660 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:14.663 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:14.664 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:14.664 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:14.664 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:14.664 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:14.665 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:14.665 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:14.665 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:14.665 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:14.665 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:14.665 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:14.665 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:14.666 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:14.666 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:18.665 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:18.667 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:18.667 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:18.667 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:18.667 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:18.667 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:18.667 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:18.668 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:18.668 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:18.668 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:18.668 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:18.668 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:18.668 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:18.668 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:22.671 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:22.673 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:22.673 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:22.673 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:22.673 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:22.673 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:22.673 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:22.674 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:22.674 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:22.674 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:22.674 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:22.674 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:22.674 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:22.674 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:26.676 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:26.678 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:26.678 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:26.678 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:26.678 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:26.679 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:26.679 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:26.679 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:26.679 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:26.680 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:26.680 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:26.680 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:26.680 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:26.680 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:30.679 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:30.681 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:30.681 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:30.681 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:30.681 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:30.682 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:30.682 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:30.682 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:30.683 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:30.683 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:30.683 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:30.683 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:30.683 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:30.683 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:34.682 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:34.684 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:34.684 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:34.684 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:34.684 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:34.685 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:34.685 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:34.685 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:34.685 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:34.685 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:34.685 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:34.686 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:34.686 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:34.686 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:38.687 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:38.689 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:38.689 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:38.690 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:38.690 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:38.690 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:38.690 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:38.691 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:38.691 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:38.691 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:38.691 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:38.691 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:38.692 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:38.692 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:42.691 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:42.692 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:42.693 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:42.693 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:42.693 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:42.693 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:42.693 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:42.693 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:42.693 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:42.694 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:42.694 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:42.694 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:42.694 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:42.694 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:46.695 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:46.698 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:46.699 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:46.699 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:46.699 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:46.699 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:46.699 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:46.700 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:46.700 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:46.700 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:46.700 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:46.700 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:46.701 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:46.701 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:50.700 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:50.702 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:50.702 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:50.702 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:50.702 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:50.702 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:50.702 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:50.703 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:50.703 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:50.703 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:50.703 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:50.703 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:50.703 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:50.703 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:54.705 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:54.707 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:54.707 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:54.707 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:54.708 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:54.708 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:54.708 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:54.708 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:54.708 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:54.708 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:54.708 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:54.709 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:54.709 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:54.709 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:58.541 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:56:58.709 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:58.714 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:58.714 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:58.715 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:58.715 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:58.715 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:58.715 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:58.715 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:58.716 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:58.716 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:58.716 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:58.716 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:58.716 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:58.716 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:58.800 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:58.800 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. database_usage = [ << disk_space_used_1k = 10262368 disk_space_available_1k = 70828116 analytics_db_size_1k = 32040 >>, ] >> 03/20/2025 01:56:58.800 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:02.714 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:02.719 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:57:02.719 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:02.719 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:02.720 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:02.720 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:02.720 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:57:02.721 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:57:02.721 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:02.721 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:57:02.721 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:02.721 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:02.722 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:57:02.722 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:06.719 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:06.721 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:57:06.722 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:06.722 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:06.722 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:06.722 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:06.722 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:57:06.723 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:57:06.723 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:06.723 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:57:06.723 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:06.723 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:06.723 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:57:06.723 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:10.723 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:10.725 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:57:10.725 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:10.725 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:10.726 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:10.726 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:10.726 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:57:10.726 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:57:10.726 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:10.727 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:57:10.727 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:10.727 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:10.727 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:57:10.728 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:10.761 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10275384, partition_space_available_1k=70815100, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1330720, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12028, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44512, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=4608640, free=1433168, buffers=392248, cached=9741572, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.71, five_min_avg=6.08, fifteen_min_avg=4.28, cpu_share=6.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:57:10.762 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:10.762 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 10275384 partition_space_available_1k = 70815100 percentage_partition_space_used = 13 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1330720 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 12028 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 44512 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 16175628 used = 4608640 free = 1433168 buffers = 392248 cached = 9741572 node_type = database-node >> system_cpu_usage = << one_min_avg = 3.71 five_min_avg = 6.08 fifteen_min_avg = 4.28 cpu_share = 6.25 node_type = database-node >> >> 03/20/2025 01:57:10.762 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:14.727 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:14.728 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:57:14.728 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:14.728 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:14.729 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:14.729 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:14.729 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:57:14.729 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:57:14.729 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:14.730 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:57:14.730 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:14.730 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:14.730 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:57:14.730 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:18.729 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:18.731 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:57:18.731 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:18.731 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:18.731 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:18.732 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:18.732 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:57:18.732 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:57:18.732 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:18.733 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:57:18.733 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:18.733 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:18.733 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:57:18.733 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:22.732 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:22.735 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:57:22.735 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:22.735 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:22.736 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:22.736 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:22.736 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:57:22.737 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:57:22.737 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:22.737 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:57:22.737 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:22.738 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:22.738 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:57:22.738 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:26.738 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:26.739 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:57:26.739 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:26.739 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:26.740 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:26.740 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:26.740 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:57:26.740 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:57:26.740 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:26.740 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:57:26.741 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:26.741 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:26.741 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:57:26.741 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:30.740 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:30.742 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:57:30.742 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:30.742 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:30.742 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:30.743 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:30.743 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:57:30.743 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:57:30.743 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:30.743 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:57:30.743 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:30.743 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:30.743 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:57:30.744 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:34.743 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:34.744 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:57:34.745 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:34.745 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:34.745 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:34.745 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:34.745 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:57:34.746 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:57:34.746 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:34.746 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:57:34.746 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:34.746 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:34.746 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:57:34.746 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:38.747 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:38.751 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:57:38.752 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:38.752 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:38.752 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:38.752 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:38.752 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:57:38.753 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:57:38.753 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:38.753 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:57:38.753 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:38.753 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:38.753 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:57:38.753 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:42.752 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:42.754 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:57:42.754 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:42.754 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:42.755 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:42.755 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:42.755 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:57:42.756 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:57:42.756 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:42.756 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:57:42.756 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:42.756 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:42.756 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1742435577111490 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:57:42.756 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:46.757 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:46.761 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 03/20/2025 01:57:46.761 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:46.761 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:46.762 7f86d21a7048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:46.762 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:46.763 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 03/20/2025 01:57:46.763 7f86d21a7048 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 03/20/2025 01:57:46.764 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 03/20/2025 01:57:46.764 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:57:46.764 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:57:47.487 7f86d216b348 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001016 03/20/2025 01:57:47.488 7f86d216b348 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 03/20/2025 01:57:47.488 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 03/20/2025 01:57:47.488 7f86d21a7048 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 03/20/2025 01:57:47.489 7f86d21a7048 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 03/20/2025 01:57:47.489 7f86d21a7048 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 03/20/2025 01:57:47.489 7f86d21a7048 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/20/2025 01:57:47.489 7f86d21a7048 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/20/2025 01:57:47.489 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 03/20/2025 01:57:47.490 7f86d21a7348 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/20/2025 01:57:47.492 7f86d21a7348 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/20/2025 01:57:57.951 7f86d216b348 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001582 03/20/2025 01:57:57.952 7f86d216b348 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 03/20/2025 01:57:57.952 7f86d21a7348 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 03/20/2025 01:57:57.953 7f86d21a7348 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 4 03/20/2025 01:57:57.953 7f86d21a7348 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 03/20/2025 01:57:57.953 7f86d21a7348 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 03/20/2025 01:57:57.953 7f86d21a7348 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/20/2025 01:57:57.953 7f86d21a7348 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/20/2025 01:57:57.954 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/20/2025 01:57:57.954 7f86d21a7048 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/20/2025 01:57:58.681 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:58:10.803 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10403212, partition_space_available_1k=70687272, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1385044, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12436, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44636, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5968444, free=328272, buffers=396712, cached=9482200, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.91, five_min_avg=5.6, fifteen_min_avg=4.22, cpu_share=10.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:58:58.612 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 01:59:02.647 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742435942647565, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=1), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 03/20/2025 01:59:07.727 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10831632, partition_space_available_1k=70258852, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1374124, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12508, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44704, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5930864, free=282720, buffers=405564, cached=9556480, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.54, five_min_avg=5.19, fifteen_min_avg=4.17, cpu_share=8.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 01:59:58.584 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:00:07.725 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10845572, partition_space_available_1k=70244912, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1368232, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11860, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44716, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6001716, free=261580, buffers=406308, cached=9506024, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.03, five_min_avg=4.48, fifteen_min_avg=3.99, cpu_share=7.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:00:58.606 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:01:07.728 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10844908, partition_space_available_1k=70245576, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1388064, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11928, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44720, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6074384, free=247196, buffers=405388, cached=9448660, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.97, five_min_avg=3.72, fifteen_min_avg=3.76, cpu_share=5.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:01:58.625 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:02:08.744 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11270088, partition_space_available_1k=69820396, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1395252, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12940, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44720, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6166048, free=199304, buffers=410952, cached=9399324, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.35, five_min_avg=3.31, fifteen_min_avg=3.61, cpu_share=7.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:02:58.737 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:03:09.808 7f86d9168f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12050540, partition_space_available_1k=69039944, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1383664, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=0, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44756, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5533188, free=379356, buffers=428752, cached=9834332, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.16, five_min_avg=3.62, fifteen_min_avg=3.69, cpu_share=13.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:03:37.819 7fcabaedac08 [contrail-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 03/20/2025 02:03:37.819 7fcabaedac08 [contrail-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 03/20/2025 02:03:37.819 7fcabaedac08 [contrail-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 03/20/2025 02:03:37.830 7fcabac656d8 [contrail-database-nodemgr] [ERROR]: Cannot write http_port 8103 to /tmp/contrail-database-nodemgr.0.http_port 03/20/2025 02:03:37.830 7fcabac656d8 [contrail-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8103 03/20/2025 02:03:37.831 7fcabac6e048 [contrail-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 03/20/2025 02:03:37.833 7fcabac6e048 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/20/2025 02:03:37.833 7fcabac6e048 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/20/2025 02:03:37.833 7fcabac6e048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 03/20/2025 02:03:37.834 7fcabac6e048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:03:37.834 7fcabac6e048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/20/2025 02:03:37.834 7fcabac6e048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:03:37.834 7fcabac6e048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:03:37.834 7fcabac6e048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1742436217833561 collector_name = collector_ip = collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 03/20/2025 02:03:37.834 7fcabac6e048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:03:38.007 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 14743158731118881825322385583308295068801876566607282990812934120512220915624, 'start': '1742436215000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-query-engine', 'group': 'contrail-query-engine', 'pid': 90409191760734423752729965798547742953675864137097332145219687642213187583024, 'start': '1742436215000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-database-nodemgr', 'group': 'contrail-database-nodemgr', 'pid': 91967697672902944286717777343569859404393212819763246730842494069078890591186, 'start': '1742436215000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 03/20/2025 02:03:38.168 7fcab8be54e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:03:38.168 7fcab8be54e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-03-20 00:47:32.457292", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8220-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 4 num_core_per_socket = 4 num_thread_per_core = 1 >> running_package_version = nightly-nightly.el7 installed_package_version = nightly-nightly.el7 >> 03/20/2025 02:03:38.169 7fcab8be54e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:03:38.169 7fcabaedac08 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742436218169515, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 03/20/2025 02:03:38.170 7fcabaedac08 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742436218169999, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742436215000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742436215000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742436215000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-03-20 00:47:32.457292", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8220-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 03/20/2025 02:03:38.172 7fcab88fb0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:03:38.172 7fcab88fb0f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/20/2025 02:03:38.173 7fcab88fb0f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:03:38.173 7fcab88fb0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:03:38.173 7fcab88fb0f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742436215000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742436215000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742436215000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-03-20 00:47:32.457292", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8220-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 03/20/2025 02:03:38.173 7fcab88fb0f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:03:38.701 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:03:39.186 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:DatabaseUsage] map 03/20/2025 02:03:39.187 7fcab87f62e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:03:39.188 7fcab87f62e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. database_usage = [ << disk_space_used_1k = 12048640 disk_space_available_1k = 69041844 analytics_db_size_1k = 46232 >>, ] >> 03/20/2025 02:03:39.188 7fcab87f62e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:03:39.759 7fcabaedac08 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL: Result: 1 03/20/2025 02:03:39.760 7fcabaedac08 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742436219760012, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 03/20/2025 02:03:39.761 7fcac00c4780 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:03:39.761 7fcac00c4780 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/20/2025 02:03:39.761 7fcac00c4780 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:03:41.834 7fcab8bf4b70 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 02:03:41.836 7fcab8bf4b70 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 03/20/2025 02:03:41.836 7fcab8bf4b70 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 02:03:41.837 7fcab8bf4b70 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 02:03:41.837 7fcab8bf4b70 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 02:03:41.837 7fcab8bf4b70 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 02:03:41.837 7fcab8bf4b70 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 03/20/2025 02:03:41.837 7fcab8bf4b70 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 03/20/2025 02:03:41.837 7fcab8bf4b70 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 03/20/2025 02:03:41.838 7fcab8bf4b70 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 02:03:41.838 7fcab8bf4b70 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 02:03:41.842 7fcab8bf4eb8 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001017 03/20/2025 02:03:41.842 7fcab8bf4eb8 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 03/20/2025 02:03:41.843 7fcabc371c18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 03/20/2025 02:03:41.843 7fcabc371c18 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 03/20/2025 02:03:41.843 7fcabc371c18 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 03/20/2025 02:03:41.844 7fcabc371c18 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/20/2025 02:03:41.844 7fcabc371c18 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/20/2025 02:03:41.844 7fcabc371c18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 03/20/2025 02:03:41.844 7fcab752a2e8 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/20/2025 02:03:41.845 7fcab752a2e8 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/20/2025 02:03:41.989 7fcabaedac08 [contrail-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 2 03/20/2025 02:03:45.491 7fcabaedac08 [contrail-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7200 repair -pr ContrailAnalyticsCql'. Error is: Result: 2 03/20/2025 02:03:50.522 7fcabaedac08 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742436230522082, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742436215000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742436215000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742436215000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-03-20 00:47:32.457292", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8220-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 03/20/2025 02:03:50.522 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12052220, partition_space_available_1k=69038264, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1284784, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=9724, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51128, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5468824, free=432332, buffers=431616, cached=9842856, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.73, five_min_avg=3.56, fifteen_min_avg=3.66, cpu_share=0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:04:38.458 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:04:39.499 7fcabaedac08 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742436279499560, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.47:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 03/20/2025 02:04:42.020 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742436282019934, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 03/20/2025 02:04:42.020 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 03/20/2025 02:04:47.049 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12057368, partition_space_available_1k=69033116, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1302288, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=9896, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51328, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5481764, free=405464, buffers=431676, cached=9856724, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.56, five_min_avg=3.31, fifteen_min_avg=3.58, cpu_share=9.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:05:38.445 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:05:46.897 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12062544, partition_space_available_1k=69027940, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1317000, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10276, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51336, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5802112, free=281328, buffers=430688, cached=9661500, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.26, five_min_avg=3.17, fifteen_min_avg=3.52, cpu_share=6.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:06:38.481 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:06:46.583 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12066052, partition_space_available_1k=69024432, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1325328, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10372, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51332, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5820752, free=286464, buffers=430676, cached=9637736, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.26, five_min_avg=2.71, fifteen_min_avg=3.33, cpu_share=4.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:07:38.474 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:07:46.656 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12069480, partition_space_available_1k=69021004, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1353172, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10408, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51328, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5829604, free=300236, buffers=430612, cached=9615176, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.43, five_min_avg=2.48, fifteen_min_avg=3.21, cpu_share=4.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:08:38.451 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:08:46.595 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12075752, partition_space_available_1k=69014732, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1370572, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10428, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51332, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5874684, free=263472, buffers=430452, cached=9607020, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.48, five_min_avg=2.29, fifteen_min_avg=3.1, cpu_share=5.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:09:38.495 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:09:46.883 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12074624, partition_space_available_1k=69015860, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1348944, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10424, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51332, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5843388, free=292604, buffers=430456, cached=9609180, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.0, five_min_avg=2.01, fifteen_min_avg=2.95, cpu_share=4.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:10:38.511 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:10:46.718 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12077996, partition_space_available_1k=69012488, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1354432, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10484, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51332, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5836656, free=295956, buffers=430456, cached=9612560, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.52, five_min_avg=1.68, fifteen_min_avg=2.78, cpu_share=4.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:11:38.499 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:11:46.809 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12081388, partition_space_available_1k=69009096, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1381916, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10484, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51336, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5854116, free=277660, buffers=430432, cached=9613420, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.77, five_min_avg=1.55, fifteen_min_avg=2.66, cpu_share=4.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:12:38.524 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:12:46.620 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12084924, partition_space_available_1k=69005560, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1370660, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10504, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51340, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5864224, free=278504, buffers=430384, cached=9602516, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.67, five_min_avg=1.39, fifteen_min_avg=2.53, cpu_share=4.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:13:38.524 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:13:46.671 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12088596, partition_space_available_1k=69001888, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1373744, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10568, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51344, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5866944, free=280288, buffers=430268, cached=9598128, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.22, five_min_avg=1.41, fifteen_min_avg=2.47, cpu_share=4.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:14:38.543 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:14:46.658 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12092644, partition_space_available_1k=68997840, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1385444, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10608, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51336, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5885488, free=278052, buffers=430088, cached=9582000, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.67, five_min_avg=1.22, fifteen_min_avg=2.34, cpu_share=4.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:15:38.540 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:15:46.712 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12095888, partition_space_available_1k=68994596, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1395824, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10600, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51344, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5891648, free=274376, buffers=430044, cached=9579560, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.67, five_min_avg=1.13, fifteen_min_avg=2.23, cpu_share=5.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:16:38.565 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:16:46.520 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12099028, partition_space_available_1k=68991456, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1393484, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10676, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51344, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5893152, free=288632, buffers=429788, cached=9564056, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.74, five_min_avg=1.07, fifteen_min_avg=2.14, cpu_share=5.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:17:12.116 7fcab86d6048 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/20/2025 02:17:36.724 7fcaba95e390 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/20/2025 02:17:38.557 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:17:46.762 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12456368, partition_space_available_1k=68634116, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1398284, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10916, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51244, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5901988, free=313724, buffers=433580, cached=9526336, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.14, five_min_avg=1.15, fifteen_min_avg=2.1, cpu_share=5.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:17:52.668 7fcab88fe6d8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/20/2025 02:18:07.929 7fcac00c4780 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/20/2025 02:18:38.572 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:18:41.319 7fcab8bd5ac8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/20/2025 02:18:47.279 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12464208, partition_space_available_1k=68626276, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1402900, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11004, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51228, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5936956, free=324628, buffers=433548, cached=9480496, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.33, five_min_avg=1.24, fifteen_min_avg=2.07, cpu_share=6.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:19:38.567 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:19:46.687 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12467868, partition_space_available_1k=68622616, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1411396, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11024, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51232, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5920416, free=335560, buffers=433616, cached=9486036, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.68, five_min_avg=1.06, fifteen_min_avg=1.96, cpu_share=4.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:20:01.639 7fcab8be12e8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/20/2025 02:20:38.583 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:20:47.224 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12470656, partition_space_available_1k=68619828, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1438276, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12416, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51232, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5989792, free=262572, buffers=433516, cached=9489748, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.94, five_min_avg=1.38, fifteen_min_avg=2.01, cpu_share=6.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:21:38.581 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:21:46.965 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12483236, partition_space_available_1k=68607248, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1455140, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12320, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51232, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5961312, free=299896, buffers=433000, cached=9481420, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.12, five_min_avg=1.73, fifteen_min_avg=2.08, cpu_share=8.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:22:38.595 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:22:46.778 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12498832, partition_space_available_1k=68591652, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1480312, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12320, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51228, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5993276, free=279808, buffers=432612, cached=9469932, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.96, five_min_avg=2.02, fifteen_min_avg=2.17, cpu_share=9.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:23:38.582 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:23:46.766 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12509408, partition_space_available_1k=68581076, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1498628, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12376, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51220, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6017608, free=288740, buffers=432520, cached=9436760, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.6, five_min_avg=1.8, fifteen_min_avg=2.08, cpu_share=7.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:24:38.591 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:24:46.795 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12523556, partition_space_available_1k=68566928, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1518428, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12428, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51212, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6035760, free=296076, buffers=432464, cached=9411328, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.43, five_min_avg=1.74, fifteen_min_avg=2.04, cpu_share=7.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:25:38.581 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:25:46.592 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12533144, partition_space_available_1k=68557340, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1535276, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12476, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51212, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6031400, free=304552, buffers=432460, cached=9407216, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.95, five_min_avg=1.56, fifteen_min_avg=1.96, cpu_share=6.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:26:38.602 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:26:46.734 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12555612, partition_space_available_1k=68534872, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1659616, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12584, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51196, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6109596, free=289352, buffers=432432, cached=9344248, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.36, five_min_avg=1.6, fifteen_min_avg=1.95, cpu_share=7.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:27:14.625 7fcab8bf3588 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/20/2025 02:27:38.658 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:27:46.822 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12562636, partition_space_available_1k=68527848, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1664120, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12648, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51192, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6125076, free=291132, buffers=432408, cached=9327012, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.53, five_min_avg=1.89, fifteen_min_avg=2.03, cpu_share=8.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/20/2025 02:28:38.713 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/20/2025 02:28:39.495 7fcab8bdb978 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/20/2025 02:28:42.908 7fcabaedac08 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='Database', _instance_id='0', _timestamp=1742437722906788, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=2)], _table='ObjectDatabaseInfo'), _seqnum=0)