05/28/2026 01:59:39.071 7f79f21d8500 [contrail-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 05/28/2026 01:59:39.071 7f79f21d8500 [contrail-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 05/28/2026 01:59:39.071 7f79f21d8500 [contrail-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 05/28/2026 01:59:39.081 7f79ef358ea0 [contrail-database-nodemgr] [ERROR]: Cannot write http_port 8103 to /tmp/contrail-database-nodemgr.0.http_port 05/28/2026 01:59:39.081 7f79ef358ea0 [contrail-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8103 05/28/2026 01:59:39.082 7f79ef02a2c0 [contrail-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 05/28/2026 01:59:39.083 7f79ef02a2c0 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 05/28/2026 01:59:39.083 7f79ef02a2c0 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 05/28/2026 01:59:39.083 7f79ef02a2c0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 05/28/2026 01:59:39.083 7f79ef02a2c0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:39.084 7f79ef02a2c0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 05/28/2026 01:59:39.084 7f79ef02a2c0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:39.084 7f79ef02a2c0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:39.084 7f79ef02a2c0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1779933579083619 collector_name = collector_ip = collector_list = [ 10.0.0.242:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 05/28/2026 01:59:39.084 7f79ef02a2c0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:39.242 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 12382186090607316454120746513984234607834938451214397873253205942834157458756, 'start': '1779933577000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-query-engine', 'group': 'contrail-query-engine', 'pid': 27313095394213697120589430876632399804353770077120078144538421099179190141548, 'start': '1779933577000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-database-nodemgr', 'group': 'contrail-database-nodemgr', 'pid': 25344674779743857933283436187680647680077105102143774870048102142295720907867, 'start': '1779933577000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 05/28/2026 01:59:39.292 7f79ef02a360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:39.293 7f79ef02a360 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. build_info = {"build-info": [{"build-version": "v0+nightly", "build-time": "2026-05-28 00:52:49.610042", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-15064-1.novalocal", "build-id": "v0+nightly", "build-number": "202605280007"}]} system_cpu_info = << num_socket = 1 num_cpu = 8 num_core_per_socket = 8 num_thread_per_core = 1 >> >> 05/28/2026 01:59:39.293 7f79ef02a360 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:39.294 7f79f21d8500 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5838-1.', _node_type='Database', _instance_id='0', _timestamp=1779933579293985, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 05/28/2026 01:59:39.294 7f79f21d8500 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5838-1.', _node_type='Database', _instance_id='0', _timestamp=1779933579294416, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1779933577000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1779933577000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1779933577000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info": [{"build-version": "v0+nightly", "build-time": "2026-05-28 00:52:49.610042", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-15064-1.novalocal", "build-id": "v0+nightly", "build-number": "202605280007"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 05/28/2026 01:59:39.297 7f79ef02a360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:39.297 7f79ef02a360 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 05/28/2026 01:59:39.297 7f79ef02a360 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:39.298 7f79ef02a360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:39.298 7f79ef02a360 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1779933577000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1779933577000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1779933577000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info": [{"build-version": "v0+nightly", "build-time": "2026-05-28 00:52:49.610042", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-15064-1.novalocal", "build-id": "v0+nightly", "build-number": "202605280007"}]} >> 05/28/2026 01:59:39.298 7f79ef02a360 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:39.658 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 05/28/2026 01:59:40.011 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:DatabaseUsage] map 05/28/2026 01:59:40.014 7f79ef02aae0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:40.014 7f79ef02aae0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. database_usage = [ << disk_space_used_1k = 14681220 disk_space_available_1k = 66409264 analytics_db_size_1k = 108 >>, ] >> 05/28/2026 01:59:40.014 7f79ef02aae0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:40.421 7f79f21d8500 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL and socket: Result: 1 05/28/2026 01:59:40.421 7f79f21d8500 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5838-1.', _node_type='Database', _instance_id='0', _timestamp=1779933580421516, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 05/28/2026 01:59:40.423 7f79ef02aae0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:40.424 7f79ef02aae0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 05/28/2026 01:59:40.424 7f79ef02aae0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:42.848 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5838-1.', _node_type='Database', _instance_id='0', _timestamp=1779933582848809, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 05/28/2026 01:59:42.849 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 05/28/2026 01:59:42.850 7f79ef02aa40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:42.850 7f79ef02aa40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] CassandraStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. cassandra_compaction_task = << pending_compaction_tasks = 0 >> thread_pool_stats = [ << pool_name = ReadStage active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = CompactionExecutor active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = MutationStage active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = MemtableFlushWriter active = 0 pending = 0 all_time_blocked = 0 >>, ] >> 05/28/2026 01:59:42.850 7f79ef02aa40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:43.085 7f79efbfe900 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 05/28/2026 01:59:43.087 7f79efbfe900 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 05/28/2026 01:59:43.087 7f79efbfe900 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 05/28/2026 01:59:43.088 7f79efbfe900 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 05/28/2026 01:59:43.088 7f79efbfe900 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.242:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 05/28/2026 01:59:43.088 7f79efbfe900 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 05/28/2026 01:59:43.088 7f79efbfe900 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 05/28/2026 01:59:43.088 7f79efbfe900 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 05/28/2026 01:59:43.088 7f79efbfe900 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:43.089 7f79efbfe900 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.242:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 05/28/2026 01:59:43.089 7f79efbfe900 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:43.089 7f79efbfe900 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:43.089 7f79efbfe900 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1779933579083619 collector_name = collector_ip = 10.0.0.242:8086 collector_list = [ 10.0.0.242:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 05/28/2026 01:59:43.089 7f79efbfe900 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:44.000 7f79f21d8500 [contrail-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7200 repair -pr ContrailAnalyticsCql'. Error is: Result: 1 05/28/2026 01:59:47.088 7f79efbfe860 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 05/28/2026 01:59:47.090 7f79efbfe860 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 05/28/2026 01:59:47.091 7f79efbfe860 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 05/28/2026 01:59:47.091 7f79efbfe860 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 05/28/2026 01:59:47.092 7f79efbfe860 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.242:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 05/28/2026 01:59:47.092 7f79efbfe860 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 05/28/2026 01:59:47.092 7f79efbfe860 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 05/28/2026 01:59:47.092 7f79efbfe860 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 05/28/2026 01:59:47.092 7f79efbfe860 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:47.092 7f79efbfe860 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.242:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 05/28/2026 01:59:47.092 7f79efbfe860 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:47.093 7f79efbfe860 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:47.093 7f79efbfe860 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1779933579083619 collector_name = collector_ip = 10.0.0.242:8086 collector_list = [ 10.0.0.242:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 05/28/2026 01:59:47.093 7f79efbfe860 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:49.041 7f79f21d8500 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5838-1.', _node_type='Database', _instance_id='0', _timestamp=1779933589041074, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1779933577000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1779933577000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1779933577000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info": [{"build-version": "v0+nightly", "build-time": "2026-05-28 00:52:49.610042", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-15064-1.novalocal", "build-id": "v0+nightly", "build-number": "202605280007"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 05/28/2026 01:59:49.041 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14682728, partition_space_available_1k=66407756, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1303296, cpu_share=0.0, mem_res=1293464, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=7964, cpu_share=0.0, mem_res=6224, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55400, cpu_share=0.0, mem_res=45776, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=7929900, free=10007004, buffers=650024, cached=14550568, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.79, five_min_avg=2.38, fifteen_min_avg=1.45, cpu_share=0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 05/28/2026 01:59:49.041 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:49.041 7f79ef02aea0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1779933577000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1779933577000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1779933577000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info": [{"build-version": "v0+nightly", "build-time": "2026-05-28 00:52:49.610042", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-15064-1.novalocal", "build-id": "v0+nightly", "build-number": "202605280007"}]} >> 05/28/2026 01:59:49.042 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:49.042 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:49.042 7f79ef02aea0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 14682728 partition_space_available_1k = 66407756 percentage_partition_space_used = 18 >> } all_core_file_list = [ ] process_mem_cpu_usage = { cassandra : << mem_virt = 1303296 cpu_share = 0.0 mem_res = 1293464 >>contrail-query-engine : << mem_virt = 7964 cpu_share = 0.0 mem_res = 6224 >>contrail-database-nodemgr : << mem_virt = 55400 cpu_share = 0.0 mem_res = 45776 >> } system_mem_usage = << total = 32665120 used = 7929900 free = 10007004 buffers = 650024 cached = 14550568 node_type = database-node >> system_cpu_usage = << one_min_avg = 3.79 five_min_avg = 2.38 fifteen_min_avg = 1.45 cpu_share = 0 node_type = database-node >> >> 05/28/2026 01:59:49.042 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:51.092 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 05/28/2026 01:59:51.093 7f79ef02aea0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 05/28/2026 01:59:51.094 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 05/28/2026 01:59:51.094 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 05/28/2026 01:59:51.094 7f79ef02aea0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.242:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 05/28/2026 01:59:51.094 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 05/28/2026 01:59:51.094 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 05/28/2026 01:59:51.094 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 05/28/2026 01:59:51.094 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:51.095 7f79ef02aea0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.242:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 05/28/2026 01:59:51.095 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:51.095 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:51.095 7f79ef02aea0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1779933579083619 collector_name = collector_ip = 10.0.0.242:8086 collector_list = [ 10.0.0.242:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 05/28/2026 01:59:51.095 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:55.095 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 05/28/2026 01:59:55.096 7f79ef02aea0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 05/28/2026 01:59:55.096 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 05/28/2026 01:59:55.096 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 05/28/2026 01:59:55.096 7f79ef02aea0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.242:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 05/28/2026 01:59:55.096 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 05/28/2026 01:59:55.096 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 05/28/2026 01:59:55.097 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 05/28/2026 01:59:55.097 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:55.097 7f79ef02aea0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.242:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 05/28/2026 01:59:55.097 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:55.097 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:55.097 7f79ef02aea0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1779933579083619 collector_name = collector_ip = 10.0.0.242:8086 collector_list = [ 10.0.0.242:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 05/28/2026 01:59:55.098 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:59.100 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 05/28/2026 01:59:59.101 7f79ef02aea0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 05/28/2026 01:59:59.101 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 05/28/2026 01:59:59.101 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 05/28/2026 01:59:59.101 7f79ef02aea0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.242:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 05/28/2026 01:59:59.101 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 05/28/2026 01:59:59.102 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 05/28/2026 01:59:59.102 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 05/28/2026 01:59:59.102 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:59.102 7f79ef02aea0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.242:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 05/28/2026 01:59:59.102 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:59.102 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 01:59:59.102 7f79ef02aea0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1779933579083619 collector_name = collector_ip = 10.0.0.242:8086 collector_list = [ 10.0.0.242:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 05/28/2026 01:59:59.102 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 02:00:03.104 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 05/28/2026 02:00:03.105 7f79ef02aea0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 05/28/2026 02:00:03.106 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 05/28/2026 02:00:03.106 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 05/28/2026 02:00:03.106 7f79ef02aea0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.242:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 05/28/2026 02:00:03.106 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 05/28/2026 02:00:03.106 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 05/28/2026 02:00:03.107 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 05/28/2026 02:00:03.107 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 02:00:03.107 7f79ef02aea0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.242:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 05/28/2026 02:00:03.107 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 02:00:03.107 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 02:00:03.108 7f79ef02aea0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1779933579083619 collector_name = collector_ip = 10.0.0.242:8086 collector_list = [ 10.0.0.242:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 05/28/2026 02:00:03.108 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 02:00:07.110 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 05/28/2026 02:00:07.111 7f79ef02aea0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 05/28/2026 02:00:07.112 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 05/28/2026 02:00:07.112 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 05/28/2026 02:00:07.112 7f79ef02aea0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.242:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 05/28/2026 02:00:07.112 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 05/28/2026 02:00:07.112 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 05/28/2026 02:00:07.113 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 05/28/2026 02:00:07.113 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 02:00:07.113 7f79ef02aea0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.242:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 05/28/2026 02:00:07.113 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 02:00:07.113 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 02:00:07.113 7f79ef02aea0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1779933579083619 collector_name = collector_ip = 10.0.0.242:8086 collector_list = [ 10.0.0.242:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 05/28/2026 02:00:07.113 7f79ef02aea0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 02:00:11.113 7f79efbfee00 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 05/28/2026 02:00:11.115 7f79efbfee00 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 05/28/2026 02:00:11.115 7f79efbfee00 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 05/28/2026 02:00:11.115 7f79efbfee00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 05/28/2026 02:00:11.115 7f79efbfee00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.242:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 05/28/2026 02:00:11.115 7f79efbfee00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 05/28/2026 02:00:11.115 7f79efbfee00 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 05/28/2026 02:00:11.116 7f79efbfee00 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 05/28/2026 02:00:11.116 7f79efbfee00 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 05/28/2026 02:00:11.116 7f79efbfee00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 05/28/2026 02:00:11.116 7f79efbfee00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 05/28/2026 02:00:22.214 7f79ef02a680 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 05/28/2026 02:00:22.214 7f79efd540e0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 05/28/2026 02:00:22.214 7f79efd540e0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 05/28/2026 02:00:22.219 7f79efd540e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 02:00:22.220 7f79efd540e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.242:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Cassandra state detected DOWN. Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 05/28/2026 02:00:22.220 7f79efd540e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 02:00:22.220 7f79efd540e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 05/28/2026 02:00:22.220 7f79efd540e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1779933579083619 collector_name = collector_ip = 10.0.0.242:8086 collector_list = [ 10.0.0.242:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 05/28/2026 02:00:22.220 7f79efd540e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 05/28/2026 02:00:26.216 7f79efdada40 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 05/28/2026 02:00:26.217 7f79efdada40 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 05/28/2026 02:00:26.219 7f79efdada40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 05/28/2026 02:00:26.220 7f79efdada40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 05/28/2026 02:00:26.220 7f79efdada40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5838-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.242:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 05/28/2026 02:00:26.220 7f79efdada40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 05/28/2026 02:00:26.220 7f79efdada40 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 05/28/2026 02:00:26.220 7f79efdada40 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 05/28/2026 02:00:26.221 7f79efdada40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 05/28/2026 02:00:26.221 7f79efdada40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 05/28/2026 02:00:26.221 7f79efdada40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 05/28/2026 02:00:26.226 7f79efbfe7c0 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001017 05/28/2026 02:00:26.227 7f79efbfe7c0 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 05/28/2026 02:00:26.227 7f79efdada40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 05/28/2026 02:00:26.227 7f79efdada40 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 05/28/2026 02:00:26.227 7f79efdada40 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 05/28/2026 02:00:26.227 7f79efdada40 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 05/28/2026 02:00:26.227 7f79efdada40 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 05/28/2026 02:00:26.228 7f79efdada40 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 05/28/2026 02:00:26.228 7f79efdada40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 05/28/2026 02:00:26.228 7f79efbfe040 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 05/28/2026 02:00:26.230 7f79efbfe040 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 05/28/2026 02:00:33.858 7f79efbfe7c0 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001582 05/28/2026 02:00:33.858 7f79efbfe7c0 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 05/28/2026 02:00:33.859 7f79efdada40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 05/28/2026 02:00:33.859 7f79efdada40 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 4 05/28/2026 02:00:33.860 7f79efdada40 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 05/28/2026 02:00:33.860 7f79efdada40 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 05/28/2026 02:00:33.860 7f79efdada40 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 05/28/2026 02:00:33.860 7f79efdada40 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 05/28/2026 02:00:33.860 7f79efbfe040 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 05/28/2026 02:00:33.861 7f79efbfe040 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 05/28/2026 02:00:39.572 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 05/28/2026 02:00:40.580 7f79f21d8500 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5838-1.', _node_type='Database', _instance_id='0', _timestamp=1779933640580358, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.242:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 05/28/2026 02:00:42.916 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5838-1.', _node_type='Database', _instance_id='0', _timestamp=1779933642916273, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 05/28/2026 02:00:47.967 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=15656760, partition_space_available_1k=65433724, percentage_partition_space_used=19, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1326928, cpu_share=0.34, mem_res=1311348, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11864, cpu_share=0.0, mem_res=9444, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55436, cpu_share=0.02, mem_res=45772, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=9088788, free=7878036, buffers=673896, cached=15497648, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.94, five_min_avg=2.92, fifteen_min_avg=1.69, cpu_share=3.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 05/28/2026 02:01:39.863 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 05/28/2026 02:01:48.478 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=16596576, partition_space_available_1k=64493908, percentage_partition_space_used=20, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1362824, cpu_share=0.14, mem_res=1331232, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11840, cpu_share=0.0, mem_res=9780, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47924, cpu_share=0.02, mem_res=45772, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=9433420, free=20291440, buffers=689892, cached=2723784, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.12, five_min_avg=3.07, fifteen_min_avg=1.82, cpu_share=4.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 05/28/2026 02:02:39.577 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 05/28/2026 02:02:47.839 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=16607156, partition_space_available_1k=64483328, percentage_partition_space_used=20, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1334796, cpu_share=0.09, mem_res=1318472, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11936, cpu_share=0.0, mem_res=9876, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=48008, cpu_share=0.02, mem_res=45880, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=9301512, free=20302160, buffers=692980, cached=2841824, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.65, five_min_avg=2.84, fifteen_min_avg=1.82, cpu_share=3.12, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 05/28/2026 02:03:18.724 7f79ef02a900 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 05/28/2026 02:03:31.197 7f79ef060040 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 05/28/2026 02:03:33.436 7f79ef060040 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 05/28/2026 02:03:39.586 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 05/28/2026 02:03:47.833 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=16764044, partition_space_available_1k=64326440, percentage_partition_space_used=21, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1387008, cpu_share=0.14, mem_res=1368840, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12336, cpu_share=0.0, mem_res=10272, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=48000, cpu_share=0.02, mem_res=45880, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10073744, free=19435924, buffers=696520, cached=2931236, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.3, five_min_avg=2.68, fifteen_min_avg=1.83, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 05/28/2026 02:04:39.623 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 05/28/2026 02:04:41.891 7f79ef02a680 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 05/28/2026 02:04:48.237 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=18050440, partition_space_available_1k=63040044, percentage_partition_space_used=22, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1373352, cpu_share=0.14, mem_res=1353524, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12544, cpu_share=0.0, mem_res=10476, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=48036, cpu_share=0.02, mem_res=45880, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10363632, free=17723632, buffers=729416, cached=4320932, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.83, five_min_avg=2.74, fifteen_min_avg=1.91, cpu_share=4.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 05/28/2026 02:05:39.643 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 05/28/2026 02:05:47.977 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=18111432, partition_space_available_1k=62979052, percentage_partition_space_used=22, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1377372, cpu_share=0.18, mem_res=1351596, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18160, cpu_share=0.0, mem_res=15996, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=48036, cpu_share=0.02, mem_res=45880, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10702492, free=17292320, buffers=730124, cached=4412796, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.25, five_min_avg=2.84, fifteen_min_avg=1.99, cpu_share=4.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 05/28/2026 02:06:39.637 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 05/28/2026 02:06:48.672 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=18260700, partition_space_available_1k=62829784, percentage_partition_space_used=23, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1416648, cpu_share=0.17, mem_res=1386660, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18092, cpu_share=0.0, mem_res=15932, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=48036, cpu_share=0.02, mem_res=45880, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=11219368, free=16620068, buffers=730228, cached=4568040, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.91, five_min_avg=3.06, fifteen_min_avg=2.12, cpu_share=5.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 05/28/2026 02:07:39.663 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 05/28/2026 02:07:47.800 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=18288908, partition_space_available_1k=62801576, percentage_partition_space_used=23, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1403744, cpu_share=0.14, mem_res=1369624, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18176, cpu_share=0.0, mem_res=16016, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=48044, cpu_share=0.02, mem_res=45880, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10913548, free=16882280, buffers=730832, cached=4616600, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.47, five_min_avg=3.12, fifteen_min_avg=2.2, cpu_share=4.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 05/28/2026 02:08:39.689 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 05/28/2026 02:08:47.927 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=18316448, partition_space_available_1k=62774036, percentage_partition_space_used=23, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1406844, cpu_share=0.16, mem_res=1367932, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18292, cpu_share=0.0, mem_res=16132, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=48044, cpu_share=0.02, mem_res=45880, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=11040684, free=16718428, buffers=730904, cached=4646896, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.9, five_min_avg=3.25, fifteen_min_avg=2.3, cpu_share=4.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 05/28/2026 02:09:39.613 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 05/28/2026 02:09:47.812 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=18410556, partition_space_available_1k=62679928, percentage_partition_space_used=23, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1416416, cpu_share=0.11, mem_res=1374060, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18420, cpu_share=0.0, mem_res=16260, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=48044, cpu_share=0.02, mem_res=45880, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=11926588, free=15785888, buffers=731012, cached=4693448, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.34, five_min_avg=3.86, fifteen_min_avg=2.58, cpu_share=4.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 05/28/2026 02:10:39.639 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 05/28/2026 02:10:47.712 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=18405972, partition_space_available_1k=62684512, percentage_partition_space_used=23, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1419068, cpu_share=0.12, mem_res=1372800, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18420, cpu_share=0.0, mem_res=16260, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=48056, cpu_share=0.02, mem_res=45880, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=11319820, free=16364960, buffers=731068, cached=4720672, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.78, five_min_avg=3.92, fifteen_min_avg=2.68, cpu_share=3.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 05/28/2026 02:11:39.666 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 05/28/2026 02:11:48.051 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=18430440, partition_space_available_1k=62660044, percentage_partition_space_used=23, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1435036, cpu_share=0.11, mem_res=1385768, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18536, cpu_share=0.0, mem_res=16376, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=48068, cpu_share=0.02, mem_res=45880, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=11148920, free=16510184, buffers=731100, cached=4746624, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.77, five_min_avg=4.03, fifteen_min_avg=2.79, cpu_share=4.12, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 05/28/2026 02:12:39.731 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 05/28/2026 02:12:48.446 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=18452920, partition_space_available_1k=62637564, percentage_partition_space_used=23, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1443176, cpu_share=0.11, mem_res=1390328, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18488, cpu_share=0.0, mem_res=16328, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=48056, cpu_share=0.02, mem_res=45880, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=11899208, free=15733184, buffers=731124, cached=4772472, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.04, five_min_avg=3.91, fifteen_min_avg=2.82, cpu_share=4.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 05/28/2026 02:13:39.716 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 05/28/2026 02:13:48.191 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=18547700, partition_space_available_1k=62542784, percentage_partition_space_used=23, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1449032, cpu_share=0.1, mem_res=1393508, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18540, cpu_share=0.0, mem_res=16380, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=48064, cpu_share=0.02, mem_res=45880, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=11403408, free=16127316, buffers=731168, cached=4873804, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.43, five_min_avg=3.75, fifteen_min_avg=2.84, cpu_share=4.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 05/28/2026 02:14:39.661 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 05/28/2026 02:14:47.951 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=18553764, partition_space_available_1k=62536720, percentage_partition_space_used=23, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1457436, cpu_share=0.1, mem_res=1399820, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18644, cpu_share=0.0, mem_res=16484, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=48060, cpu_share=0.02, mem_res=45880, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=11198468, free=16324868, buffers=731180, cached=4881140, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.7, five_min_avg=3.44, fifteen_min_avg=2.78, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 05/28/2026 02:15:15.577 7f79ef3587c0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 05/28/2026 02:15:39.699 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 05/28/2026 02:15:47.910 7f79f21d8500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5838-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=18613520, partition_space_available_1k=62476964, percentage_partition_space_used=23, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1459172, cpu_share=0.12, mem_res=1398172, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18736, cpu_share=0.0, mem_res=16572, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=48072, cpu_share=0.02, mem_res=45880, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10985096, free=16465540, buffers=733588, cached=4951404, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.49, five_min_avg=3.53, fifteen_min_avg=2.85, cpu_share=3.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 05/28/2026 02:16:12.326 7f79ef02ac20 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map