07/31/2025 02:18:50.724 7f8dcfd17c88 [contrail-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 07/31/2025 02:18:50.724 7f8dcfd17c88 [contrail-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 07/31/2025 02:18:50.724 7f8dcfd17c88 [contrail-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 07/31/2025 02:18:50.733 7f8dcd28e8d0 [contrail-database-nodemgr] [ERROR]: Cannot write http_port 8103 to /tmp/contrail-database-nodemgr.0.http_port 07/31/2025 02:18:50.734 7f8dcd28e8d0 [contrail-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8103 07/31/2025 02:18:50.735 7f8dcd298240 [contrail-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 07/31/2025 02:18:50.735 7f8dcd298240 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 07/31/2025 02:18:50.736 7f8dcd298240 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 07/31/2025 02:18:50.736 7f8dcd298240 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 07/31/2025 02:18:50.736 7f8dcd298240 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:18:50.736 7f8dcd298240 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/31/2025 02:18:50.736 7f8dcd298240 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:18:50.736 7f8dcd298240 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:18:50.736 7f8dcd298240 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1753928330735950 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 07/31/2025 02:18:50.736 7f8dcd298240 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:18:50.972 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 9965531517279928158269678838120921375475402772753235743223497923290104275532, 'start': '1753928328000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-query-engine', 'group': 'contrail-query-engine', 'pid': 28785211228812262139860895161079019974516488221949985672786159270958480026718, 'start': '1753928328000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-database-nodemgr', 'group': 'contrail-database-nodemgr', 'pid': 92239741728902389523732912904412731870488360013904810550823051525180370544947, 'start': '1753928328000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 07/31/2025 02:18:51.132 7f8dd5f23a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:18:51.133 7f8dd5f23a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-07-31 00:51:12.408828", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10082-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 8 num_core_per_socket = 8 num_thread_per_core = 1 >> running_package_version = nightly-nightly.el7 installed_package_version = nightly-nightly.el7 >> 07/31/2025 02:18:51.133 7f8dd5f23a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:18:51.134 7f8dcfd17c88 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3729-1.', _node_type='Database', _instance_id='0', _timestamp=1753928331134384, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 07/31/2025 02:18:51.135 7f8dcfd17c88 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3729-1.', _node_type='Database', _instance_id='0', _timestamp=1753928331135017, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1753928328000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1753928328000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1753928328000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-07-31 00:51:12.408828", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10082-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 07/31/2025 02:18:51.138 7f8dcd2cd240 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:18:51.138 7f8dcd2cd240 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/31/2025 02:18:51.138 7f8dcd2cd240 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:18:51.138 7f8dcd2cd240 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:18:51.139 7f8dcd2cd240 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1753928328000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1753928328000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1753928328000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-07-31 00:51:12.408828", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10082-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 07/31/2025 02:18:51.139 7f8dcd2cd240 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:18:51.615 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 07/31/2025 02:18:52.064 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:DatabaseUsage] map 07/31/2025 02:18:52.066 7f8dcfb914e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:18:52.066 7f8dcfb914e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. database_usage = [ << disk_space_used_1k = 15544532 disk_space_available_1k = 65545952 analytics_db_size_1k = 120 >>, ] >> 07/31/2025 02:18:52.066 7f8dcfb914e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:18:52.607 7f8dcfd17c88 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL and socket: Result: 1 07/31/2025 02:18:52.607 7f8dcfd17c88 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3729-1.', _node_type='Database', _instance_id='0', _timestamp=1753928332607247, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 07/31/2025 02:18:52.608 7f8dcc411c18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:18:52.609 7f8dcc411c18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/31/2025 02:18:52.609 7f8dcc411c18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:18:54.736 7f8dcc4248d0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/31/2025 02:18:54.738 7f8dcc4248d0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/31/2025 02:18:54.738 7f8dcc4248d0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/31/2025 02:18:54.738 7f8dcc4248d0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/31/2025 02:18:54.738 7f8dcc4248d0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/31/2025 02:18:54.738 7f8dcc4248d0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/31/2025 02:18:54.738 7f8dcc4248d0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/31/2025 02:18:54.739 7f8dcc4248d0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/31/2025 02:18:54.739 7f8dcc4248d0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:18:54.739 7f8dcc4248d0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/31/2025 02:18:54.739 7f8dcc4248d0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:18:54.739 7f8dcc4248d0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:18:54.740 7f8dcc4248d0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1753928330735950 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/31/2025 02:18:54.740 7f8dcc4248d0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:18:55.274 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3729-1.', _node_type='Database', _instance_id='0', _timestamp=1753928335274068, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=1), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 07/31/2025 02:18:55.274 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 07/31/2025 02:18:55.276 7f8dcfb912e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:18:55.277 7f8dcfb912e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] CassandraStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. cassandra_compaction_task = << pending_compaction_tasks = 1 >> thread_pool_stats = [ << pool_name = ReadStage active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = CompactionExecutor active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = MutationStage active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = MemtableFlushWriter active = 0 pending = 0 all_time_blocked = 0 >>, ] >> 07/31/2025 02:18:55.277 7f8dcfb912e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:18:56.613 7f8dcfd17c88 [contrail-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7200 repair -pr ContrailAnalyticsCql'. Error is: Result: 1 07/31/2025 02:18:58.739 7f8dd0f52630 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/31/2025 02:18:58.741 7f8dd0f52630 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/31/2025 02:18:58.741 7f8dd0f52630 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/31/2025 02:18:58.741 7f8dd0f52630 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/31/2025 02:18:58.741 7f8dd0f52630 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/31/2025 02:18:58.741 7f8dd0f52630 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/31/2025 02:18:58.741 7f8dd0f52630 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/31/2025 02:18:58.742 7f8dd0f52630 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/31/2025 02:18:58.742 7f8dd0f52630 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:18:58.742 7f8dd0f52630 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/31/2025 02:18:58.742 7f8dd0f52630 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:18:58.742 7f8dd0f52630 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:18:58.742 7f8dd0f52630 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1753928330735950 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/31/2025 02:18:58.742 7f8dd0f52630 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:01.654 7f8dcfd17c88 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3729-1.', _node_type='Database', _instance_id='0', _timestamp=1753928341654419, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1753928328000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1753928328000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1753928328000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-07-31 00:51:12.408828", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10082-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 07/31/2025 02:19:01.655 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=15768188, partition_space_available_1k=65322296, percentage_partition_space_used=19, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1219488, cpu_share=0.0, mem_res=1208268, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=8528, cpu_share=0.0, mem_res=6116, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52248, cpu_share=0.0, mem_res=40424, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=7762040, free=8068488, buffers=702872, cached=16131720, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.01, five_min_avg=2.44, fifteen_min_avg=1.53, cpu_share=0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 07/31/2025 02:19:01.655 7f8dcd65f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:01.655 7f8dcd65f2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1753928328000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1753928328000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1753928328000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-07-31 00:51:12.408828", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10082-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 07/31/2025 02:19:01.656 7f8dcd65f2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:01.656 7f8dcd65f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:01.656 7f8dcd65f2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 15768188 partition_space_available_1k = 65322296 percentage_partition_space_used = 19 >> } all_core_file_list = [ ] process_mem_cpu_usage = { cassandra : << mem_virt = 1219488 cpu_share = 0.0 mem_res = 1208268 >>contrail-query-engine : << mem_virt = 8528 cpu_share = 0.0 mem_res = 6116 >>contrail-database-nodemgr : << mem_virt = 52248 cpu_share = 0.0 mem_res = 40424 >> } system_mem_usage = << total = 32665120 used = 7762040 free = 8068488 buffers = 702872 cached = 16131720 node_type = database-node >> system_cpu_usage = << one_min_avg = 3.01 five_min_avg = 2.44 fifteen_min_avg = 1.53 cpu_share = 0 node_type = database-node >> >> 07/31/2025 02:19:01.656 7f8dcd65f2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:02.742 7f8dcd65f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/31/2025 02:19:02.743 7f8dcd65f2e8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/31/2025 02:19:02.743 7f8dcd65f2e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/31/2025 02:19:02.743 7f8dcd65f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/31/2025 02:19:02.744 7f8dcd65f2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/31/2025 02:19:02.744 7f8dcd65f2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/31/2025 02:19:02.744 7f8dcd65f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/31/2025 02:19:02.744 7f8dcd65f2e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/31/2025 02:19:02.744 7f8dcd65f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:02.744 7f8dcd65f2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/31/2025 02:19:02.744 7f8dcd65f2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:02.744 7f8dcd65f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:02.745 7f8dcd65f2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1753928330735950 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/31/2025 02:19:02.745 7f8dcd65f2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:06.744 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/31/2025 02:19:06.745 7f8dcd28ec18 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/31/2025 02:19:06.746 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/31/2025 02:19:06.746 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/31/2025 02:19:06.746 7f8dcd28ec18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/31/2025 02:19:06.746 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/31/2025 02:19:06.746 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/31/2025 02:19:06.746 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/31/2025 02:19:06.746 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:06.747 7f8dcd28ec18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/31/2025 02:19:06.747 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:06.747 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:06.747 7f8dcd28ec18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1753928330735950 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/31/2025 02:19:06.747 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:10.750 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/31/2025 02:19:10.751 7f8dcd28ec18 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/31/2025 02:19:10.751 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/31/2025 02:19:10.751 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/31/2025 02:19:10.751 7f8dcd28ec18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/31/2025 02:19:10.751 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/31/2025 02:19:10.752 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/31/2025 02:19:10.752 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/31/2025 02:19:10.752 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:10.752 7f8dcd28ec18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/31/2025 02:19:10.752 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:10.752 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:10.753 7f8dcd28ec18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1753928330735950 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/31/2025 02:19:10.753 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:14.754 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/31/2025 02:19:14.755 7f8dcd28ec18 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/31/2025 02:19:14.755 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/31/2025 02:19:14.755 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/31/2025 02:19:14.756 7f8dcd28ec18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/31/2025 02:19:14.756 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/31/2025 02:19:14.756 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/31/2025 02:19:14.756 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/31/2025 02:19:14.756 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:14.756 7f8dcd28ec18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/31/2025 02:19:14.756 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:14.757 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:14.757 7f8dcd28ec18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1753928330735950 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/31/2025 02:19:14.757 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:18.757 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/31/2025 02:19:18.758 7f8dcd28ec18 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/31/2025 02:19:18.758 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/31/2025 02:19:18.758 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/31/2025 02:19:18.758 7f8dcd28ec18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/31/2025 02:19:18.758 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/31/2025 02:19:18.759 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/31/2025 02:19:18.759 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/31/2025 02:19:18.759 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:18.759 7f8dcd28ec18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/31/2025 02:19:18.759 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:18.759 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:18.760 7f8dcd28ec18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1753928330735950 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/31/2025 02:19:18.760 7f8dcd28ec18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:22.760 7f8dcc411c18 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/31/2025 02:19:22.761 7f8dcc411c18 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/31/2025 02:19:22.761 7f8dcc411c18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/31/2025 02:19:22.761 7f8dcc411c18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/31/2025 02:19:22.762 7f8dcc411c18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/31/2025 02:19:22.762 7f8dcc411c18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/31/2025 02:19:22.762 7f8dcc411c18 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/31/2025 02:19:22.762 7f8dcc411c18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/31/2025 02:19:22.763 7f8dcc411c18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:22.763 7f8dcc411c18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/31/2025 02:19:22.763 7f8dcc411c18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:22.763 7f8dcc411c18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:22.764 7f8dcc411c18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1753928330735950 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/31/2025 02:19:22.764 7f8dcc411c18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/31/2025 02:19:26.763 7f8dcc411c18 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/31/2025 02:19:26.765 7f8dcc411c18 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 07/31/2025 02:19:26.765 7f8dcc411c18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/31/2025 02:19:26.765 7f8dcc411c18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/31/2025 02:19:26.765 7f8dcc411c18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3729-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/31/2025 02:19:26.765 7f8dcc411c18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/31/2025 02:19:26.765 7f8dcc411c18 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 07/31/2025 02:19:26.765 7f8dcc411c18 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 07/31/2025 02:19:26.766 7f8dcc411c18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 07/31/2025 02:19:26.766 7f8dcc411c18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 07/31/2025 02:19:26.766 7f8dcc411c18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 07/31/2025 02:19:35.305 7f8dcf859978 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001016 07/31/2025 02:19:35.306 7f8dcf859978 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 07/31/2025 02:19:35.306 7f8dcd5720f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 07/31/2025 02:19:35.306 7f8dcd5720f0 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 07/31/2025 02:19:35.306 7f8dcd5720f0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 07/31/2025 02:19:35.306 7f8dcd5720f0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 07/31/2025 02:19:35.307 7f8dcd5720f0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 07/31/2025 02:19:35.307 7f8dcd5720f0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 07/31/2025 02:19:35.307 7f8dcd5720f0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 07/31/2025 02:19:35.307 7f8dcd73d630 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 07/31/2025 02:19:35.309 7f8dcd73d630 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 07/31/2025 02:19:45.313 7f8dcf859978 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001582 07/31/2025 02:19:45.314 7f8dcf859978 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 07/31/2025 02:19:45.315 7f8dcd72ca20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 07/31/2025 02:19:45.315 7f8dcd72ca20 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 4 07/31/2025 02:19:45.315 7f8dcd72ca20 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 07/31/2025 02:19:45.315 7f8dcd72ca20 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 07/31/2025 02:19:45.315 7f8dcd72ca20 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 07/31/2025 02:19:45.315 7f8dcd72ca20 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 07/31/2025 02:19:45.316 7f8dcd9c6198 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 07/31/2025 02:19:45.316 7f8dcd9c6198 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 07/31/2025 02:19:51.491 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 07/31/2025 02:19:52.747 7f8dcfd17c88 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3729-1.', _node_type='Database', _instance_id='0', _timestamp=1753928392747010, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.50:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 07/31/2025 02:19:55.007 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3729-1.', _node_type='Database', _instance_id='0', _timestamp=1753928395007206, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=3), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 07/31/2025 02:20:00.895 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=16595332, partition_space_available_1k=64495152, percentage_partition_space_used=20, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1316996, cpu_share=0.34, mem_res=1300376, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10360, cpu_share=0.0, mem_res=7732, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52796, cpu_share=0.03, mem_res=40928, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=9018236, free=5647632, buffers=727328, cached=17271924, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.21, five_min_avg=2.81, fifteen_min_avg=1.71, cpu_share=4.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 07/31/2025 02:20:51.497 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 07/31/2025 02:21:00.893 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=17811756, partition_space_available_1k=63278728, percentage_partition_space_used=22, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1315984, cpu_share=0.12, mem_res=1301912, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10148, cpu_share=0.0, mem_res=7916, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43544, cpu_share=0.03, mem_res=41044, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=8999948, free=20241060, buffers=747360, cached=2676752, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.62, five_min_avg=2.91, fifteen_min_avg=1.82, cpu_share=4.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 07/31/2025 02:21:51.512 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 07/31/2025 02:22:00.901 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=17817208, partition_space_available_1k=63273276, percentage_partition_space_used=22, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1304068, cpu_share=0.1, mem_res=1287576, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10348, cpu_share=0.0, mem_res=8116, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43528, cpu_share=0.03, mem_res=41184, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=8882052, free=20288104, buffers=750340, cached=2744624, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.89, five_min_avg=2.85, fifteen_min_avg=1.87, cpu_share=3.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 07/31/2025 02:22:31.886 7f8dd11e8c18 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 07/31/2025 02:22:34.410 7f8dd11e8c18 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 07/31/2025 02:22:51.510 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 07/31/2025 02:23:00.909 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=18199764, partition_space_available_1k=62890720, percentage_partition_space_used=22, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1312848, cpu_share=0.13, mem_res=1294364, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10596, cpu_share=0.0, mem_res=8356, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43532, cpu_share=0.03, mem_res=41188, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=9624168, free=19062192, buffers=756528, cached=3222232, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.73, five_min_avg=2.82, fifteen_min_avg=1.92, cpu_share=2.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 07/31/2025 02:23:41.479 7f8dcd2b08d0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 07/31/2025 02:23:51.592 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 07/31/2025 02:24:00.922 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19375104, partition_space_available_1k=61715380, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1313644, cpu_share=0.13, mem_res=1293232, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10744, cpu_share=0.0, mem_res=8504, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43544, cpu_share=0.03, mem_res=41188, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10194412, free=17266096, buffers=786828, cached=4417784, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.07, five_min_avg=2.89, fifteen_min_avg=2.01, cpu_share=4.38, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 07/31/2025 02:24:51.585 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 07/31/2025 02:24:55.343 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3729-1.', _node_type='Database', _instance_id='0', _timestamp=1753928695343654, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=84), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 07/31/2025 02:25:00.947 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19407204, partition_space_available_1k=61683280, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1354132, cpu_share=0.21, mem_res=1325988, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=14864, cpu_share=0.01, mem_res=12520, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43552, cpu_share=0.03, mem_res=41188, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10267060, free=17132488, buffers=787524, cached=4478048, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.59, five_min_avg=3.33, fifteen_min_avg=2.21, cpu_share=5.12, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 07/31/2025 02:25:51.520 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 07/31/2025 02:25:59.935 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19437548, partition_space_available_1k=61652936, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1381732, cpu_share=0.13, mem_res=1350080, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=14784, cpu_share=0.0, mem_res=12452, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43556, cpu_share=0.03, mem_res=41188, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10918312, free=16442044, buffers=787580, cached=4517184, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.32, five_min_avg=3.53, fifteen_min_avg=2.35, cpu_share=4.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 07/31/2025 02:26:51.576 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 07/31/2025 02:27:00.952 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19468076, partition_space_available_1k=61622408, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1373212, cpu_share=0.17, mem_res=1336304, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=14888, cpu_share=0.0, mem_res=12556, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43564, cpu_share=0.03, mem_res=41188, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10411368, free=16905456, buffers=788140, cached=4560156, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.2, five_min_avg=3.62, fifteen_min_avg=2.46, cpu_share=5.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 07/31/2025 02:27:51.607 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 07/31/2025 02:28:00.957 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19498452, partition_space_available_1k=61592032, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1381292, cpu_share=0.13, mem_res=1340284, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=14968, cpu_share=0.0, mem_res=12636, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43560, cpu_share=0.03, mem_res=41188, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10424352, free=16861316, buffers=788192, cached=4591260, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.21, five_min_avg=3.88, fifteen_min_avg=2.61, cpu_share=4.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 07/31/2025 02:28:51.619 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 07/31/2025 02:29:00.985 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19587252, partition_space_available_1k=61503232, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1389244, cpu_share=0.13, mem_res=1344844, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=15016, cpu_share=0.0, mem_res=12684, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43564, cpu_share=0.03, mem_res=41188, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=11544708, free=15694872, buffers=788304, cached=4637236, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.28, five_min_avg=3.91, fifteen_min_avg=2.71, cpu_share=4.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 07/31/2025 02:29:51.553 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 07/31/2025 02:29:59.969 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19585204, partition_space_available_1k=61505280, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1397480, cpu_share=0.12, mem_res=1349552, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=15076, cpu_share=0.0, mem_res=12744, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43568, cpu_share=0.03, mem_res=41188, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=11106452, free=16107052, buffers=788336, cached=4663280, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.91, five_min_avg=3.9, fifteen_min_avg=2.78, cpu_share=4.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 07/31/2025 02:30:51.643 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 07/31/2025 02:31:00.974 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19606256, partition_space_available_1k=61484228, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1402748, cpu_share=0.1, mem_res=1351848, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=15088, cpu_share=0.0, mem_res=12756, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43572, cpu_share=0.03, mem_res=41188, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10542720, free=16644888, buffers=788408, cached=4689104, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.74, five_min_avg=3.87, fifteen_min_avg=2.84, cpu_share=4.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 07/31/2025 02:31:51.720 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 07/31/2025 02:31:55.301 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3729-1.', _node_type='Database', _instance_id='0', _timestamp=1753929115301423, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=87), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 07/31/2025 02:32:00.975 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19694480, partition_space_available_1k=61396004, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1411728, cpu_share=0.11, mem_res=1357308, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=15072, cpu_share=0.0, mem_res=12740, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43576, cpu_share=0.03, mem_res=41188, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10438568, free=16630380, buffers=788508, cached=4807664, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.37, five_min_avg=4.02, fifteen_min_avg=2.96, cpu_share=4.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 07/31/2025 02:32:51.585 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 07/31/2025 02:32:59.973 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19724448, partition_space_available_1k=61366036, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1426068, cpu_share=0.1, mem_res=1368912, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=15132, cpu_share=0.0, mem_res=12800, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43576, cpu_share=0.03, mem_res=41188, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10504256, free=16556584, buffers=788528, cached=4815752, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.08, five_min_avg=4.02, fifteen_min_avg=3.03, cpu_share=3.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 07/31/2025 02:33:51.587 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 07/31/2025 02:34:00.976 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19725844, partition_space_available_1k=61364640, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1422536, cpu_share=0.1, mem_res=1362516, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=15176, cpu_share=0.0, mem_res=12844, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43584, cpu_share=0.03, mem_res=41188, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10523600, free=16534608, buffers=788540, cached=4818372, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.92, five_min_avg=3.7, fifteen_min_avg=2.98, cpu_share=3.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 07/31/2025 02:34:04.490 7f8dcd65f2e8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 07/31/2025 02:34:51.540 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 07/31/2025 02:34:57.710 7f8dcd6cbf60 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 07/31/2025 02:34:59.983 7f8dcfd17c88 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3729-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19798676, partition_space_available_1k=61291808, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1428524, cpu_share=0.1, mem_res=1365780, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=15308, cpu_share=0.0, mem_res=12976, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43584, cpu_share=0.03, mem_res=41188, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10344936, free=16520412, buffers=791684, cached=5008088, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.36, five_min_avg=3.38, fifteen_min_avg=2.91, cpu_share=3.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo')