07/28/2025 02:21:23.539 7f757b4c2948 [contrail-config-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 07/28/2025 02:21:23.539 7f757b4c2948 [contrail-config-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 07/28/2025 02:21:23.539 7f757b4c2948 [contrail-config-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 07/28/2025 02:21:23.550 7f757871a8d0 [contrail-config-database-nodemgr] [ERROR]: Cannot write http_port 8112 to /tmp/contrail-config-database-nodemgr.0.http_port 07/28/2025 02:21:23.550 7f757871a8d0 [contrail-config-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8112 07/28/2025 02:21:23.550 7f7578721240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 07/28/2025 02:21:23.557 7f7578721240 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:21:23.558 7f7578721240 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 07/28/2025 02:21:23.558 7f7578721240 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 07/28/2025 02:21:23.558 7f7578721240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:23.558 7f7578721240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/28/2025 02:21:23.558 7f7578721240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:23.558 7f7578721240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:23.558 7f7578721240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 07/28/2025 02:21:23.558 7f7578721240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:23.606 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 29791629709540892749936774319767928529005213001326777440121096650328005441494, 'start': '1753669281000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'zookeeper', 'group': 'zookeeper', 'pid': 106880608109041154579224366673152780189900753111626781336154460057198940742604, 'start': '1753669281000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-config-database-nodemgr', 'group': 'contrail-config-database-nodemgr', 'pid': 43220259230840020260472400667773374014001897675319014293964582863045037756557, 'start': '1753669281000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 07/28/2025 02:21:23.754 7f7581738a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:23.754 7f7581738a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-07-28 00:44:25.362769", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10069-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 4 num_core_per_socket = 4 num_thread_per_core = 1 >> running_package_version = nightly-nightly.el7 installed_package_version = nightly-nightly.el7 >> 07/28/2025 02:21:23.755 7f7581738a20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:23.755 7f757b4c2948 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3717-3.', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1753669283755447, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 07/28/2025 02:21:23.755 7f757b4c2948 [contrail-config-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3717-3.', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1753669283755766, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1753669281000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='zookeeper', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1753669281000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-config-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1753669281000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-07-28 00:44:25.362769", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10069-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 07/28/2025 02:21:23.757 7f7578e81a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:23.757 7f7578e81a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/28/2025 02:21:23.757 7f7578e81a20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:23.757 7f7578e81a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:23.758 7f7578e81a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1753669281000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = zookeeper process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1753669281000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-config-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1753669281000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-07-28 00:44:25.362769", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10069-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 07/28/2025 02:21:23.758 7f7578e81a20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:24.026 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:21:24.277 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:DatabaseUsage] map 07/28/2025 02:21:24.278 7f757aff74e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:24.278 7f757aff74e0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. database_usage = [ << disk_space_used_1k = 4658956 disk_space_available_1k = 76431528 config_db_size_1k = 4 >>, ] >> 07/28/2025 02:21:24.278 7f757aff74e0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:24.781 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to connect to database by CQL and socket: Result: 1 07/28/2025 02:21:24.782 7f757b4c2948 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3717-3.', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1753669284782189, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 07/28/2025 02:21:24.783 7f7578a3e8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:24.783 7f7578a3e8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/28/2025 02:21:24.784 7f7578a3e8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:26.501 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:21:27.558 7f7578a3b438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:21:27.560 7f7578a3b438 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:21:27.560 7f7578a3b438 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:21:27.560 7f7578a3b438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:21:27.560 7f7578a3b438 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:21:27.560 7f7578a3b438 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:21:27.560 7f7578a3b438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:21:27.561 7f7578a3b438 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:21:27.561 7f7578a3b438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:27.561 7f7578a3b438 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:21:27.561 7f7578a3b438 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:27.561 7f7578a3b438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:27.561 7f7578a3b438 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:21:27.561 7f7578a3b438 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:29.044 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr config_db_uuid'. Error is: Result: 1 07/28/2025 02:21:29.924 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr to_bgp_keyspace'. Error is: Result: 1 07/28/2025 02:21:30.778 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr svc_monitor_keyspace'. Error is: Result: 1 07/28/2025 02:21:31.561 7f7578a4e6d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:21:31.562 7f7578a4e6d8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:21:31.563 7f7578a4e6d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:21:31.564 7f7578a4e6d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:21:31.564 7f7578a4e6d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:21:31.564 7f7578a4e6d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:21:31.564 7f7578a4e6d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:21:31.564 7f7578a4e6d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:21:31.565 7f7578a4e6d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:31.565 7f7578a4e6d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:21:31.565 7f7578a4e6d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:31.565 7f7578a4e6d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:31.565 7f7578a4e6d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:21:31.565 7f7578a4e6d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:31.630 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr DISCOVERY_SERVER'. Error is: Result: 1 07/28/2025 02:21:33.015 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr useragent'. Error is: Result: 1 07/28/2025 02:21:33.966 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr dm_keyspace'. Error is: Result: 1 07/28/2025 02:21:35.563 7f7578a348d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:21:35.565 7f7578a348d0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:21:35.565 7f7578a348d0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:21:35.565 7f7578a348d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:21:35.565 7f7578a348d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:21:35.565 7f7578a348d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:21:35.565 7f7578a348d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:21:35.566 7f7578a348d0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:21:35.566 7f7578a348d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:35.566 7f7578a348d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:21:35.566 7f7578a348d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:35.566 7f7578a348d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:35.566 7f7578a348d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:21:35.566 7f7578a348d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:39.013 7f757b4c2948 [contrail-config-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3717-3.', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1753669299012914, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1753669281000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='zookeeper', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1753669281000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-config-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1753669281000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-07-28 00:44:25.362769", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10069-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 07/28/2025 02:21:39.013 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=4945364, partition_space_available_1k=76145120, percentage_partition_space_used=6, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1238620, cpu_share=0.0, mem_res=1225724, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59996, cpu_share=0.0, mem_res=57216, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52280, cpu_share=0.0, mem_res=40084, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=1771476, free=9731800, buffers=137208, cached=4535156, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.36, five_min_avg=0.7, fifteen_min_avg=0.26, cpu_share=0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:21:39.013 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:39.013 7f7578a14240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1753669281000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = zookeeper process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1753669281000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-config-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1753669281000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-07-28 00:44:25.362769", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10069-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 07/28/2025 02:21:39.014 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:39.014 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:39.014 7f7578a14240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 4945364 partition_space_available_1k = 76145120 percentage_partition_space_used = 6 >> } all_core_file_list = [ ] process_mem_cpu_usage = { cassandra : << mem_virt = 1238620 cpu_share = 0.0 mem_res = 1225724 >>zookeeper : << mem_virt = 59996 cpu_share = 0.0 mem_res = 57216 >>contrail-config-database-nodemgr : << mem_virt = 52280 cpu_share = 0.0 mem_res = 40084 >> } system_mem_usage = << total = 16175640 used = 1771476 free = 9731800 buffers = 137208 cached = 4535156 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 2.36 five_min_avg = 0.7 fifteen_min_avg = 0.26 cpu_share = 0 node_type = config-database-node >> >> 07/28/2025 02:21:39.014 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:39.565 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:21:39.567 7f7578a14240 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:21:39.567 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:21:39.567 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:21:39.567 7f7578a14240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:21:39.567 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:21:39.567 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:21:39.567 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:21:39.568 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:39.568 7f7578a14240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:21:39.568 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:39.568 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:39.568 7f7578a14240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:21:39.568 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:43.571 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:21:43.572 7f7578a14240 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:21:43.572 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:21:43.572 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:21:43.573 7f7578a14240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:21:43.573 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:21:43.573 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:21:43.573 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:21:43.573 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:43.573 7f7578a14240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:21:43.573 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:43.573 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:43.574 7f7578a14240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:21:43.574 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:47.575 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:21:47.576 7f7578a14240 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:21:47.576 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:21:47.576 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:21:47.577 7f7578a14240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:21:47.577 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:21:47.577 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:21:47.577 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:21:47.577 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:47.577 7f7578a14240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:21:47.578 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:47.578 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:47.578 7f7578a14240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:21:47.578 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:51.579 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:21:51.580 7f7578a14240 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:21:51.581 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:21:51.581 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:21:51.581 7f7578a14240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:21:51.581 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:21:51.581 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:21:51.581 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:21:51.582 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:51.582 7f7578a14240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:21:51.582 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:51.582 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:51.582 7f7578a14240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:21:51.582 7f7578a14240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:55.583 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:21:55.585 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:21:55.585 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:21:55.585 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:21:55.585 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:21:55.586 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:21:55.586 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:21:55.586 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:21:55.586 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:55.586 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:21:55.586 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:55.586 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:55.587 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:21:55.587 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:59.587 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:21:59.589 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:21:59.589 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:21:59.589 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:21:59.589 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:21:59.589 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:21:59.590 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:21:59.590 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:21:59.590 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:59.590 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:21:59.590 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:59.591 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:21:59.591 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:21:59.591 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:03.591 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:22:03.593 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:22:03.593 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:22:03.594 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:03.594 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:22:03.594 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:03.594 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:22:03.594 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:22:03.594 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:03.595 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:22:03.595 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:03.595 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:03.595 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:22:03.595 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:07.595 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:22:07.597 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:22:07.597 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:22:07.597 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:07.597 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:22:07.598 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:07.598 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:22:07.598 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:22:07.598 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:07.598 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:22:07.599 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:07.599 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:07.599 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:22:07.599 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:11.599 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:22:11.600 7f758117c240 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:22:11.600 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:22:11.600 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:11.600 7f758117c240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:22:11.601 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:11.601 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:22:11.601 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:22:11.601 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:11.601 7f758117c240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:22:11.601 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:11.601 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:11.601 7f758117c240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:22:11.601 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:15.604 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:22:15.606 7f758117c240 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:22:15.607 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:22:15.608 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:15.609 7f758117c240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:22:15.609 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:15.609 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:22:15.609 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:22:15.609 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:15.610 7f758117c240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:22:15.610 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:15.611 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:15.611 7f758117c240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:22:15.611 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:19.607 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:22:19.608 7f758117c240 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:22:19.608 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:22:19.608 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:19.609 7f758117c240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:22:19.609 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:19.609 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:22:19.610 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:22:19.610 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:19.610 7f758117c240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:22:19.610 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:19.613 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:19.613 7f758117c240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:22:19.613 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:23.611 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:22:23.612 7f758117c240 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:22:23.612 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:22:23.613 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:23.613 7f758117c240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:22:23.613 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:23.613 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:22:23.613 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:22:23.613 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:23.613 7f758117c240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:22:23.614 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:23.614 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:23.614 7f758117c240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:22:23.614 7f758117c240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:23.911 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:22:24.054 7f7578ccecc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:24.055 7f7578ccecc0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. database_usage = [ << disk_space_used_1k = 5644776 disk_space_available_1k = 75445708 config_db_size_1k = 1364 >>, ] >> 07/28/2025 02:22:24.055 7f7578ccecc0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:24.407 7f757b4c2948 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3717-3.', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1753669344407754, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.61:8086'], status='Down', description='Connect to Idle on EvTcpConnectFail')]), flag_infos=None, description='Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 07/28/2025 02:22:24.409 7f7578a30b70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:24.409 7f7578a30b70 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:22:24.409 7f7578a30b70 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:25.175 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:22:27.614 7f7578a30f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:22:27.616 7f7578a30f60 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:22:27.616 7f7578a30f60 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:22:27.616 7f7578a30f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:27.617 7f7578a30f60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:22:27.617 7f7578a30f60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:27.617 7f7578a30f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:22:27.617 7f7578a30f60 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:22:27.617 7f7578a30f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:27.617 7f7578a30f60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:22:27.617 7f7578a30f60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:27.617 7f7578a30f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:27.618 7f7578a30f60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:22:27.618 7f7578a30f60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:30.210 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=5644956, partition_space_available_1k=75445528, percentage_partition_space_used=7, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1329224, cpu_share=0.48, mem_res=1310864, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=72428, cpu_share=0.01, mem_res=69276, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52388, cpu_share=0.01, mem_res=40084, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=2217756, free=8712228, buffers=197644, cached=5048012, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.0, five_min_avg=1.11, fifteen_min_avg=0.42, cpu_share=10.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:22:30.210 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:30.210 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 5644956 partition_space_available_1k = 75445528 percentage_partition_space_used = 7 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1329224 cpu_share = 0.48 mem_res = 1310864 >>zookeeper : << mem_virt = 72428 cpu_share = 0.01 mem_res = 69276 >>contrail-config-database-nodemgr : << mem_virt = 52388 cpu_share = 0.01 mem_res = 40084 >> } system_mem_usage = << total = 16175640 used = 2217756 free = 8712228 buffers = 197644 cached = 5048012 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 3.0 five_min_avg = 1.11 fifteen_min_avg = 0.42 cpu_share = 10.0 node_type = config-database-node >> >> 07/28/2025 02:22:30.210 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:31.618 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:22:31.619 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:22:31.620 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:22:31.620 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:31.620 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:22:31.620 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:31.620 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:22:31.621 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:22:31.621 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:31.621 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:22:31.621 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:31.621 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:31.621 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:22:31.621 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:35.623 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:22:35.624 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:22:35.624 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:22:35.624 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:35.625 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:22:35.625 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:35.625 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:22:35.625 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:22:35.625 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:35.625 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:22:35.626 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:35.626 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:35.626 7f7578cd3390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:22:35.626 7f7578cd3390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:39.626 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:22:39.628 7f7578a33e10 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:22:39.628 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:22:39.628 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:39.628 7f7578a33e10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:22:39.628 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:39.628 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:22:39.629 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:22:39.629 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:39.629 7f7578a33e10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:22:39.629 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:39.629 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:39.629 7f7578a33e10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:22:39.629 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:43.631 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:22:43.632 7f7578a33e10 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:22:43.632 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:22:43.633 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:43.633 7f7578a33e10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:22:43.633 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:43.633 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:22:43.633 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:22:43.633 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:43.633 7f7578a33e10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:22:43.634 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:43.634 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:43.634 7f7578a33e10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:22:43.634 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:47.635 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:22:47.636 7f7578a33e10 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:22:47.637 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:22:47.637 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:47.637 7f7578a33e10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:22:47.637 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:47.637 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:22:47.638 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:22:47.638 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:47.638 7f7578a33e10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:22:47.638 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:47.638 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:47.638 7f7578a33e10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:22:47.638 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:51.640 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:22:51.641 7f7578a33e10 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:22:51.641 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:22:51.641 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:51.641 7f7578a33e10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:22:51.641 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:51.641 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:22:51.642 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:22:51.642 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:51.642 7f7578a33e10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:22:51.642 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:51.642 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:51.642 7f7578a33e10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:22:51.642 7f7578a33e10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:55.643 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:22:55.644 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:22:55.645 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:22:55.645 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:55.645 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:22:55.645 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:55.645 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:22:55.646 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:22:55.646 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:55.646 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:22:55.646 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:55.646 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:55.647 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:22:55.647 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:59.647 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:22:59.648 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:22:59.648 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:22:59.648 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:59.649 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:22:59.649 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:22:59.649 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:22:59.649 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:22:59.649 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:59.649 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:22:59.649 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:59.649 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:22:59.650 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:22:59.650 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:03.652 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:23:03.653 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:23:03.654 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:23:03.654 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:03.654 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:23:03.654 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:03.654 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:23:03.654 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:23:03.655 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:03.655 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:23:03.655 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:03.655 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:03.655 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:23:03.655 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:07.655 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:23:07.656 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:23:07.657 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:23:07.657 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:07.657 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:23:07.657 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:07.657 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:23:07.658 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:23:07.658 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:07.658 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:23:07.658 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:07.658 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:07.658 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:23:07.659 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:11.660 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:23:11.661 7f7577858a20 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:23:11.661 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:23:11.661 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:11.662 7f7577858a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:23:11.662 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:11.662 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:23:11.662 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:23:11.662 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:11.663 7f7577858a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:23:11.663 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:11.663 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:11.663 7f7577858a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:23:11.663 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:15.665 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:23:15.666 7f7577858a20 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:23:15.666 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:23:15.667 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:15.667 7f7577858a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:23:15.667 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:15.667 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:23:15.667 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:23:15.667 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:15.668 7f7577858a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:23:15.668 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:15.668 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:15.668 7f7577858a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:23:15.668 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:19.671 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:23:19.673 7f7577858a20 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:23:19.673 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:23:19.673 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:19.673 7f7577858a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:23:19.674 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:19.674 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:23:19.674 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:23:19.674 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:19.674 7f7577858a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:23:19.674 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:19.674 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:19.675 7f7577858a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:23:19.675 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:23.675 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:23:23.676 7f7577858a20 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:23:23.676 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:23:23.677 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:23.677 7f7577858a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:23:23.677 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:23.677 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:23:23.677 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:23:23.677 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:23.677 7f7577858a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:23:23.678 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:23.678 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:23.678 7f7577858a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:23:23.678 7f7577858a20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:24.089 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:23:25.122 7f75806d1780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:25.122 7f75806d1780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. database_usage = [ << disk_space_used_1k = 6074968 disk_space_available_1k = 75015516 config_db_size_1k = 1444 >>, ] >> 07/28/2025 02:23:25.123 7f75806d1780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:27.407 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:23:27.677 7f7577b9ca20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:23:27.680 7f7577b9ca20 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:23:27.681 7f7577b9ca20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:23:27.681 7f7577b9ca20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:27.681 7f7577b9ca20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:23:27.681 7f7577b9ca20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:27.681 7f7577b9ca20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:23:27.682 7f7577b9ca20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:23:27.682 7f7577b9ca20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:27.682 7f7577b9ca20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:23:27.682 7f7577b9ca20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:27.682 7f7577b9ca20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:27.682 7f7577b9ca20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:23:27.682 7f7577b9ca20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:31.682 7f757be9c630 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:23:31.683 7f757be9c630 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:23:31.683 7f757be9c630 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:23:31.683 7f757be9c630 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:31.683 7f757be9c630 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:23:31.683 7f757be9c630 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:31.684 7f757be9c630 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:23:31.684 7f757be9c630 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:23:31.684 7f757be9c630 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:31.684 7f757be9c630 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:23:31.684 7f757be9c630 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:31.685 7f757be9c630 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:31.685 7f757be9c630 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:23:31.685 7f757be9c630 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:32.687 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=6089644, partition_space_available_1k=75000840, percentage_partition_space_used=8, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1673428, cpu_share=0.33, mem_res=1645776, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=78544, cpu_share=0.01, mem_res=75220, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52532, cpu_share=0.01, mem_res=40084, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=3040116, free=7099800, buffers=222184, cached=5813540, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.47, five_min_avg=1.29, fifteen_min_avg=0.53, cpu_share=8.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:23:32.687 7f7578a3e390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:32.687 7f7578a3e390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 6089644 partition_space_available_1k = 75000840 percentage_partition_space_used = 8 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1673428 cpu_share = 0.33 mem_res = 1645776 >>zookeeper : << mem_virt = 78544 cpu_share = 0.01 mem_res = 75220 >>contrail-config-database-nodemgr : << mem_virt = 52532 cpu_share = 0.01 mem_res = 40084 >> } system_mem_usage = << total = 16175640 used = 3040116 free = 7099800 buffers = 222184 cached = 5813540 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 2.47 five_min_avg = 1.29 fifteen_min_avg = 0.53 cpu_share = 8.25 node_type = config-database-node >> >> 07/28/2025 02:23:32.687 7f7578a3e390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:35.686 7f7578a3e390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:23:35.687 7f7578a3e390 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:23:35.687 7f7578a3e390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:23:35.687 7f7578a3e390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:35.687 7f7578a3e390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:23:35.688 7f7578a3e390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:35.688 7f7578a3e390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:23:35.688 7f7578a3e390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:23:35.688 7f7578a3e390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:35.688 7f7578a3e390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:23:35.689 7f7578a3e390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:35.689 7f7578a3e390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:35.689 7f7578a3e390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:23:35.689 7f7578a3e390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:39.688 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:23:39.689 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:23:39.689 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:23:39.689 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:39.689 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:23:39.689 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:39.689 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:23:39.690 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:23:39.690 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:39.690 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:23:39.690 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:39.690 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:39.690 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:23:39.690 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:43.694 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:23:43.695 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:23:43.695 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:23:43.695 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:43.695 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:23:43.695 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:43.696 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:23:43.696 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:23:43.696 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:43.696 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:23:43.696 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:43.696 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:43.696 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:23:43.696 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:47.699 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:23:47.701 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:23:47.701 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:23:47.701 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:47.701 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:23:47.701 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:47.701 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:23:47.702 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:23:47.702 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:47.702 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:23:47.702 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:47.703 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:47.703 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:23:47.703 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:51.701 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:23:51.703 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:23:51.703 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:23:51.703 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:51.703 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:23:51.703 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:51.703 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:23:51.704 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:23:51.704 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:51.704 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:23:51.704 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:51.704 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:51.704 7f75777736d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:23:51.704 7f75777736d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:55.703 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:23:55.705 7f7577778390 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:23:55.705 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:23:55.705 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:55.706 7f7577778390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:23:55.706 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:55.706 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:23:55.707 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:23:55.707 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:55.707 7f7577778390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:23:55.707 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:55.708 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:55.708 7f7577778390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:23:55.708 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:59.707 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:23:59.708 7f7577778390 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:23:59.708 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:23:59.708 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:59.709 7f7577778390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:23:59.709 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:23:59.709 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:23:59.709 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:23:59.709 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:59.709 7f7577778390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:23:59.709 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:59.709 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:23:59.710 7f7577778390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:23:59.710 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:03.711 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:24:03.713 7f7577778390 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:24:03.713 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:24:03.713 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:24:03.713 7f7577778390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:24:03.713 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:24:03.713 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:24:03.714 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:24:03.714 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:03.714 7f7577778390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:24:03.714 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:03.714 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:03.715 7f7577778390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:24:03.715 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:07.715 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:24:07.717 7f7577778390 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:24:07.718 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:24:07.718 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:24:07.718 7f7577778390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:24:07.718 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:24:07.718 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:24:07.718 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:24:07.718 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:07.719 7f7577778390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:24:07.719 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:07.719 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:07.719 7f7577778390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:24:07.719 7f7577778390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:11.719 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:24:11.720 7f757871f2e8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:24:11.720 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:24:11.721 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:24:11.721 7f757871f2e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:24:11.721 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:24:11.721 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:24:11.722 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:24:11.722 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:11.722 7f757871f2e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:24:11.722 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:11.722 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:11.722 7f757871f2e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:24:11.723 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:15.725 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:24:15.726 7f757871f2e8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:24:15.727 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:24:15.727 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:24:15.727 7f757871f2e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:24:15.727 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:24:15.727 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:24:15.727 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:24:15.728 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:15.728 7f757871f2e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:24:15.728 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:15.728 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:15.728 7f757871f2e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:24:15.728 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:19.731 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:24:19.733 7f757871f2e8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:24:19.733 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:24:19.734 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:24:19.734 7f757871f2e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:24:19.734 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:24:19.734 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:24:19.734 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:24:19.735 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:19.735 7f757871f2e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:24:19.735 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:19.735 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:19.735 7f757871f2e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:24:19.735 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:23.735 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:24:23.736 7f757871f2e8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:24:23.736 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:24:23.736 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:24:23.737 7f757871f2e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:24:23.737 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:24:23.737 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:24:23.737 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:24:23.737 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:23.737 7f757871f2e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:24:23.737 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:23.737 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:23.738 7f757871f2e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:24:23.738 7f757871f2e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:23.951 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:24:24.141 7f7578ccea20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:24.141 7f7578ccea20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. database_usage = [ << disk_space_used_1k = 7014124 disk_space_available_1k = 74076360 config_db_size_1k = 1624 >>, ] >> 07/28/2025 02:24:24.141 7f7578ccea20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:25.545 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:24:27.738 7f7578a30a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:24:27.740 7f7578a30a20 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:24:27.740 7f7578a30a20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:24:27.740 7f7578a30a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:24:27.740 7f7578a30a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:24:27.740 7f7578a30a20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:24:27.740 7f7578a30a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:24:27.741 7f7578a30a20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:24:27.741 7f7578a30a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:27.741 7f7578a30a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:24:27.741 7f7578a30a20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:27.741 7f7578a30a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:27.741 7f7578a30a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:24:27.741 7f7578a30a20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:30.581 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7014332, partition_space_available_1k=74076152, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1709564, cpu_share=0.38, mem_res=1680168, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=88884, cpu_share=0.01, mem_res=85632, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52628, cpu_share=0.01, mem_res=40084, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=4373988, free=4976708, buffers=244776, cached=6580168, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.91, five_min_avg=1.34, fifteen_min_avg=0.6, cpu_share=8.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:24:30.581 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:30.581 7f7578a4e8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 7014332 partition_space_available_1k = 74076152 percentage_partition_space_used = 9 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1709564 cpu_share = 0.38 mem_res = 1680168 >>zookeeper : << mem_virt = 88884 cpu_share = 0.01 mem_res = 85632 >>contrail-config-database-nodemgr : << mem_virt = 52628 cpu_share = 0.01 mem_res = 40084 >> } system_mem_usage = << total = 16175640 used = 4373988 free = 4976708 buffers = 244776 cached = 6580168 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 1.91 five_min_avg = 1.34 fifteen_min_avg = 0.6 cpu_share = 8.25 node_type = config-database-node >> >> 07/28/2025 02:24:30.582 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:31.741 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:24:31.743 7f7578a4e8d0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:24:31.743 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:24:31.743 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:24:31.744 7f7578a4e8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:24:31.744 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:24:31.744 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:24:31.744 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:24:31.744 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:31.744 7f7578a4e8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:24:31.744 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:31.745 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:31.745 7f7578a4e8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:24:31.745 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:35.746 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:24:35.747 7f7578a4e8d0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:24:35.747 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:24:35.747 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:24:35.747 7f7578a4e8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:24:35.747 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:24:35.747 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:24:35.748 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:24:35.748 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:35.748 7f7578a4e8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:24:35.748 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:35.748 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:35.748 7f7578a4e8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:24:35.748 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:39.751 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:24:39.752 7f7578a4e8d0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/28/2025 02:24:39.753 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:24:39.753 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:24:39.753 7f7578a4e8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:24:39.753 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:24:39.753 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/28/2025 02:24:39.753 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/28/2025 02:24:39.754 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:39.754 7f7578a4e8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/28/2025 02:24:39.754 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:39.754 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:39.754 7f7578a4e8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1753669283557958 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, 10.0.0.249:8086, 10.0.0.61:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/28/2025 02:24:39.754 7f7578a4e8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/28/2025 02:24:43.755 7f7579192b70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/28/2025 02:24:43.757 7f7579192b70 [contrail-config-database-nodemgr] [INFO]: Session Event: TCP Connected 07/28/2025 02:24:43.757 7f7579192b70 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/28/2025 02:24:43.757 7f7579192b70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:24:43.757 7f7579192b70 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3717-3. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/28/2025 02:24:43.757 7f7579192b70 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/28/2025 02:24:43.757 7f7579192b70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 07/28/2025 02:24:43.758 7f7579192b70 [contrail-config-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 07/28/2025 02:24:43.758 7f7579192b70 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 07/28/2025 02:24:43.758 7f7579192b70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 07/28/2025 02:24:43.758 7f7579192b70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 07/28/2025 02:24:55.366 7f757af37240 [contrail-config-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001016 07/28/2025 02:24:55.367 7f757af37240 [contrail-config-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 07/28/2025 02:24:55.367 7f7579192b70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 07/28/2025 02:24:55.367 7f7579192b70 [contrail-config-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 07/28/2025 02:24:55.368 7f7579192b70 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:DatabaseUsage] map 07/28/2025 02:24:55.369 7f7579192b70 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:24:55.369 7f7579192b70 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 07/28/2025 02:24:55.369 7f7579192b70 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 07/28/2025 02:24:55.369 7f7578a64b70 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 07/28/2025 02:24:55.370 7f7578a64b70 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 07/28/2025 02:25:05.709 7f757af37240 [contrail-config-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001439 07/28/2025 02:25:05.710 7f757af37240 [contrail-config-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 07/28/2025 02:25:05.710 7f757aff74e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 07/28/2025 02:25:05.710 7f757aff74e0 [contrail-config-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 3 07/28/2025 02:25:05.710 7f757aff74e0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:DatabaseUsage] map 07/28/2025 02:25:05.711 7f757aff74e0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:25:05.711 7f757aff74e0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 07/28/2025 02:25:05.711 7f7579163240 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 07/28/2025 02:25:05.711 7f7579163240 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 07/28/2025 02:25:23.994 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:25:25.931 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:25:30.971 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7526944, partition_space_available_1k=73563540, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1707652, cpu_share=0.07, mem_res=1678552, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=142720, cpu_share=0.04, mem_res=138908, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52624, cpu_share=0.01, mem_res=40112, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5573428, free=3187536, buffers=270388, cached=7144288, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.19, five_min_avg=1.84, fifteen_min_avg=0.82, cpu_share=10.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:26:24.013 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:26:25.957 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:26:31.006 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7530400, partition_space_available_1k=73560084, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1716360, cpu_share=0.05, mem_res=1687288, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=145724, cpu_share=0.01, mem_res=141868, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52668, cpu_share=0.01, mem_res=40152, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5592304, free=3164708, buffers=270464, cached=7148164, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.58, five_min_avg=1.88, fifteen_min_avg=0.89, cpu_share=5.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:27:24.095 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:27:26.171 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:27:28.960 7f7578a334e0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:27:31.212 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7952576, partition_space_available_1k=73137908, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1707912, cpu_share=0.05, mem_res=1678720, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=145744, cpu_share=0.0, mem_res=141884, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52684, cpu_share=0.01, mem_res=40156, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5656660, free=2714692, buffers=275788, cached=7528500, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.99, five_min_avg=1.81, fifteen_min_avg=0.93, cpu_share=6.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:27:47.659 7f757af37978 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:28:07.770 7f7578a34828 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:28:24.015 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:28:25.880 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:28:26.841 7f757aff74e0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:28:30.922 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7899724, partition_space_available_1k=73190760, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1711568, cpu_share=0.05, mem_res=1681536, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=148544, cpu_share=0.0, mem_res=144688, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52692, cpu_share=0.01, mem_res=40156, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5687576, free=2678164, buffers=275960, cached=7533940, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.52, five_min_avg=1.7, fifteen_min_avg=0.95, cpu_share=5.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:28:45.542 7f7579192b70 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:29:04.509 7f75778678d0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:29:24.122 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:29:24.621 7f7578a64240 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:29:25.822 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:29:30.862 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7902688, partition_space_available_1k=73187796, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1724020, cpu_share=0.05, mem_res=1693328, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=148568, cpu_share=0.0, mem_res=144708, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52688, cpu_share=0.01, mem_res=40156, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5714196, free=2645968, buffers=276380, cached=7539096, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.64, five_min_avg=1.67, fifteen_min_avg=0.99, cpu_share=6.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:29:44.024 7f7577781198 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:30:03.617 7f7577ba5b70 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:30:22.762 7f757be9cc18 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:30:24.033 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:30:25.803 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:30:30.842 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7909328, partition_space_available_1k=73181156, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1776996, cpu_share=0.2, mem_res=1742936, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=148592, cpu_share=0.0, mem_res=144736, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52696, cpu_share=0.01, mem_res=40156, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5780672, free=2571016, buffers=277516, cached=7546436, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.26, five_min_avg=1.53, fifteen_min_avg=0.98, cpu_share=6.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:30:43.155 7f757add82e8 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:31:02.335 7f757871aeb8 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:31:21.221 7f757be90ac8 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:31:24.088 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:31:25.844 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:31:30.900 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7915024, partition_space_available_1k=73175460, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1839040, cpu_share=0.15, mem_res=1802948, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=148648, cpu_share=0.0, mem_res=144788, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52740, cpu_share=0.02, mem_res=40192, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5839540, free=2504812, buffers=278560, cached=7552728, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.8, five_min_avg=1.65, fifteen_min_avg=1.06, cpu_share=6.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:31:40.431 7f757c5e8e10 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:32:00.636 7f7577b9f828 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:32:19.703 7f7578a4ec18 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:32:24.039 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:32:25.897 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:32:30.938 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7921632, partition_space_available_1k=73168852, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1848220, cpu_share=0.12, mem_res=1810152, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=148664, cpu_share=0.0, mem_res=144788, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52752, cpu_share=0.01, mem_res=40204, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5902832, free=2432816, buffers=279600, cached=7560392, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.42, five_min_avg=1.57, fifteen_min_avg=1.07, cpu_share=6.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:32:38.997 7f757b1f12e8 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:32:58.273 7f75806c8f60 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:33:18.647 7f757871f0f0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:33:24.048 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:33:25.783 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:33:30.817 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7928124, partition_space_available_1k=73162360, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1875264, cpu_share=0.12, mem_res=1834216, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=148664, cpu_share=0.0, mem_res=144788, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52788, cpu_share=0.01, mem_res=40228, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5927296, free=2400056, buffers=280576, cached=7567712, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.91, five_min_avg=1.4, fifteen_min_avg=1.04, cpu_share=5.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:33:37.728 7f757ae49630 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:33:56.809 7f7578e9a630 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:34:16.245 7f75806c8f60 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:34:24.069 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:34:25.774 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:34:30.812 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7934264, partition_space_available_1k=73156220, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1890432, cpu_share=0.12, mem_res=1847232, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=148668, cpu_share=0.0, mem_res=144792, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52792, cpu_share=0.01, mem_res=40228, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5946748, free=2373300, buffers=281432, cached=7574160, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.28, five_min_avg=1.41, fifteen_min_avg=1.07, cpu_share=5.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:34:36.974 7f757b327a20 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:34:56.106 7f7577b98438 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:35:15.505 7f757b29b048 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:35:24.036 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:35:25.808 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:35:30.847 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7939828, partition_space_available_1k=73150656, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1919508, cpu_share=0.11, mem_res=1874332, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=150372, cpu_share=0.0, mem_res=146496, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52800, cpu_share=0.01, mem_res=40232, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5991200, free=2321804, buffers=282132, cached=7580504, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.05, five_min_avg=1.33, fifteen_min_avg=1.06, cpu_share=5.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:35:34.925 7f757871f390 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:35:55.370 7f757b259438 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:36:15.055 7f7578a21390 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:36:24.056 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:36:25.765 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:36:30.805 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7944564, partition_space_available_1k=73145920, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1951012, cpu_share=0.11, mem_res=1904120, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=150388, cpu_share=0.0, mem_res=146512, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52804, cpu_share=0.01, mem_res=40232, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6016524, free=2290176, buffers=282840, cached=7586100, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.7, five_min_avg=1.18, fifteen_min_avg=1.02, cpu_share=5.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:36:34.385 7f757ae49630 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:36:53.302 7f7577867198 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:37:13.829 7f7577867f60 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:37:24.052 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:37:25.704 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:37:30.742 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7950668, partition_space_available_1k=73139816, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1980152, cpu_share=0.11, mem_res=1931000, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=164396, cpu_share=0.01, mem_res=160484, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52808, cpu_share=0.01, mem_res=40232, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6070680, free=2228832, buffers=283364, cached=7592764, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.72, five_min_avg=1.09, fifteen_min_avg=1.0, cpu_share=6.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:37:33.024 7f75791926d8 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:37:52.082 7f757b31c588 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:38:11.311 7f757b24e048 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:38:24.093 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:38:25.752 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:38:30.787 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7956788, partition_space_available_1k=73133696, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2015576, cpu_share=0.1, mem_res=1963160, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=164400, cpu_share=0.0, mem_res=160484, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52832, cpu_share=0.01, mem_res=40240, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6139712, free=2151752, buffers=283892, cached=7600284, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.91, five_min_avg=1.08, fifteen_min_avg=1.0, cpu_share=5.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:38:31.989 7f7578debc18 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:38:51.267 7f757b24e048 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:39:10.765 7f7578a3e240 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:39:24.112 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:39:25.880 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:39:30.058 7f757776ceb8 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:39:30.921 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7963900, partition_space_available_1k=73126584, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2058140, cpu_share=0.12, mem_res=2003080, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=164400, cpu_share=0.0, mem_res=160484, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52828, cpu_share=0.01, mem_res=40240, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6168800, free=2114492, buffers=284384, cached=7607964, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.39, five_min_avg=1.19, fifteen_min_avg=1.04, cpu_share=5.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:39:50.824 7f757776c2e8 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:40:10.403 7f757b31c588 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:40:24.134 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:40:25.699 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:40:29.915 7f7578a3e630 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:40:30.735 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7970648, partition_space_available_1k=73119836, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2109308, cpu_share=0.12, mem_res=2051396, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=164396, cpu_share=0.0, mem_res=160484, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52836, cpu_share=0.01, mem_res=40240, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6250848, free=2024512, buffers=284792, cached=7615488, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.26, five_min_avg=1.45, fifteen_min_avg=1.14, cpu_share=5.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:40:49.071 7f7577b9cf60 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:41:10.273 7f757871aeb8 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:41:24.123 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:41:25.842 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:41:29.880 7f7578a21ac8 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:41:30.895 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7977372, partition_space_available_1k=73113112, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2141420, cpu_share=0.13, mem_res=2080808, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=164400, cpu_share=0.0, mem_res=160484, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52840, cpu_share=0.01, mem_res=40240, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6283724, free=1983888, buffers=285064, cached=7622964, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.98, five_min_avg=1.53, fifteen_min_avg=1.19, cpu_share=6.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:41:49.222 7f757b51b8d0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:42:08.693 7f75820dd0f0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:42:24.090 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:42:25.809 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:42:29.510 7f7578a3ed68 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:42:30.851 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7983424, partition_space_available_1k=73107060, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2155768, cpu_share=0.11, mem_res=2093004, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=164396, cpu_share=0.0, mem_res=160484, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52844, cpu_share=0.01, mem_res=40240, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6299428, free=1961120, buffers=285280, cached=7629812, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.51, five_min_avg=1.49, fifteen_min_avg=1.19, cpu_share=5.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:42:48.745 7f7578a5dd68 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:43:08.390 7f757871aeb8 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:43:24.126 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:43:25.961 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:43:27.700 7f758117c240 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:43:31.008 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7999440, partition_space_available_1k=73091044, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2165468, cpu_share=0.12, mem_res=2099440, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=164400, cpu_share=0.0, mem_res=160484, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52852, cpu_share=0.01, mem_res=40240, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6309924, free=1941592, buffers=285540, cached=7638584, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.28, five_min_avg=1.42, fifteen_min_avg=1.18, cpu_share=5.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:43:48.310 7f757871aeb8 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:44:07.764 7f757788f630 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:44:24.129 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:44:26.022 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:44:27.167 7f757b51bb70 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:44:31.058 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8006288, partition_space_available_1k=73084196, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2173864, cpu_share=0.13, mem_res=2105328, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=164396, cpu_share=0.0, mem_res=160484, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52856, cpu_share=0.01, mem_res=40240, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6323740, free=1920476, buffers=285728, cached=7645696, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.2, five_min_avg=1.36, fifteen_min_avg=1.18, cpu_share=5.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:44:46.507 7f7578721588 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:45:07.383 7f757b24e048 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:45:24.117 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:45:26.094 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:45:26.655 7f7578a34240 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:45:31.139 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8014740, partition_space_available_1k=73075744, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2176140, cpu_share=0.1, mem_res=2104932, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=164400, cpu_share=0.0, mem_res=160484, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52864, cpu_share=0.01, mem_res=40240, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6333148, free=1903572, buffers=285932, cached=7652988, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.67, five_min_avg=1.48, fifteen_min_avg=1.23, cpu_share=5.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:45:45.947 7f7577867f60 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:46:05.288 7f757c72bc18 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:46:24.168 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:46:26.029 7f7578a33d68 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:46:26.129 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:46:31.179 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8022744, partition_space_available_1k=73067740, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2180608, cpu_share=0.1, mem_res=2106944, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=164404, cpu_share=0.0, mem_res=160484, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52864, cpu_share=0.01, mem_res=40240, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6335096, free=1894672, buffers=286112, cached=7659760, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.09, five_min_avg=1.34, fifteen_min_avg=1.2, cpu_share=5.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:46:45.369 7f7578a34588 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:47:04.627 7f7578a5de10 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:47:13.438 7f757b51b8d0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:47:21.986 7f757b31c588 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 07/28/2025 02:47:24.120 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:47:25.902 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:47:30.942 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8032504, partition_space_available_1k=73057980, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2194408, cpu_share=0.11, mem_res=2118044, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=164396, cpu_share=0.0, mem_res=160484, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52876, cpu_share=0.01, mem_res=40240, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6351428, free=1871384, buffers=286312, cached=7666516, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.02, five_min_avg=1.26, fifteen_min_avg=1.18, cpu_share=5.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:48:24.131 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 07/28/2025 02:48:25.849 7f757b4c2948 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 07/28/2025 02:48:30.884 7f757b4c2948 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3717-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8040416, partition_space_available_1k=73050068, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2207756, cpu_share=0.12, mem_res=2128076, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=164404, cpu_share=0.0, mem_res=160484, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52876, cpu_share=0.01, mem_res=40240, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6378780, free=1835288, buffers=286416, cached=7675156, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.0, five_min_avg=1.22, fifteen_min_avg=1.17, cpu_share=5.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 07/28/2025 02:48:32.566 7f757b327a20 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map