03/31/2025 02:14:13.708 7f3b7ae92688 [contrail-config-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 03/31/2025 02:14:13.709 7f3b7ae92688 [contrail-config-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 03/31/2025 02:14:13.709 7f3b7ae92688 [contrail-config-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 03/31/2025 02:14:13.721 7f3b7acd0630 [contrail-config-database-nodemgr] [ERROR]: Cannot write http_port 8112 to /tmp/contrail-config-database-nodemgr.0.http_port 03/31/2025 02:14:13.722 7f3b7acd0630 [contrail-config-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8112 03/31/2025 02:14:13.722 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 03/31/2025 02:14:13.723 7f3b7acd4f60 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:14:13.723 7f3b7acd4f60 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/31/2025 02:14:13.723 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 03/31/2025 02:14:13.723 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:13.724 7f3b7acd4f60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/31/2025 02:14:13.724 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:13.724 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:13.724 7f3b7acd4f60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 03/31/2025 02:14:13.724 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:13.794 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 113370157922811932021652176638088914642877587018928507921216205148212741384747, 'start': '1743387250000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'zookeeper', 'group': 'zookeeper', 'pid': 110288921287738532081123864175437181357044470361072435697421676703147985874057, 'start': '1743387250000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-config-database-nodemgr', 'group': 'contrail-config-database-nodemgr', 'pid': 78741151415527806699669221234284388994238570717781533623148772753290839720556, 'start': '1743387250000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 03/31/2025 02:14:13.959 7f3b810b9a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:13.960 7f3b810b9a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-03-31 00:53:57.142496", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8423-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 8 num_core_per_socket = 8 num_thread_per_core = 1 >> running_package_version = nightly-nightly.el7 installed_package_version = nightly-nightly.el7 >> 03/31/2025 02:14:13.960 7f3b810b9a20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:13.975 7f3b7ae92688 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-494-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1743387253975443, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/31/2025 02:14:13.976 7f3b7ae92688 [contrail-config-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-494-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1743387253976063, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1743387250000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='zookeeper', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1743387250000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-config-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1743387250000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-03-31 00:53:57.142496", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8423-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/31/2025 02:14:13.979 7f3b771f66d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:13.979 7f3b771f66d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/31/2025 02:14:13.979 7f3b771f66d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:13.979 7f3b771f66d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:13.980 7f3b771f66d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1743387250000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = zookeeper process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1743387250000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-config-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1743387250000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-03-31 00:53:57.142496", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8423-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 03/31/2025 02:14:13.980 7f3b771f66d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:14.281 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:14:14.609 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:DatabaseUsage] map 03/31/2025 02:14:14.611 7f3b7851e2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:14.611 7f3b7851e2e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 database_usage = [ << disk_space_used_1k = 50208964 disk_space_available_1k = 30881520 config_db_size_1k = 4 >>, ] >> 03/31/2025 02:14:14.611 7f3b7851e2e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:15.790 7f3b7ae92688 [contrail-config-database-nodemgr] [ERROR]: Failed to connect to database by CQL: Result: 1 03/31/2025 02:14:15.791 7f3b7ae92688 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-494-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1743387255791083, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/31/2025 02:14:15.794 7f3b7851e2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:15.794 7f3b7851e2e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/31/2025 02:14:15.794 7f3b7851e2e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:17.724 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:14:17.726 7f3b7acebc18 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:14:17.726 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:14:17.726 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:17.726 7f3b7acebc18 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:14:17.727 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:17.727 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:14:17.727 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:14:17.727 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:17.727 7f3b7acebc18 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:14:17.728 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:17.728 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:17.728 7f3b7acebc18 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/31/2025 02:14:17.728 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:17.734 7f3b7ae92688 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 2 03/31/2025 02:14:19.161 7f3b786eacc0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:14:20.415 7f3b7ae92688 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr config_db_uuid'. Error is: Result: 1 03/31/2025 02:14:21.727 7f3b7acfa780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:14:21.728 7f3b7acfa780 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:14:21.728 7f3b7acfa780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:14:21.729 7f3b7acfa780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:21.729 7f3b7acfa780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:14:21.729 7f3b7acfa780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:21.729 7f3b7acfa780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:14:21.729 7f3b7acfa780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:14:21.731 7f3b7acfa780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:21.731 7f3b7acfa780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:14:21.731 7f3b7acfa780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:21.731 7f3b7acfa780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:21.731 7f3b7acfa780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/31/2025 02:14:21.731 7f3b7acfa780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:21.797 7f3b7ae92688 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr to_bgp_keyspace'. Error is: Result: 1 03/31/2025 02:14:23.138 7f3b7ae92688 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr svc_monitor_keyspace'. Error is: Result: 1 03/31/2025 02:14:24.094 7f3b7ae92688 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr DISCOVERY_SERVER'. Error is: Result: 1 03/31/2025 02:14:25.001 7f3b7ae92688 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr useragent'. Error is: Result: 1 03/31/2025 02:14:25.629 7f3b786ec048 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:14:25.729 7f3b786ec048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:14:25.730 7f3b786ec048 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:14:25.731 7f3b786ec048 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:14:25.731 7f3b786ec048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:25.731 7f3b786ec048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:14:25.731 7f3b786ec048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:25.731 7f3b786ec048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:14:25.732 7f3b786ec048 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:14:25.732 7f3b786ec048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:25.732 7f3b786ec048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:14:25.732 7f3b786ec048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:25.732 7f3b786ec048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:25.732 7f3b786ec048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/31/2025 02:14:25.732 7f3b786ec048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:25.928 7f3b7ae92688 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr dm_keyspace'. Error is: Result: 1 03/31/2025 02:14:29.731 7f3b7721f438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:14:29.733 7f3b7721f438 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:14:29.733 7f3b7721f438 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:14:29.733 7f3b7721f438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:29.733 7f3b7721f438 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:14:29.734 7f3b7721f438 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:29.734 7f3b7721f438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:14:29.734 7f3b7721f438 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:14:29.734 7f3b7721f438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:29.734 7f3b7721f438 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:14:29.734 7f3b7721f438 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:29.734 7f3b7721f438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:29.735 7f3b7721f438 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/31/2025 02:14:29.735 7f3b7721f438 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:30.963 7f3b7ae92688 [contrail-config-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-494-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1743387270962927, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1743387250000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='zookeeper', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1743387250000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-config-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1743387250000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-03-31 00:53:57.142496", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8423-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/31/2025 02:14:30.963 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50258768, partition_space_available_1k=30831716, percentage_partition_space_used=62, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1239604, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51300, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51084, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=11386500, free=5648024, buffers=766708, cached=14863880, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.54, five_min_avg=3.4, fifteen_min_avg=2.3, cpu_share=0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:14:30.963 7f3b786ead68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:30.964 7f3b786ead68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1743387250000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = zookeeper process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1743387250000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-config-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1743387250000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-03-31 00:53:57.142496", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8423-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 03/31/2025 02:14:30.964 7f3b786ead68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:30.964 7f3b786ead68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:30.964 7f3b786ead68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 50258768 partition_space_available_1k = 30831716 percentage_partition_space_used = 62 >> } all_core_file_list = [ ] process_mem_cpu_usage = { cassandra : << mem_virt = 1239604 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 51300 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 51084 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 11386500 free = 5648024 buffers = 766708 cached = 14863880 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 5.54 five_min_avg = 3.4 fifteen_min_avg = 2.3 cpu_share = 0 node_type = config-database-node >> >> 03/31/2025 02:14:30.964 7f3b786ead68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:31.429 7f3b786ead68 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:14:33.734 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:14:33.735 7f3b80d1a048 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:14:33.737 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:14:33.737 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:33.737 7f3b80d1a048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:14:33.737 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:33.738 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:14:33.738 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:14:33.738 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:33.738 7f3b80d1a048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:14:33.739 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:33.739 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:33.739 7f3b80d1a048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/31/2025 02:14:33.739 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:37.739 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:14:37.741 7f3b80d1a048 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:14:37.741 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:14:37.741 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:37.741 7f3b80d1a048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:14:37.741 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:37.741 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:14:37.741 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:14:37.742 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:37.742 7f3b80d1a048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:14:37.742 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:37.742 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:37.742 7f3b80d1a048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/31/2025 02:14:37.742 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:41.745 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:14:41.746 7f3b80d1a048 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:14:41.747 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:14:41.747 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:41.747 7f3b80d1a048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:14:41.747 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:41.747 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:14:41.748 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:14:41.748 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:41.748 7f3b80d1a048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:14:41.748 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:41.748 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:41.748 7f3b80d1a048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/31/2025 02:14:41.748 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:42.661 7f3b80d1a048 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:14:45.747 7f3b772104e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:14:45.748 7f3b772104e0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:14:45.749 7f3b772104e0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:14:45.749 7f3b772104e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:45.749 7f3b772104e0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:14:45.749 7f3b772104e0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:45.749 7f3b772104e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:14:45.751 7f3b772104e0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:14:45.751 7f3b772104e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:45.751 7f3b772104e0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:14:45.751 7f3b772104e0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:45.751 7f3b772104e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:45.751 7f3b772104e0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/31/2025 02:14:45.751 7f3b772104e0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:49.753 7f3b772104e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:14:49.759 7f3b772104e0 [contrail-config-database-nodemgr] [INFO]: Session Event: TCP Connected 03/31/2025 02:14:49.760 7f3b772104e0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:14:49.760 7f3b772104e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:49.760 7f3b772104e0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:14:49.760 7f3b772104e0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:49.760 7f3b772104e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 03/31/2025 02:14:49.760 7f3b772104e0 [contrail-config-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 03/31/2025 02:14:49.761 7f3b772104e0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 03/31/2025 02:14:49.761 7f3b772104e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/31/2025 02:14:49.761 7f3b772104e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/31/2025 02:14:49.767 7f3b7721ea20 [contrail-config-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001011 03/31/2025 02:14:49.767 7f3b7721ea20 [contrail-config-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 03/31/2025 02:14:49.768 7f3b7863eac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 03/31/2025 02:14:49.768 7f3b7863eac8 [contrail-config-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 03/31/2025 02:14:49.768 7f3b7863eac8 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:DatabaseUsage] map 03/31/2025 02:14:49.768 7f3b7863eac8 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:14:49.768 7f3b7863eac8 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/31/2025 02:14:49.769 7f3b7863eac8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 03/31/2025 02:14:49.769 7f3b772104e0 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/31/2025 02:14:49.770 7f3b772104e0 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/31/2025 02:15:07.706 7f3b7acd82e8 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:15:14.522 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:15:16.343 7f3b7ae92688 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-494-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1743387316342917, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.17:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/31/2025 02:15:20.526 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-494-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1743387320526697, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=1, pending=2, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=4, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=1, pending=1, all_time_blocked=0)], _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/31/2025 02:15:20.527 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:CassandraStatusUVE] map 03/31/2025 02:15:25.650 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50838504, partition_space_available_1k=30251980, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1324980, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=151544, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42428, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14515248, free=11157836, buffers=806520, cached=6185508, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.84, five_min_avg=4.12, fifteen_min_avg=2.61, cpu_share=8.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:15:26.481 7f3b7aceb240 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:15:41.424 7f3b7ab39828 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:16:02.741 7f3b771f6cc0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:16:13.328 7f3b771f6cc0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:16:14.304 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:16:17.522 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-494-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1743387377522179, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/31/2025 02:16:22.648 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50848648, partition_space_available_1k=30241836, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1309696, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=157148, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42440, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14658924, free=9716976, buffers=841028, cached=7448184, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=8.52, five_min_avg=5.33, fifteen_min_avg=3.13, cpu_share=5.62, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:17:00.826 7f3b7acfaf60 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:17:14.288 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:17:22.653 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50855516, partition_space_available_1k=30234968, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1303452, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=175672, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42448, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14625540, free=9709168, buffers=842904, cached=7487500, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.53, five_min_avg=4.77, fifteen_min_avg=3.07, cpu_share=2.88, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:18:14.284 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:18:22.676 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50858900, partition_space_available_1k=30231584, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1302108, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=178028, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42456, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14615816, free=9713232, buffers=842976, cached=7493088, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.75, five_min_avg=4.23, fifteen_min_avg=2.99, cpu_share=2.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:19:10.569 7f3b80021780 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:19:14.292 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:19:17.313 7f3b7ae92688 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-494-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1743387557313142, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.17:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='NTP state unsynchronized. ')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/31/2025 02:19:19.345 7f3b786ec780 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:19:22.701 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50866828, partition_space_available_1k=30223656, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1334112, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=178412, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42464, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14652532, free=9666192, buffers=843116, cached=7503272, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.16, five_min_avg=3.76, fifteen_min_avg=2.91, cpu_share=2.62, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:19:30.937 7f3b7863eac8 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:19:53.731 7f3b7851e0f0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:20:14.328 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:20:22.680 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50881980, partition_space_available_1k=30208504, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1304508, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=178856, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42472, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14710804, free=9505720, buffers=843172, cached=7605416, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.52, five_min_avg=4.59, fifteen_min_avg=3.26, cpu_share=5.88, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:21:14.315 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:21:22.685 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50878296, partition_space_available_1k=30212188, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1304148, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=181892, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42476, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14646644, free=9496460, buffers=843180, cached=7678828, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.21, five_min_avg=4.41, fifteen_min_avg=3.28, cpu_share=2.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:22:14.325 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:22:22.695 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50882348, partition_space_available_1k=30208136, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1326516, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=189304, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42480, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14648920, free=9489096, buffers=843180, cached=7683916, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.67, five_min_avg=3.96, fifteen_min_avg=3.2, cpu_share=2.38, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:23:14.326 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:23:22.706 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50885880, partition_space_available_1k=30204604, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1306632, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=189724, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42484, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14640468, free=9493300, buffers=843192, cached=7688152, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.9, five_min_avg=3.52, fifteen_min_avg=3.09, cpu_share=2.38, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:23:46.481 7f3b7a798390 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:23:55.117 7f3b786c2d68 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:24:10.584 7f3b785fe0f0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:24:14.340 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:24:17.314 7f3b7ae92688 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-494-1', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1743387857314695, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.17:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/31/2025 02:24:20.166 7f3b7860fb70 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:24:22.724 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50898896, partition_space_available_1k=30191588, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1305092, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=190048, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42488, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14706844, free=9409348, buffers=843244, cached=7705676, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.73, five_min_avg=4.56, fifteen_min_avg=3.47, cpu_share=4.38, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:25:14.326 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:25:22.722 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50899852, partition_space_available_1k=30190632, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1310384, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=190384, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42488, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14692476, free=9416948, buffers=843244, cached=7712444, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.42, five_min_avg=4.0, fifteen_min_avg=3.35, cpu_share=2.62, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:26:14.336 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:26:22.729 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51006316, partition_space_available_1k=30084168, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1307516, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=190508, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42496, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14679120, free=9423604, buffers=843248, cached=7719140, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.16, five_min_avg=3.83, fifteen_min_avg=3.33, cpu_share=2.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:27:14.343 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:27:22.731 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50907556, partition_space_available_1k=30182928, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1306772, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=190816, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42496, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14669792, free=9495840, buffers=843248, cached=7656232, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.4, five_min_avg=3.49, fifteen_min_avg=3.25, cpu_share=2.62, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:28:08.860 7f3b7acfa390 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:28:14.348 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:28:17.366 7f3b771f6a20 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:28:22.732 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50915560, partition_space_available_1k=30174924, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1317700, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=191172, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42500, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14744936, free=9412196, buffers=843248, cached=7664732, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.77, five_min_avg=3.38, fifteen_min_avg=3.22, cpu_share=2.88, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:28:28.476 7f3b7acd4240 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:28:38.043 7f3b7a89c588 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:29:14.359 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:29:22.746 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50914948, partition_space_available_1k=30175536, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1307108, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=202488, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42512, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14683104, free=9473624, buffers=843248, cached=7665136, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.01, five_min_avg=3.02, fifteen_min_avg=3.1, cpu_share=2.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:30:14.367 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:30:23.751 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50924840, partition_space_available_1k=30165644, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1309060, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=202836, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42504, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14714596, free=9422360, buffers=843252, cached=7684904, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.42, five_min_avg=3.25, fifteen_min_avg=3.18, cpu_share=4.38, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:31:14.365 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:31:22.752 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50929140, partition_space_available_1k=30161344, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1310136, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=203168, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42508, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14697960, free=9434524, buffers=843280, cached=7689348, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.39, five_min_avg=2.98, fifteen_min_avg=3.08, cpu_share=2.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:32:14.365 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:32:22.751 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50932740, partition_space_available_1k=30157744, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1309996, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=203496, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42512, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14689932, free=9437108, buffers=843296, cached=7694776, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.33, five_min_avg=2.82, fifteen_min_avg=3.02, cpu_share=2.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:33:14.378 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:33:22.753 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50936152, partition_space_available_1k=30154332, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1309464, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=203840, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42516, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14758192, free=9359628, buffers=843312, cached=7703980, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.9, five_min_avg=2.62, fifteen_min_avg=2.94, cpu_share=2.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:33:27.249 7f3b7acd4cc0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:33:41.269 7f3b77213588 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:33:49.837 7f3b7aceb048 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:34:00.865 7f3b786fa8d0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:34:14.369 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:34:22.770 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50940320, partition_space_available_1k=30150164, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1313612, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=204168, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42524, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14690304, free=9418880, buffers=843560, cached=7712368, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.6, five_min_avg=2.65, fifteen_min_avg=2.93, cpu_share=2.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:35:14.398 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:35:22.759 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50950396, partition_space_available_1k=30140088, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1310280, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=204520, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42520, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14721328, free=9349848, buffers=844576, cached=7749360, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.1, five_min_avg=3.21, fifteen_min_avg=3.11, cpu_share=4.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:36:14.394 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:36:22.761 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50956084, partition_space_available_1k=30134400, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1311084, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=204760, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42524, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14728184, free=9335328, buffers=844580, cached=7757020, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.75, five_min_avg=3.3, fifteen_min_avg=3.15, cpu_share=2.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:37:14.395 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:37:22.777 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50959620, partition_space_available_1k=30130864, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1311052, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=204916, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42532, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14740216, free=9318224, buffers=844584, cached=7762088, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.1, five_min_avg=2.88, fifteen_min_avg=3.01, cpu_share=2.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:37:49.925 7f3b7acebc18 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:37:53.361 7f3b7721ea20 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 03/31/2025 02:37:53.361 7f3b7c359f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[Established] 03/31/2025 02:37:58.368 7f3b7c359f60 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:37:58.368 7f3b7c359f60 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[Established] -> State[Connect] 03/31/2025 02:37:58.368 7f3b7c359f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:37:58.368 7f3b7c359f60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Established to Connect on EvTcpClose >>, ] description = Collector:Collector[Established to Connect on EvTcpClose] connection down >>, ] >> 03/31/2025 02:37:58.369 7f3b7c359f60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:37:58.369 7f3b7c359f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:37:58.369 7f3b7c359f60 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:37:58.369 7f3b7c359f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:58.369 7f3b7c359f60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:37:58.369 7f3b7c359f60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:58.370 7f3b7c359f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:58.370 7f3b7c359f60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:37:58.370 7f3b7c359f60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:02.369 7f3b7aceb828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:38:07.374 7f3b7aceb828 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:38:07.374 7f3b7aceb828 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:38:07.374 7f3b7aceb828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:07.375 7f3b7aceb828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:38:07.375 7f3b7aceb828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:07.375 7f3b7aceb828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:38:07.375 7f3b7aceb828 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:38:07.375 7f3b7aceb828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:07.376 7f3b7aceb828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:38:07.376 7f3b7aceb828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:07.376 7f3b7aceb828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:07.376 7f3b7aceb828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:38:07.376 7f3b7aceb828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:11.377 7f3b7aceb828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:38:14.377 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:38:16.379 7f3b7aceb828 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:38:16.379 7f3b7aceb828 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:38:16.379 7f3b7aceb828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:16.379 7f3b7aceb828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:38:16.380 7f3b7aceb828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:16.380 7f3b7aceb828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:16.380 7f3b7aceb828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 database_usage = [ << disk_space_used_1k = 50962976 disk_space_available_1k = 30127508 config_db_size_1k = 1028 >>, ] >> 03/31/2025 02:38:16.380 7f3b7aceb828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:16.380 7f3b7aceb828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:38:16.380 7f3b7aceb828 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:38:16.380 7f3b7aceb828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:16.381 7f3b7aceb828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:38:16.381 7f3b7aceb828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:16.381 7f3b7aceb828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:16.381 7f3b7aceb828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:38:16.381 7f3b7aceb828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:20.381 7f3b7720e828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:38:22.788 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50963152, partition_space_available_1k=30127332, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1311344, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=205240, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42584, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14722272, free=9331680, buffers=844592, cached=7766568, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.95, five_min_avg=2.68, fifteen_min_avg=2.93, cpu_share=2.38, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:38:25.385 7f3b7720e828 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:38:25.385 7f3b7720e828 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:38:25.386 7f3b7720e828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:25.386 7f3b7720e828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:38:25.386 7f3b7720e828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:25.386 7f3b7720e828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:25.386 7f3b7720e828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 50963152 partition_space_available_1k = 30127332 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1311344 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 205240 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 42584 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 14722272 free = 9331680 buffers = 844592 cached = 7766568 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 1.95 five_min_avg = 2.68 fifteen_min_avg = 2.93 cpu_share = 2.38 node_type = config-database-node >> >> 03/31/2025 02:38:25.386 7f3b7720e828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:25.386 7f3b7720e828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:38:25.387 7f3b7720e828 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:38:25.387 7f3b7720e828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:25.387 7f3b7720e828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:38:25.387 7f3b7720e828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:25.387 7f3b7720e828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:25.387 7f3b7720e828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:38:25.387 7f3b7720e828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:29.386 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:38:34.391 7f3b7acebc18 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:38:34.392 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:38:34.392 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:34.392 7f3b7acebc18 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:38:34.392 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:34.392 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:38:34.393 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:38:34.393 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:34.393 7f3b7acebc18 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:38:34.393 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:34.393 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:34.394 7f3b7acebc18 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:38:34.394 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:38.395 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:38:43.401 7f3b7acebc18 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:38:43.401 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:38:43.401 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:43.401 7f3b7acebc18 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:38:43.401 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:43.402 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:38:43.402 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:38:43.402 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:43.402 7f3b7acebc18 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:38:43.407 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:43.407 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:43.407 7f3b7acebc18 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:38:43.407 7f3b7acebc18 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:47.405 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:38:52.414 7f3b80021780 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:38:52.415 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:38:52.415 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:52.416 7f3b80021780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:38:52.416 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:52.416 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:38:52.417 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:38:52.417 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:52.417 7f3b80021780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:38:52.417 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:52.417 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:52.417 7f3b80021780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:38:52.417 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:56.418 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:39:01.421 7f3b80021780 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:39:01.421 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:39:01.421 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:01.422 7f3b80021780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:39:01.422 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:01.422 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:39:01.422 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:39:01.423 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:01.423 7f3b80021780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:39:01.423 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:01.423 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:01.423 7f3b80021780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:39:01.423 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:05.425 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:39:10.429 7f3b80021780 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:39:10.430 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:39:10.430 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:10.430 7f3b80021780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:39:10.430 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:10.430 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:39:10.431 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:39:10.431 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:10.431 7f3b80021780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:39:10.431 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:10.431 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:10.431 7f3b80021780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:39:10.431 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:14.124 7f3b7851e0f0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:39:14.388 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:39:14.430 7f3b786ec8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:39:19.432 7f3b786ec8d0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:39:19.432 7f3b786ec8d0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:39:19.433 7f3b786ec8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:19.433 7f3b786ec8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:39:19.433 7f3b786ec8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:19.433 7f3b786ec8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:19.433 7f3b786ec8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 database_usage = [ << disk_space_used_1k = 50971492 disk_space_available_1k = 30118992 config_db_size_1k = 1028 >>, ] >> 03/31/2025 02:39:19.433 7f3b786ec8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:19.433 7f3b786ec8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:39:19.434 7f3b786ec8d0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:39:19.434 7f3b786ec8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:19.434 7f3b786ec8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:39:19.434 7f3b786ec8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:19.435 7f3b786ec8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:19.435 7f3b786ec8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:39:19.435 7f3b786ec8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:22.745 7f3b772134e0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:39:22.801 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50976208, partition_space_available_1k=30114276, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1311036, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=205588, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42596, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14839072, free=9193952, buffers=844592, cached=7787496, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.71, five_min_avg=3.43, fifteen_min_avg=3.18, cpu_share=4.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:39:22.801 7f3b77202ac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:22.801 7f3b77202ac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 50976208 partition_space_available_1k = 30114276 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1311036 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 205588 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 42596 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 14839072 free = 9193952 buffers = 844592 cached = 7787496 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 4.71 five_min_avg = 3.43 fifteen_min_avg = 3.18 cpu_share = 4.0 node_type = config-database-node >> >> 03/31/2025 02:39:22.801 7f3b77202ac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:23.433 7f3b77202ac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:39:28.439 7f3b77202ac8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:39:28.439 7f3b77202ac8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:39:28.440 7f3b77202ac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:28.440 7f3b77202ac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:39:28.440 7f3b77202ac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:28.440 7f3b77202ac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:39:28.440 7f3b77202ac8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:39:28.441 7f3b77202ac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:28.441 7f3b77202ac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:39:28.441 7f3b77202ac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:28.441 7f3b77202ac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:28.441 7f3b77202ac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:39:28.441 7f3b77202ac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:32.441 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:39:34.038 7f3b7865d438 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:39:37.446 7f3b7a7b7978 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:39:37.447 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:39:37.447 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:37.447 7f3b7a7b7978 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:39:37.447 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:37.447 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:39:37.448 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:39:37.448 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:37.448 7f3b7a7b7978 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:39:37.448 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:37.448 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:37.448 7f3b7a7b7978 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:39:37.448 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:41.450 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:39:46.453 7f3b7a7b7978 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:39:46.453 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:39:46.453 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:46.454 7f3b7a7b7978 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:39:46.454 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:46.454 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:39:46.454 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:39:46.454 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:46.455 7f3b7a7b7978 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:39:46.455 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:46.455 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:46.455 7f3b7a7b7978 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:39:46.455 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:50.457 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:39:55.462 7f3b7a7b7978 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:39:55.463 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:39:55.463 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:55.463 7f3b7a7b7978 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:39:55.463 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:55.463 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:39:55.464 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:39:55.464 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:55.464 7f3b7a7b7978 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:39:55.464 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:55.464 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:55.464 7f3b7a7b7978 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:39:55.464 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:59.466 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:40:04.469 7f3b7a7b7978 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:40:04.469 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:40:04.469 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:04.469 7f3b7a7b7978 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:40:04.469 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:04.469 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:40:04.470 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:40:04.470 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:04.470 7f3b7a7b7978 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:40:04.470 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:04.470 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:04.470 7f3b7a7b7978 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:40:04.470 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:08.471 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:40:13.478 7f3b7a7b7978 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:40:13.478 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:40:13.478 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:13.478 7f3b7a7b7978 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:40:13.479 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:13.479 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:40:13.479 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:40:13.479 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:13.480 7f3b7a7b7978 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:40:13.480 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:13.480 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:13.480 7f3b7a7b7978 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:40:13.480 7f3b7a7b7978 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:14.386 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:40:14.702 7f3b786ec198 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:14.702 7f3b786ec198 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 database_usage = [ << disk_space_used_1k = 50971696 disk_space_available_1k = 30118788 config_db_size_1k = 1028 >>, ] >> 03/31/2025 02:40:14.702 7f3b786ec198 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:17.478 7f3b786caeb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:40:22.481 7f3b786caeb8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:40:22.482 7f3b786caeb8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:40:22.482 7f3b786caeb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:22.482 7f3b786caeb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:40:22.482 7f3b786caeb8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:22.482 7f3b786caeb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:40:22.483 7f3b786caeb8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:40:22.483 7f3b786caeb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:22.483 7f3b786caeb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:40:22.483 7f3b786caeb8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:22.483 7f3b786caeb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:22.483 7f3b786caeb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:40:22.484 7f3b786caeb8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:22.803 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50972236, partition_space_available_1k=30118248, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1312224, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=205908, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42648, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14843720, free=9189584, buffers=844596, cached=7787212, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.95, five_min_avg=3.14, fifteen_min_avg=3.09, cpu_share=2.38, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:40:22.804 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:22.804 7f3b7acd4f60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 50972236 partition_space_available_1k = 30118248 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1312224 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 205908 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 42648 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 14843720 free = 9189584 buffers = 844596 cached = 7787212 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 2.95 five_min_avg = 3.14 fifteen_min_avg = 3.09 cpu_share = 2.38 node_type = config-database-node >> >> 03/31/2025 02:40:22.804 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:26.483 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:40:31.486 7f3b7acd4f60 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:40:31.486 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:40:31.486 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:31.486 7f3b7acd4f60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:40:31.486 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:31.486 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:40:31.487 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:40:31.487 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:31.487 7f3b7acd4f60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:40:31.487 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:31.487 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:31.487 7f3b7acd4f60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:40:31.487 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:35.490 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:40:40.496 7f3b7acd4f60 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:40:40.496 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:40:40.496 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:40.496 7f3b7acd4f60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:40:40.497 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:40.497 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:40:40.497 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:40:40.497 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:40.497 7f3b7acd4f60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:40:40.497 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:40.498 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:40.498 7f3b7acd4f60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:40:40.498 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:44.500 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:40:49.505 7f3b7acd4f60 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:40:49.505 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:40:49.505 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:49.505 7f3b7acd4f60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:40:49.506 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:49.506 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:40:49.506 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:40:49.506 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:49.506 7f3b7acd4f60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:40:49.507 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:49.507 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:49.508 7f3b7acd4f60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:40:49.508 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:53.509 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:40:58.515 7f3b7acd4f60 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:40:58.516 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:40:58.516 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:58.516 7f3b7acd4f60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:40:58.516 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:58.516 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:40:58.517 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:40:58.517 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:58.517 7f3b7acd4f60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:40:58.517 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:58.517 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:58.517 7f3b7acd4f60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:40:58.517 7f3b7acd4f60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:02.517 7f3b784e4eb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:41:07.521 7f3b784e4eb8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:41:07.521 7f3b784e4eb8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:41:07.521 7f3b784e4eb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:07.521 7f3b784e4eb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:41:07.521 7f3b784e4eb8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:07.522 7f3b784e4eb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:41:07.522 7f3b784e4eb8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:41:07.522 7f3b784e4eb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:07.522 7f3b784e4eb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:41:07.522 7f3b784e4eb8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:07.523 7f3b784e4eb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:07.523 7f3b784e4eb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:41:07.523 7f3b784e4eb8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:11.524 7f3b784e4eb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:41:14.452 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:41:16.527 7f3b784e4eb8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:41:16.527 7f3b784e4eb8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:41:16.527 7f3b784e4eb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:16.527 7f3b784e4eb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:41:16.528 7f3b784e4eb8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:16.528 7f3b784e4eb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:16.528 7f3b784e4eb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 database_usage = [ << disk_space_used_1k = 50976464 disk_space_available_1k = 30114020 config_db_size_1k = 1028 >>, ] >> 03/31/2025 02:41:16.528 7f3b784e4eb8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:16.528 7f3b784e4eb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:41:16.529 7f3b784e4eb8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:41:16.529 7f3b784e4eb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:16.529 7f3b784e4eb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:41:16.530 7f3b784e4eb8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:16.530 7f3b784e4eb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:16.530 7f3b784e4eb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:41:16.530 7f3b784e4eb8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:20.529 7f3b77dd50f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:41:22.813 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50976652, partition_space_available_1k=30113832, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1312444, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=206240, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43028, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14809928, free=9216340, buffers=844596, cached=7794248, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.42, five_min_avg=2.95, fifteen_min_avg=3.03, cpu_share=2.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:41:25.533 7f3b77dd50f0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:41:25.533 7f3b77dd50f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:41:25.533 7f3b77dd50f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:25.533 7f3b77dd50f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:41:25.533 7f3b77dd50f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:25.534 7f3b77dd50f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:25.534 7f3b77dd50f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 50976652 partition_space_available_1k = 30113832 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1312444 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 206240 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 43028 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 14809928 free = 9216340 buffers = 844596 cached = 7794248 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 2.42 five_min_avg = 2.95 fifteen_min_avg = 3.03 cpu_share = 2.0 node_type = config-database-node >> >> 03/31/2025 02:41:25.534 7f3b77dd50f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:25.534 7f3b77dd50f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:41:25.534 7f3b77dd50f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:41:25.535 7f3b77dd50f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:25.535 7f3b77dd50f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:41:25.535 7f3b77dd50f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:25.535 7f3b77dd50f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:25.535 7f3b77dd50f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:41:25.535 7f3b77dd50f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:29.537 7f3b786c2cc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:41:34.541 7f3b786c2cc0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:41:34.541 7f3b786c2cc0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:41:34.541 7f3b786c2cc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:34.541 7f3b786c2cc0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:41:34.541 7f3b786c2cc0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:34.542 7f3b786c2cc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:41:34.542 7f3b786c2cc0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:41:34.542 7f3b786c2cc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:34.542 7f3b786c2cc0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:41:34.542 7f3b786c2cc0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:34.542 7f3b786c2cc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:34.543 7f3b786c2cc0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:41:34.543 7f3b786c2cc0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:38.545 7f3b784e46d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:41:43.550 7f3b784e46d8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:41:43.550 7f3b784e46d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:41:43.551 7f3b784e46d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:43.551 7f3b784e46d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:41:43.551 7f3b784e46d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:43.551 7f3b784e46d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:41:43.551 7f3b784e46d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:41:43.551 7f3b784e46d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:43.552 7f3b784e46d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:41:43.552 7f3b784e46d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:43.552 7f3b784e46d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:43.552 7f3b784e46d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:41:43.552 7f3b784e46d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:47.551 7f3b7bfb8c18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:41:52.557 7f3b7bfb8c18 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:41:52.558 7f3b7bfb8c18 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:41:52.558 7f3b7bfb8c18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:52.558 7f3b7bfb8c18 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:41:52.558 7f3b7bfb8c18 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:52.558 7f3b7bfb8c18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:41:52.559 7f3b7bfb8c18 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:41:52.559 7f3b7bfb8c18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:52.559 7f3b7bfb8c18 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:41:52.559 7f3b7bfb8c18 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:52.559 7f3b7bfb8c18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:52.559 7f3b7bfb8c18 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:41:52.559 7f3b7bfb8c18 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:56.561 7f3b7bfb8c18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:42:01.567 7f3b7bfb8c18 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:42:01.567 7f3b7bfb8c18 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:42:01.567 7f3b7bfb8c18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:01.568 7f3b7bfb8c18 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:42:01.568 7f3b7bfb8c18 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:01.568 7f3b7bfb8c18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:42:01.568 7f3b7bfb8c18 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:42:01.568 7f3b7bfb8c18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:01.568 7f3b7bfb8c18 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:42:01.569 7f3b7bfb8c18 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:01.569 7f3b7bfb8c18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:01.569 7f3b7bfb8c18 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:42:01.569 7f3b7bfb8c18 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:05.570 7f3b772108d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:42:10.577 7f3b772108d0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:42:10.577 7f3b772108d0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:42:10.577 7f3b772108d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:10.577 7f3b772108d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:42:10.577 7f3b772108d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:10.578 7f3b772108d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:42:10.578 7f3b772108d0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:42:10.578 7f3b772108d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:10.578 7f3b772108d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:42:10.578 7f3b772108d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:10.578 7f3b772108d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:10.579 7f3b772108d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:42:10.579 7f3b772108d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:11.219 7f3b772108d0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:42:14.392 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:42:14.578 7f3b77213390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:42:19.583 7f3b77213390 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:42:19.584 7f3b77213390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:42:19.584 7f3b77213390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:19.584 7f3b77213390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:42:19.584 7f3b77213390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:19.585 7f3b77213390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:19.585 7f3b77213390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 database_usage = [ << disk_space_used_1k = 50980364 disk_space_available_1k = 30110120 config_db_size_1k = 1028 >>, ] >> 03/31/2025 02:42:19.585 7f3b77213390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:19.585 7f3b77213390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:42:19.586 7f3b77213390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:42:19.587 7f3b77213390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:19.587 7f3b77213390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:42:19.587 7f3b77213390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:19.587 7f3b77213390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:19.587 7f3b77213390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:42:19.587 7f3b77213390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:22.823 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50980584, partition_space_available_1k=30109900, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1312872, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=206588, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43048, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14812688, free=9207748, buffers=844596, cached=7800080, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.73, five_min_avg=2.93, fifteen_min_avg=3.02, cpu_share=2.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:42:22.824 7f3b771f6780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:22.824 7f3b771f6780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 50980584 partition_space_available_1k = 30109900 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1312872 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 206588 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 43048 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 14812688 free = 9207748 buffers = 844596 cached = 7800080 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 2.73 five_min_avg = 2.93 fifteen_min_avg = 3.02 cpu_share = 2.25 node_type = config-database-node >> >> 03/31/2025 02:42:22.824 7f3b771f6780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:23.585 7f3b771f6780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:42:28.591 7f3b771f6780 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:42:28.591 7f3b771f6780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:42:28.591 7f3b771f6780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:28.591 7f3b771f6780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:42:28.591 7f3b771f6780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:28.591 7f3b771f6780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:42:28.592 7f3b771f6780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:42:28.592 7f3b771f6780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:28.592 7f3b771f6780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:42:28.592 7f3b771f6780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:28.592 7f3b771f6780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:28.593 7f3b771f6780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:42:28.593 7f3b771f6780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:32.592 7f3b7ae1fa20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:42:37.598 7f3b7ae1fa20 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:42:37.598 7f3b7ae1fa20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:42:37.598 7f3b7ae1fa20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:37.598 7f3b7ae1fa20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:42:37.599 7f3b7ae1fa20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:37.599 7f3b7ae1fa20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:42:37.599 7f3b7ae1fa20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:42:37.599 7f3b7ae1fa20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:37.599 7f3b7ae1fa20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:42:37.599 7f3b7ae1fa20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:37.599 7f3b7ae1fa20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:37.600 7f3b7ae1fa20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:42:37.601 7f3b7ae1fa20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:41.602 7f3b7ae1fa20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:42:46.608 7f3b7ae1fa20 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:42:46.608 7f3b7ae1fa20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:42:46.608 7f3b7ae1fa20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:46.609 7f3b7ae1fa20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:42:46.609 7f3b7ae1fa20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:46.609 7f3b7ae1fa20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:42:46.609 7f3b7ae1fa20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:42:46.609 7f3b7ae1fa20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:46.610 7f3b7ae1fa20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:42:46.610 7f3b7ae1fa20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:46.610 7f3b7ae1fa20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:46.610 7f3b7ae1fa20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:42:46.610 7f3b7ae1fa20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:50.611 7f3b7ae1fac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:42:55.617 7f3b7ae1fac8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:42:55.617 7f3b7ae1fac8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:42:55.617 7f3b7ae1fac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:55.617 7f3b7ae1fac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:42:55.617 7f3b7ae1fac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:55.618 7f3b7ae1fac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:42:55.618 7f3b7ae1fac8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:42:55.618 7f3b7ae1fac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:55.618 7f3b7ae1fac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:42:55.618 7f3b7ae1fac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:55.618 7f3b7ae1fac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:55.619 7f3b7ae1fac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:42:55.619 7f3b7ae1fac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:59.621 7f3b7ae1fac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:43:04.625 7f3b7ae1fac8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:43:04.625 7f3b7ae1fac8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:43:04.625 7f3b7ae1fac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:04.625 7f3b7ae1fac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:43:04.625 7f3b7ae1fac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:04.626 7f3b7ae1fac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:43:04.626 7f3b7ae1fac8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:43:04.626 7f3b7ae1fac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:04.626 7f3b7ae1fac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:43:04.626 7f3b7ae1fac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:04.627 7f3b7ae1fac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:04.627 7f3b7ae1fac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:43:04.627 7f3b7ae1fac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:08.629 7f3b7a7d5048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:43:13.633 7f3b7a7d5048 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:43:13.633 7f3b7a7d5048 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:43:13.633 7f3b7a7d5048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:13.633 7f3b7a7d5048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:43:13.634 7f3b7a7d5048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:13.634 7f3b7a7d5048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:43:13.634 7f3b7a7d5048 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:43:13.634 7f3b7a7d5048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:13.634 7f3b7a7d5048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:43:13.634 7f3b7a7d5048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:13.635 7f3b7a7d5048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:13.635 7f3b7a7d5048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:43:13.635 7f3b7a7d5048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:14.393 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:43:14.707 7f3b7acfa978 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:14.707 7f3b7acfa978 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 database_usage = [ << disk_space_used_1k = 50984028 disk_space_available_1k = 30106456 config_db_size_1k = 1028 >>, ] >> 03/31/2025 02:43:14.707 7f3b7acfa978 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:17.634 7f3b7acfa390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:43:22.637 7f3b7acfa390 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:43:22.637 7f3b7acfa390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:43:22.637 7f3b7acfa390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:22.637 7f3b7acfa390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:43:22.637 7f3b7acfa390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:22.638 7f3b7acfa390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:43:22.638 7f3b7acfa390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:43:22.638 7f3b7acfa390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:22.638 7f3b7acfa390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:43:22.638 7f3b7acfa390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:22.638 7f3b7acfa390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:22.639 7f3b7acfa390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:43:22.639 7f3b7acfa390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:22.833 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50984216, partition_space_available_1k=30106268, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1311488, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=206928, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43096, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14797476, free=9218416, buffers=844596, cached=7804624, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.32, five_min_avg=2.78, fifteen_min_avg=2.96, cpu_share=1.88, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:43:22.834 7f3b789dbeb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:22.834 7f3b789dbeb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 50984216 partition_space_available_1k = 30106268 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1311488 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 206928 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 43096 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 14797476 free = 9218416 buffers = 844596 cached = 7804624 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 2.32 five_min_avg = 2.78 fifteen_min_avg = 2.96 cpu_share = 1.88 node_type = config-database-node >> >> 03/31/2025 02:43:22.834 7f3b789dbeb8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:26.641 7f3b789dbeb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:43:31.647 7f3b789dbeb8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:43:31.647 7f3b789dbeb8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:43:31.647 7f3b789dbeb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:31.647 7f3b789dbeb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:43:31.647 7f3b789dbeb8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:31.647 7f3b789dbeb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:43:31.648 7f3b789dbeb8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:43:31.648 7f3b789dbeb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:31.648 7f3b789dbeb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:43:31.648 7f3b789dbeb8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:31.648 7f3b789dbeb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:31.649 7f3b789dbeb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:43:31.649 7f3b789dbeb8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:35.649 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:43:40.653 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:43:40.653 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:43:40.653 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:40.654 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:43:40.654 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:40.654 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:43:40.654 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:43:40.654 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:40.655 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:43:40.655 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:40.655 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:40.655 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:43:40.655 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:44.657 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:43:49.661 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:43:49.661 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:43:49.661 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:49.661 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:43:49.661 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:49.662 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:43:49.662 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:43:49.662 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:49.662 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:43:49.662 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:49.662 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:49.663 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:43:49.663 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:53.662 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:43:58.667 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:43:58.667 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:43:58.667 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:58.667 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:43:58.667 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:58.667 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:43:58.668 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:43:58.668 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:58.668 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:43:58.668 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:58.668 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:58.668 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:43:58.669 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:02.670 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:44:07.677 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:44:07.677 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:44:07.677 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:07.677 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:44:07.677 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:07.677 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:44:07.678 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:44:07.678 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:07.678 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:44:07.678 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:07.678 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:07.678 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:44:07.678 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:11.681 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:44:14.403 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:44:16.683 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:44:16.683 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:44:16.683 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:16.684 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:44:16.684 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:16.684 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:16.684 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 database_usage = [ << disk_space_used_1k = 50990504 disk_space_available_1k = 30099980 config_db_size_1k = 1028 >>, ] >> 03/31/2025 02:44:16.684 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:16.684 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:44:16.685 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:44:16.685 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:16.685 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:44:16.685 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:16.685 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:16.685 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:44:16.685 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:20.685 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:44:21.061 7f3b786ea0f0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:44:22.852 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50992328, partition_space_available_1k=30098156, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1312364, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=207220, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43164, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14894536, free=9115972, buffers=844596, cached=7810008, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.05, five_min_avg=2.85, fifteen_min_avg=2.97, cpu_share=2.12, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:44:25.690 7f3b7ae24780 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:44:25.691 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:44:25.691 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:25.691 7f3b7ae24780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:44:25.691 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:25.691 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:25.691 7f3b7ae24780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 50992328 partition_space_available_1k = 30098156 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1312364 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 207220 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 43164 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 14894536 free = 9115972 buffers = 844596 cached = 7810008 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 3.05 five_min_avg = 2.85 fifteen_min_avg = 2.97 cpu_share = 2.12 node_type = config-database-node >> >> 03/31/2025 02:44:25.691 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:25.692 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:44:25.692 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:44:25.692 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:25.692 7f3b7ae24780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:44:25.692 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:25.692 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:25.692 7f3b7ae24780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:44:25.693 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:29.694 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:44:29.762 7f3b7860f2e8 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:44:34.706 7f3b7ae24780 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:44:34.706 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:44:34.706 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:34.706 7f3b7ae24780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:44:34.706 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:34.706 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:44:34.707 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:44:34.707 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:34.707 7f3b7ae24780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:44:34.707 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:34.707 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:34.708 7f3b7ae24780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:44:34.708 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:38.709 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:44:43.715 7f3b7ae24780 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:44:43.715 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:44:43.715 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:43.715 7f3b7ae24780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:44:43.716 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:43.716 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:44:43.716 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:44:43.716 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:43.716 7f3b7ae24780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:44:43.716 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:43.716 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:43.717 7f3b7ae24780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:44:43.717 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:43.920 7f3b7ae24780 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:44:47.718 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:44:52.725 7f3b7ae24780 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:44:52.725 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:44:52.725 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:52.726 7f3b7ae24780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:44:52.726 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:52.726 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:44:52.726 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:44:52.726 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:52.726 7f3b7ae24780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:44:52.727 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:52.727 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:52.727 7f3b7ae24780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:44:52.727 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:56.729 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:45:01.733 7f3b7ae24780 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:45:01.733 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:45:01.733 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:01.734 7f3b7ae24780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:45:01.734 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:01.734 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:45:01.734 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:45:01.734 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:01.735 7f3b7ae24780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:45:01.735 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:01.735 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:01.735 7f3b7ae24780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:45:01.735 7f3b7ae24780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:05.734 7f3b789dbcc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:45:10.739 7f3b789dbcc0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:45:10.740 7f3b789dbcc0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:45:10.740 7f3b789dbcc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:10.740 7f3b789dbcc0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:45:10.740 7f3b789dbcc0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:10.741 7f3b789dbcc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:45:10.741 7f3b789dbcc0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:45:10.741 7f3b789dbcc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:10.742 7f3b789dbcc0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:45:10.742 7f3b789dbcc0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:10.742 7f3b789dbcc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:10.742 7f3b789dbcc0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:45:10.742 7f3b789dbcc0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:14.412 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:45:14.740 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:45:19.743 7f3b7a798390 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:45:19.743 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:45:19.744 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:19.744 7f3b7a798390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:45:19.744 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:19.744 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:19.744 7f3b7a798390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 database_usage = [ << disk_space_used_1k = 50993256 disk_space_available_1k = 30097228 config_db_size_1k = 1028 >>, ] >> 03/31/2025 02:45:19.744 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:19.745 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:45:19.745 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:45:19.745 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:19.746 7f3b7a798390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:45:19.746 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:19.746 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:19.746 7f3b7a798390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:45:19.746 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:22.854 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50993416, partition_space_available_1k=30097068, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1314296, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=207528, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43184, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14896536, free=9100944, buffers=844596, cached=7823036, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.55, five_min_avg=3.48, fifteen_min_avg=3.19, cpu_share=4.12, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:45:22.855 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:22.855 7f3b7a798390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 50993416 partition_space_available_1k = 30097068 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1314296 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 207528 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 43184 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 14896536 free = 9100944 buffers = 844596 cached = 7823036 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 4.55 five_min_avg = 3.48 fifteen_min_avg = 3.19 cpu_share = 4.12 node_type = config-database-node >> >> 03/31/2025 02:45:22.855 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:23.744 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:45:28.750 7f3b7a798390 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:45:28.750 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:45:28.751 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:28.751 7f3b7a798390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:45:28.751 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:28.751 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:45:28.752 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:45:28.752 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:28.752 7f3b7a798390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:45:28.752 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:28.752 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:28.752 7f3b7a798390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:45:28.752 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:32.755 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:45:37.760 7f3b7a798390 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:45:37.760 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:45:37.761 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:37.761 7f3b7a798390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:45:37.761 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:37.761 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:45:37.761 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:45:37.761 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:37.762 7f3b7a798390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:45:37.762 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:37.762 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:37.762 7f3b7a798390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:45:37.762 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:41.764 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:45:46.770 7f3b7a798390 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:45:46.771 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:45:46.771 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:46.771 7f3b7a798390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:45:46.771 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:46.771 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:45:46.772 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:45:46.772 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:46.772 7f3b7a798390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:45:46.772 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:46.772 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:46.772 7f3b7a798390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:45:46.773 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:50.772 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:45:55.778 7f3b772106d8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:45:55.779 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:45:55.779 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:55.779 7f3b772106d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:45:55.779 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:55.779 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:45:55.780 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:45:55.780 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:55.780 7f3b772106d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:45:55.780 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:55.780 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:55.780 7f3b772106d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:45:55.780 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:59.781 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:46:04.784 7f3b772106d8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:46:04.784 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:46:04.784 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:04.784 7f3b772106d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:46:04.785 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:04.785 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:46:04.785 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:46:04.785 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:04.785 7f3b772106d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:46:04.785 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:04.786 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:04.786 7f3b772106d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:46:04.786 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:08.785 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:46:13.790 7f3b772106d8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:46:13.790 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:46:13.791 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:13.791 7f3b772106d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:46:13.791 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:13.791 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:46:13.791 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:46:13.792 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:13.792 7f3b772106d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:46:13.792 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:13.792 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:13.792 7f3b772106d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:46:13.792 7f3b772106d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:14.389 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:46:14.703 7f3b786dbac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:14.703 7f3b786dbac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 database_usage = [ << disk_space_used_1k = 50998036 disk_space_available_1k = 30092448 config_db_size_1k = 1028 >>, ] >> 03/31/2025 02:46:14.703 7f3b786dbac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:17.791 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:46:22.794 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:46:22.795 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:46:22.795 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:22.795 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:46:22.795 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:22.795 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:46:22.796 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:46:22.796 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:22.796 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:46:22.796 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:22.796 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:22.796 7f3b7aa50d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:46:22.796 7f3b7aa50d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:22.858 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50998204, partition_space_available_1k=30092280, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1313952, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=207592, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43244, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14943864, free=9047288, buffers=844596, cached=7829364, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.22, five_min_avg=3.28, fifteen_min_avg=3.13, cpu_share=2.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:46:22.858 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:22.858 7f3b7acd82e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 50998204 partition_space_available_1k = 30092280 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1313952 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 207592 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 43244 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 14943864 free = 9047288 buffers = 844596 cached = 7829364 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 3.22 five_min_avg = 3.28 fifteen_min_avg = 3.13 cpu_share = 2.5 node_type = config-database-node >> >> 03/31/2025 02:46:22.858 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:26.799 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:46:31.805 7f3b7acd82e8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:46:31.805 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:46:31.805 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:31.805 7f3b7acd82e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:46:31.806 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:31.806 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:46:31.806 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:46:31.806 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:31.806 7f3b7acd82e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:46:31.806 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:31.807 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:31.807 7f3b7acd82e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:46:31.807 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:35.806 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:46:40.809 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:46:40.809 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:46:40.809 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:40.810 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:46:40.810 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:40.810 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:46:40.810 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:46:40.811 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:40.811 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:46:40.811 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:40.811 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:40.811 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:46:40.811 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:44.813 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:46:49.814 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:46:49.815 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:46:49.815 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:49.815 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:46:49.815 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:49.815 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:46:49.816 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:46:49.816 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:49.816 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:46:49.816 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:49.816 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:49.816 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:46:49.816 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:52.922 7f3b7662b8d0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:46:53.815 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:46:58.822 7f3b7662b8d0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:46:58.822 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:46:58.822 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:58.822 7f3b7662b8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:46:58.822 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:58.822 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:46:58.823 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:46:58.823 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:58.823 7f3b7662b8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:46:58.823 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:58.823 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:58.823 7f3b7662b8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:46:58.823 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:02.825 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:47:07.830 7f3b7662b8d0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:47:07.830 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:47:07.831 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:07.831 7f3b7662b8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:47:07.831 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:07.831 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:47:07.832 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:47:07.832 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:07.832 7f3b7662b8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:47:07.832 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:07.832 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:07.832 7f3b7662b8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:47:07.833 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:11.834 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:47:14.386 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:47:16.836 7f3b7662b8d0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:47:16.836 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:47:16.836 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:16.836 7f3b7662b8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:47:16.836 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:16.837 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:16.837 7f3b7662b8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 database_usage = [ << disk_space_used_1k = 50997220 disk_space_available_1k = 30093264 config_db_size_1k = 1028 >>, ] >> 03/31/2025 02:47:16.837 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:16.837 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:47:16.837 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:47:16.837 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:16.838 7f3b7662b8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:47:16.838 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:16.838 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:16.838 7f3b7662b8d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:47:16.838 7f3b7662b8d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:20.837 7f3b768beb70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:47:22.865 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50997388, partition_space_available_1k=30093096, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1313524, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=207880, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43308, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14964676, free=9026400, buffers=844596, cached=7829440, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.32, five_min_avg=3.01, fifteen_min_avg=3.05, cpu_share=2.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:47:25.842 7f3b768beb70 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:47:25.843 7f3b768beb70 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:47:25.843 7f3b768beb70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:25.843 7f3b768beb70 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:47:25.843 7f3b768beb70 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:25.843 7f3b768beb70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:25.843 7f3b768beb70 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 50997388 partition_space_available_1k = 30093096 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1313524 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 207880 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 43308 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 14964676 free = 9026400 buffers = 844596 cached = 7829440 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 2.32 five_min_avg = 3.01 fifteen_min_avg = 3.05 cpu_share = 2.5 node_type = config-database-node >> >> 03/31/2025 02:47:25.843 7f3b768beb70 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:25.843 7f3b768beb70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:47:25.844 7f3b768beb70 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:47:25.844 7f3b768beb70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:25.844 7f3b768beb70 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:47:25.844 7f3b768beb70 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:25.844 7f3b768beb70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:25.844 7f3b768beb70 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:47:25.844 7f3b768beb70 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:29.845 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:47:34.846 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:47:34.847 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:47:34.847 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:34.847 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:47:34.847 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:34.847 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:47:34.847 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:47:34.848 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:34.848 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:47:34.848 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:34.848 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:34.848 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:47:34.848 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:38.849 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:47:43.855 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:47:43.855 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:47:43.855 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:43.855 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:47:43.856 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:43.856 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:47:43.856 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:47:43.856 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:43.856 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:47:43.856 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:43.857 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:43.857 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:47:43.857 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:47.859 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:47:52.862 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:47:52.863 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:47:52.863 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:52.863 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:47:52.863 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:52.863 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:47:52.863 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:47:52.864 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:52.864 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:47:52.864 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:52.864 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:52.864 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:47:52.864 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:56.865 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:48:01.870 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:48:01.871 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:48:01.871 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:01.871 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:48:01.871 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:01.871 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:48:01.871 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:48:01.872 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:01.872 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:48:01.872 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:01.872 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:01.872 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:48:01.872 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:05.872 7f3b768acc18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:48:10.878 7f3b768acc18 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:48:10.878 7f3b768acc18 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:48:10.879 7f3b768acc18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:10.879 7f3b768acc18 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:48:10.879 7f3b768acc18 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:10.879 7f3b768acc18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:48:10.880 7f3b768acc18 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:48:10.880 7f3b768acc18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:10.880 7f3b768acc18 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:48:10.880 7f3b768acc18 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:10.880 7f3b768acc18 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:10.880 7f3b768acc18 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:48:10.880 7f3b768acc18 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:14.388 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:48:14.704 7f3b786db6d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:14.705 7f3b786db6d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 database_usage = [ << disk_space_used_1k = 51000980 disk_space_available_1k = 30089504 config_db_size_1k = 1028 >>, ] >> 03/31/2025 02:48:14.705 7f3b786db6d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:14.878 7f3b786dbf60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:48:19.880 7f3b786dbf60 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:48:19.880 7f3b786dbf60 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:48:19.880 7f3b786dbf60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:19.881 7f3b786dbf60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:48:19.881 7f3b786dbf60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:19.881 7f3b786dbf60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:48:19.881 7f3b786dbf60 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:48:19.881 7f3b786dbf60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:19.882 7f3b786dbf60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:48:19.882 7f3b786dbf60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:19.882 7f3b786dbf60 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:19.882 7f3b786dbf60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:48:19.882 7f3b786dbf60 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:22.868 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51001148, partition_space_available_1k=30089336, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1314224, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=208180, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43316, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15003040, free=8983212, buffers=844596, cached=7834264, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.99, five_min_avg=2.75, fifteen_min_avg=2.95, cpu_share=2.38, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:48:22.868 7f3b77dd5630 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:22.869 7f3b77dd5630 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51001148 partition_space_available_1k = 30089336 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1314224 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 208180 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 43316 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15003040 free = 8983212 buffers = 844596 cached = 7834264 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 1.99 five_min_avg = 2.75 fifteen_min_avg = 2.95 cpu_share = 2.38 node_type = config-database-node >> >> 03/31/2025 02:48:22.869 7f3b77dd5630 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:23.881 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:48:28.885 7f3b7aa502e8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:48:28.885 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:48:28.885 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:28.885 7f3b7aa502e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:48:28.885 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:28.885 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:48:28.886 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:48:28.886 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:28.886 7f3b7aa502e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:48:28.886 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:28.886 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:28.886 7f3b7aa502e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:48:28.887 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:32.888 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:48:37.892 7f3b7aa502e8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:48:37.893 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:48:37.893 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:37.893 7f3b7aa502e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:48:37.893 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:37.893 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:48:37.894 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:48:37.894 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:37.894 7f3b7aa502e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:48:37.894 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:37.894 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:37.894 7f3b7aa502e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:48:37.894 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:41.897 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:48:46.903 7f3b7aa502e8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:48:46.903 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:48:46.903 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:46.904 7f3b7aa502e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:48:46.904 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:46.904 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:48:46.904 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:48:46.904 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:46.905 7f3b7aa502e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:48:46.905 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:46.905 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:46.905 7f3b7aa502e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:48:46.905 7f3b7aa502e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:50.903 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:48:55.909 7f3b80d1a048 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:48:55.909 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:48:55.909 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:55.909 7f3b80d1a048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:48:55.909 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:55.909 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:48:55.910 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:48:55.910 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:55.910 7f3b80d1a048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:48:55.910 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:55.910 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:55.910 7f3b80d1a048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:48:55.911 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:59.913 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:49:04.918 7f3b80d1a048 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:49:04.919 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:49:04.919 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:04.919 7f3b80d1a048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:49:04.919 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:04.919 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:49:04.920 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:49:04.920 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:04.920 7f3b80d1a048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:49:04.920 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:04.920 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:04.920 7f3b80d1a048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:49:04.920 7f3b80d1a048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:05.169 7f3b80d1a048 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:49:08.921 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:49:13.792 7f3b7721eac8 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:49:13.923 7f3b7a798390 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:49:13.923 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:49:13.923 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:13.924 7f3b7a798390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:49:13.924 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:13.924 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:49:13.924 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:49:13.924 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:13.924 7f3b7a798390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:49:13.924 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:13.925 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:13.925 7f3b7a798390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:49:13.925 7f3b7a798390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:14.413 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:49:14.757 7f3b772078d0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:14.757 7f3b772078d0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 database_usage = [ << disk_space_used_1k = 50999980 disk_space_available_1k = 30090504 config_db_size_1k = 1028 >>, ] >> 03/31/2025 02:49:14.757 7f3b772078d0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:17.924 7f3b786fae10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:49:22.889 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51000260, partition_space_available_1k=30090224, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1314524, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=208512, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43416, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15071012, free=8909544, buffers=844604, cached=7839952, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.13, five_min_avg=2.65, fifteen_min_avg=2.91, cpu_share=2.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:49:22.927 7f3b786fae10 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:49:22.927 7f3b786fae10 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:49:22.927 7f3b786fae10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:22.927 7f3b786fae10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:49:22.927 7f3b786fae10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:22.927 7f3b786fae10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:22.928 7f3b786fae10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51000260 partition_space_available_1k = 30090224 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1314524 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 208512 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 43416 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15071012 free = 8909544 buffers = 844604 cached = 7839952 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 2.13 five_min_avg = 2.65 fifteen_min_avg = 2.91 cpu_share = 2.5 node_type = config-database-node >> >> 03/31/2025 02:49:22.928 7f3b786fae10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:22.928 7f3b786fae10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:49:22.928 7f3b786fae10 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:49:22.928 7f3b786fae10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:22.929 7f3b786fae10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:49:22.929 7f3b786fae10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:22.929 7f3b786fae10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:22.929 7f3b786fae10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:49:22.929 7f3b786fae10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:25.017 7f3b7acebcc0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:49:26.927 7f3b7acebcc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:49:31.933 7f3b7acebcc0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:49:31.934 7f3b7acebcc0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:49:31.934 7f3b7acebcc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:31.934 7f3b7acebcc0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:49:31.934 7f3b7acebcc0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:31.934 7f3b7acebcc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:49:31.935 7f3b7acebcc0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:49:31.935 7f3b7acebcc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:31.935 7f3b7acebcc0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:49:31.935 7f3b7acebcc0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:31.935 7f3b7acebcc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:31.935 7f3b7acebcc0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:49:31.935 7f3b7acebcc0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:35.934 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:49:40.940 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:49:40.940 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:49:40.940 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:40.941 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:49:40.941 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:40.941 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:49:40.941 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:49:40.941 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:40.941 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:49:40.942 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:40.942 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:40.942 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:49:40.942 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:44.943 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:49:49.949 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:49:49.950 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:49:49.950 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:49.950 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:49:49.950 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:49.950 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:49:49.951 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:49:49.951 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:49.951 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:49:49.951 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:49.951 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:49.951 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:49:49.951 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:53.953 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:49:58.958 7f3b7bf2cac8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:49:58.959 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:49:58.959 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:58.959 7f3b7bf2cac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:49:58.959 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:58.959 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:49:58.960 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:49:58.960 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:58.960 7f3b7bf2cac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:49:58.960 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:58.960 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:58.961 7f3b7bf2cac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:49:58.961 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:02.962 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:50:07.965 7f3b7bf2cac8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:50:07.965 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:50:07.965 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:07.966 7f3b7bf2cac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:50:07.966 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:07.966 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:50:07.966 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:50:07.966 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:07.966 7f3b7bf2cac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:50:07.966 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:07.967 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:07.967 7f3b7bf2cac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:50:07.967 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:11.969 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:50:14.452 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:50:16.971 7f3b7bf2cac8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:50:16.972 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:50:16.972 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:16.972 7f3b7bf2cac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:50:16.972 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:16.972 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:16.972 7f3b7bf2cac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 database_usage = [ << disk_space_used_1k = 51004480 disk_space_available_1k = 30086004 config_db_size_1k = 1028 >>, ] >> 03/31/2025 02:50:16.973 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:16.973 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:50:16.973 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:50:16.973 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:16.973 7f3b7bf2cac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:50:16.973 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:16.973 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:16.974 7f3b7bf2cac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:50:16.975 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:20.972 7f3b77dd64e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:50:22.933 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51004660, partition_space_available_1k=30085824, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1316244, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=208792, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43464, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15357648, free=8616672, buffers=844608, cached=7846184, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.78, five_min_avg=2.93, fifteen_min_avg=2.98, cpu_share=3.12, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:50:25.977 7f3b77dd64e0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:50:25.977 7f3b77dd64e0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:50:25.977 7f3b77dd64e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:25.977 7f3b77dd64e0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:50:25.978 7f3b77dd64e0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:25.978 7f3b77dd64e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:25.978 7f3b77dd64e0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51004660 partition_space_available_1k = 30085824 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1316244 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 208792 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 43464 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15357648 free = 8616672 buffers = 844608 cached = 7846184 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 3.78 five_min_avg = 2.93 fifteen_min_avg = 2.98 cpu_share = 3.12 node_type = config-database-node >> >> 03/31/2025 02:50:25.978 7f3b77dd64e0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:25.978 7f3b77dd64e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:50:25.979 7f3b77dd64e0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:50:25.979 7f3b77dd64e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:25.979 7f3b77dd64e0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:50:25.979 7f3b77dd64e0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:25.979 7f3b77dd64e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:25.980 7f3b77dd64e0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:50:25.980 7f3b77dd64e0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:29.981 7f3b7acd4390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:50:34.985 7f3b7acd4390 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:50:34.986 7f3b7acd4390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:50:34.986 7f3b7acd4390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:34.986 7f3b7acd4390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:50:34.986 7f3b7acd4390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:34.986 7f3b7acd4390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:50:34.986 7f3b7acd4390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:50:34.987 7f3b7acd4390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:34.987 7f3b7acd4390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:50:34.987 7f3b7acd4390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:34.987 7f3b7acd4390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:34.987 7f3b7acd4390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:50:34.987 7f3b7acd4390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:38.986 7f3b768ac630 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:50:43.991 7f3b768ac630 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:50:43.991 7f3b768ac630 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:50:43.991 7f3b768ac630 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:43.991 7f3b768ac630 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:50:43.991 7f3b768ac630 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:43.991 7f3b768ac630 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:50:43.992 7f3b768ac630 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:50:43.992 7f3b768ac630 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:43.992 7f3b768ac630 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:50:43.992 7f3b768ac630 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:43.992 7f3b768ac630 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:43.992 7f3b768ac630 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:50:43.992 7f3b768ac630 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:47.995 7f3b768ac630 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:50:52.997 7f3b768ac630 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:50:52.997 7f3b768ac630 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:50:52.998 7f3b768ac630 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:52.998 7f3b768ac630 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:50:52.998 7f3b768ac630 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:52.998 7f3b768ac630 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:50:52.998 7f3b768ac630 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:50:52.999 7f3b768ac630 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:52.999 7f3b768ac630 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:50:52.999 7f3b768ac630 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:53.001 7f3b768ac630 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:53.001 7f3b768ac630 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:50:53.002 7f3b768ac630 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:53.973 7f3b7a836d68 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:50:57.001 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:51:02.005 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:51:02.005 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:51:02.005 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:02.006 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:51:02.006 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:02.006 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:51:02.007 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:51:02.007 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:02.007 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:51:02.007 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:02.007 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:02.007 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:51:02.008 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:06.009 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:51:11.013 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:51:11.014 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:51:11.014 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:11.014 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:51:11.014 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:11.014 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:51:11.015 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:51:11.015 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:11.015 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:51:11.015 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:11.015 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:11.015 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:51:11.016 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:14.422 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:51:14.748 7f3b7860fcc0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:14.748 7f3b7860fcc0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 database_usage = [ << disk_space_used_1k = 51014500 disk_space_available_1k = 30075984 config_db_size_1k = 1028 >>, ] >> 03/31/2025 02:51:14.748 7f3b7860fcc0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:15.014 7f3b7860fe10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:51:20.017 7f3b7860fe10 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:51:20.017 7f3b7860fe10 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:51:20.017 7f3b7860fe10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:20.018 7f3b7860fe10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:51:20.018 7f3b7860fe10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:20.018 7f3b7860fe10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:51:20.019 7f3b7860fe10 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:51:20.019 7f3b7860fe10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:20.019 7f3b7860fe10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:51:20.019 7f3b7860fe10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:20.019 7f3b7860fe10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:20.019 7f3b7860fe10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:51:20.019 7f3b7860fe10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:22.876 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51014664, partition_space_available_1k=30075820, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1315724, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=209112, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43488, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15011712, free=8949852, buffers=844612, cached=7858936, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.82, five_min_avg=2.89, fifteen_min_avg=2.97, cpu_share=3.62, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:51:22.876 7f3b7721f240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:22.876 7f3b7721f240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51014664 partition_space_available_1k = 30075820 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1315724 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 209112 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 43488 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15011712 free = 8949852 buffers = 844612 cached = 7858936 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 2.82 five_min_avg = 2.89 fifteen_min_avg = 2.97 cpu_share = 3.62 node_type = config-database-node >> >> 03/31/2025 02:51:22.877 7f3b7721f240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:24.018 7f3b7721f240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:51:29.023 7f3b7721f240 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:51:29.023 7f3b7721f240 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:51:29.023 7f3b7721f240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:29.023 7f3b7721f240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:51:29.023 7f3b7721f240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:29.023 7f3b7721f240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:51:29.024 7f3b7721f240 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:51:29.024 7f3b7721f240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:29.024 7f3b7721f240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:51:29.024 7f3b7721f240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:29.024 7f3b7721f240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:29.024 7f3b7721f240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:51:29.024 7f3b7721f240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:33.027 7f3b7721f240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:51:38.029 7f3b7721f240 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:51:38.029 7f3b7721f240 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:51:38.029 7f3b7721f240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:38.029 7f3b7721f240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:51:38.029 7f3b7721f240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:38.030 7f3b7721f240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:51:38.030 7f3b7721f240 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:51:38.030 7f3b7721f240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:38.030 7f3b7721f240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:51:38.030 7f3b7721f240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:38.030 7f3b7721f240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:38.030 7f3b7721f240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:51:38.031 7f3b7721f240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:42.033 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:51:47.039 7f3b7acebb70 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:51:47.039 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:51:47.039 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:47.039 7f3b7acebb70 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:51:47.039 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:47.039 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:51:47.040 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:51:47.040 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:47.040 7f3b7acebb70 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:51:47.040 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:47.041 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:47.041 7f3b7acebb70 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:51:47.041 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:51.043 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:51:56.048 7f3b7acebb70 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:51:56.048 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:51:56.048 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:56.049 7f3b7acebb70 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:51:56.049 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:56.049 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:51:56.049 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:51:56.049 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:56.050 7f3b7acebb70 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:51:56.050 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:56.050 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:56.050 7f3b7acebb70 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:51:56.050 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:00.049 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:52:05.055 7f3b7acebb70 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:52:05.056 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:52:05.056 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:05.056 7f3b7acebb70 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:52:05.056 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:05.056 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:52:05.057 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:52:05.057 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:05.057 7f3b7acebb70 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:52:05.057 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:05.057 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:05.057 7f3b7acebb70 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:52:05.057 7f3b7acebb70 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:09.057 7f3b77207390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:52:14.063 7f3b77207390 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:52:14.063 7f3b77207390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:52:14.064 7f3b77207390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:14.064 7f3b77207390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:52:14.064 7f3b77207390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:14.064 7f3b77207390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:52:14.064 7f3b77207390 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:52:14.065 7f3b77207390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:14.065 7f3b77207390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:52:14.065 7f3b77207390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:14.065 7f3b77207390 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:14.065 7f3b77207390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:52:14.065 7f3b77207390 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:14.408 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:52:14.734 7f3b7ae4b4e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:14.734 7f3b7ae4b4e0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 database_usage = [ << disk_space_used_1k = 51014192 disk_space_available_1k = 30076292 config_db_size_1k = 1028 >>, ] >> 03/31/2025 02:52:14.734 7f3b7ae4b4e0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:18.063 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:52:22.881 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51014372, partition_space_available_1k=30076112, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1317292, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=209408, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43560, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15049740, free=8906324, buffers=844612, cached=7864436, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.36, five_min_avg=2.73, fifteen_min_avg=2.91, cpu_share=2.62, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:52:23.066 7f3b81c630f0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:52:23.066 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:52:23.066 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:23.067 7f3b81c630f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:52:23.067 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:23.067 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:23.067 7f3b81c630f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51014372 partition_space_available_1k = 30076112 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1317292 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 209408 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 43560 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15049740 free = 8906324 buffers = 844612 cached = 7864436 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 2.36 five_min_avg = 2.73 fifteen_min_avg = 2.91 cpu_share = 2.62 node_type = config-database-node >> >> 03/31/2025 02:52:23.067 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:23.067 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:52:23.068 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:52:23.068 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:23.068 7f3b81c630f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:52:23.068 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:23.068 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:23.068 7f3b81c630f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:52:23.068 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:27.069 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:52:32.075 7f3b81c630f0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:52:32.076 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:52:32.076 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:32.076 7f3b81c630f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:52:32.076 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:32.076 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:52:32.077 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:52:32.077 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:32.077 7f3b81c630f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:52:32.077 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:32.077 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:32.077 7f3b81c630f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:52:32.078 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:36.079 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:52:41.084 7f3b81c630f0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:52:41.085 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:52:41.085 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:41.085 7f3b81c630f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:52:41.085 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:41.085 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:52:41.086 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:52:41.086 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:41.086 7f3b81c630f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:52:41.086 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:41.086 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:41.087 7f3b81c630f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:52:41.087 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:45.089 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:52:50.093 7f3b81c630f0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:52:50.093 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:52:50.093 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:50.093 7f3b81c630f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:52:50.093 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:50.094 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:52:50.094 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:52:50.094 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:50.094 7f3b81c630f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:52:50.094 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:50.094 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:50.095 7f3b81c630f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:52:50.095 7f3b81c630f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:54.094 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:52:59.100 7f3b7a7e3828 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:52:59.101 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:52:59.101 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:59.101 7f3b7a7e3828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:52:59.101 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:59.101 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:52:59.102 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:52:59.102 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:59.102 7f3b7a7e3828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:52:59.102 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:59.103 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:59.103 7f3b7a7e3828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:52:59.103 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:03.104 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:53:08.105 7f3b7a7e3828 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:53:08.106 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:53:08.106 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:08.106 7f3b7a7e3828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:53:08.106 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:08.106 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:53:08.107 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:53:08.107 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:08.107 7f3b7a7e3828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:53:08.107 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:08.107 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:08.107 7f3b7a7e3828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:53:08.107 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:12.109 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:53:14.423 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:53:17.112 7f3b7a7e3828 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:53:17.112 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:53:17.112 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:17.112 7f3b7a7e3828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:53:17.112 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:17.112 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:17.112 7f3b7a7e3828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 database_usage = [ << disk_space_used_1k = 51017804 disk_space_available_1k = 30072680 config_db_size_1k = 1028 >>, ] >> 03/31/2025 02:53:17.113 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:17.113 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:53:17.113 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:53:17.113 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:17.113 7f3b7a7e3828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:53:17.114 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:17.114 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:17.114 7f3b7a7e3828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:53:17.114 7f3b7a7e3828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:21.113 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:53:22.888 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51017984, partition_space_available_1k=30072500, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1317792, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=209700, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43600, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15026924, free=8927024, buffers=844612, cached=7866552, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.79, five_min_avg=2.48, fifteen_min_avg=2.81, cpu_share=2.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:53:26.117 7f3b7acd82e8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:53:26.117 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:53:26.117 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:26.117 7f3b7acd82e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:53:26.117 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:26.118 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:26.118 7f3b7acd82e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51017984 partition_space_available_1k = 30072500 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1317792 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 209700 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 43600 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15026924 free = 8927024 buffers = 844612 cached = 7866552 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 1.79 five_min_avg = 2.48 fifteen_min_avg = 2.81 cpu_share = 2.5 node_type = config-database-node >> >> 03/31/2025 02:53:26.118 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:26.118 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:53:26.118 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:53:26.119 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:26.119 7f3b7acd82e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:53:26.119 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:26.119 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:26.119 7f3b7acd82e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:53:26.119 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:30.121 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:53:35.125 7f3b7acd82e8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:53:35.125 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:53:35.125 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:35.125 7f3b7acd82e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:53:35.126 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:35.126 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:53:35.126 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:53:35.126 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:35.126 7f3b7acd82e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:53:35.126 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:35.126 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:35.127 7f3b7acd82e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:53:35.127 7f3b7acd82e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:39.125 7f3b772074e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:53:44.130 7f3b772074e0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:53:44.130 7f3b772074e0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:53:44.131 7f3b772074e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:44.131 7f3b772074e0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:53:44.131 7f3b772074e0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:44.131 7f3b772074e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:53:44.131 7f3b772074e0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:53:44.132 7f3b772074e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:44.132 7f3b772074e0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:53:44.132 7f3b772074e0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:44.132 7f3b772074e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:44.132 7f3b772074e0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:53:44.132 7f3b772074e0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:48.133 7f3b772074e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:53:53.138 7f3b772074e0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:53:53.138 7f3b772074e0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:53:53.138 7f3b772074e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:53.139 7f3b772074e0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:53:53.139 7f3b772074e0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:53.139 7f3b772074e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:53:53.139 7f3b772074e0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:53:53.139 7f3b772074e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:53.139 7f3b772074e0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:53:53.140 7f3b772074e0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:53.140 7f3b772074e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:53.140 7f3b772074e0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:53:53.140 7f3b772074e0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:57.142 7f3b77d83828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:54:02.148 7f3b77d83828 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:54:02.149 7f3b77d83828 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:54:02.149 7f3b77d83828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:02.149 7f3b77d83828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:54:02.149 7f3b77d83828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:02.149 7f3b77d83828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:54:02.150 7f3b77d83828 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:54:02.150 7f3b77d83828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:02.150 7f3b77d83828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:54:02.150 7f3b77d83828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:02.150 7f3b77d83828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:02.150 7f3b77d83828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:54:02.150 7f3b77d83828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:06.151 7f3b77d83828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:54:11.156 7f3b77d83828 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:54:11.156 7f3b77d83828 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:54:11.157 7f3b77d83828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:11.157 7f3b77d83828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:54:11.157 7f3b77d83828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:11.157 7f3b77d83828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:54:11.157 7f3b77d83828 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:54:11.158 7f3b77d83828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:11.158 7f3b77d83828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:54:11.158 7f3b77d83828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:11.158 7f3b77d83828 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:11.158 7f3b77d83828 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:54:11.158 7f3b77d83828 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:14.413 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:54:14.738 7f3b78a71048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:14.738 7f3b78a71048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 database_usage = [ << disk_space_used_1k = 51021316 disk_space_available_1k = 30069168 config_db_size_1k = 1028 >>, ] >> 03/31/2025 02:54:14.738 7f3b78a71048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:15.156 7f3b77213eb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:54:18.126 7f3b786db240 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:54:20.159 7f3b77213eb8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:54:20.160 7f3b77213eb8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:54:20.160 7f3b77213eb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:20.160 7f3b77213eb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:54:20.160 7f3b77213eb8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:20.160 7f3b77213eb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:54:20.161 7f3b77213eb8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:54:20.161 7f3b77213eb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:20.161 7f3b77213eb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:54:20.161 7f3b77213eb8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:20.161 7f3b77213eb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:20.162 7f3b77213eb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:54:20.162 7f3b77213eb8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:22.902 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51021472, partition_space_available_1k=30069012, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1324804, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=210008, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43616, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15092344, free=8857596, buffers=844612, cached=7870560, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.04, five_min_avg=2.41, fifteen_min_avg=2.76, cpu_share=2.62, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:54:22.903 7f3b789dbb70 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:22.903 7f3b789dbb70 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51021472 partition_space_available_1k = 30069012 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1324804 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 210008 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 43616 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15092344 free = 8857596 buffers = 844612 cached = 7870560 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 2.04 five_min_avg = 2.41 fifteen_min_avg = 2.76 cpu_share = 2.62 node_type = config-database-node >> >> 03/31/2025 02:54:22.903 7f3b789dbb70 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:24.160 7f3b7acd0e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:54:26.737 7f3b786c2b70 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:54:29.165 7f3b7acd0e10 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:54:29.165 7f3b7acd0e10 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:54:29.165 7f3b7acd0e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:29.165 7f3b7acd0e10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:54:29.166 7f3b7acd0e10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:29.166 7f3b7acd0e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:54:29.166 7f3b7acd0e10 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:54:29.166 7f3b7acd0e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:29.166 7f3b7acd0e10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:54:29.166 7f3b7acd0e10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:29.167 7f3b7acd0e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:29.167 7f3b7acd0e10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:54:29.167 7f3b7acd0e10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:33.169 7f3b7acd0e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:54:37.956 7f3b789fdeb8 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:54:38.171 7f3b7acd0e10 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:54:38.171 7f3b7acd0e10 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:54:38.172 7f3b7acd0e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:38.172 7f3b7acd0e10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:54:38.172 7f3b7acd0e10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:38.172 7f3b7acd0e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:54:38.173 7f3b7acd0e10 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:54:38.173 7f3b7acd0e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:38.173 7f3b7acd0e10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:54:38.173 7f3b7acd0e10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:38.173 7f3b7acd0e10 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:38.173 7f3b7acd0e10 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:54:38.173 7f3b7acd0e10 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:42.172 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:54:47.177 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:54:47.177 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:54:47.177 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:47.177 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:54:47.178 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:47.178 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:54:47.178 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:54:47.178 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:47.178 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:54:47.178 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:47.179 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:47.179 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:54:47.179 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:51.181 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:54:56.183 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:54:56.183 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:54:56.183 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:56.184 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:54:56.184 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:56.184 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:54:56.184 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:54:56.184 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:56.185 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:54:56.185 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:56.185 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:56.185 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:54:56.185 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:00.187 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:55:05.193 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:55:05.193 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:55:05.193 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:05.194 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:55:05.194 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:05.194 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:55:05.194 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:55:05.194 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:05.195 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:55:05.195 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:05.195 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:05.195 7f3b7a836d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:55:05.195 7f3b7a836d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:09.194 7f3b7a7b4a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:55:14.197 7f3b7a7b4a20 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:55:14.197 7f3b7a7b4a20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:55:14.197 7f3b7a7b4a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:14.197 7f3b7a7b4a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:55:14.197 7f3b7a7b4a20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:14.197 7f3b7a7b4a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:55:14.198 7f3b7a7b4a20 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:55:14.198 7f3b7a7b4a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:14.198 7f3b7a7b4a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:55:14.198 7f3b7a7b4a20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:14.198 7f3b7a7b4a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:14.198 7f3b7a7b4a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:55:14.198 7f3b7a7b4a20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:14.422 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:55:14.744 7f3b7a7d5048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:14.744 7f3b7a7d5048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 database_usage = [ << disk_space_used_1k = 51021788 disk_space_available_1k = 30068696 config_db_size_1k = 1028 >>, ] >> 03/31/2025 02:55:14.744 7f3b7a7d5048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:18.198 7f3b77dd56d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:55:22.895 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51021964, partition_space_available_1k=30068520, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1317940, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=210328, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43684, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15026748, free=8920804, buffers=844612, cached=7872948, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.78, five_min_avg=2.27, fifteen_min_avg=2.69, cpu_share=2.62, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:55:23.201 7f3b77dd56d8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:55:23.201 7f3b77dd56d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:55:23.201 7f3b77dd56d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:23.201 7f3b77dd56d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:55:23.201 7f3b77dd56d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:23.201 7f3b77dd56d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:23.202 7f3b77dd56d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51021964 partition_space_available_1k = 30068520 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1317940 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 210328 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 43684 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15026748 free = 8920804 buffers = 844612 cached = 7872948 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 1.78 five_min_avg = 2.27 fifteen_min_avg = 2.69 cpu_share = 2.62 node_type = config-database-node >> >> 03/31/2025 02:55:23.202 7f3b77dd56d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:23.202 7f3b77dd56d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:55:23.202 7f3b77dd56d8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:55:23.202 7f3b77dd56d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:23.203 7f3b77dd56d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:55:23.203 7f3b77dd56d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:23.203 7f3b77dd56d8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:23.203 7f3b77dd56d8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:55:23.203 7f3b77dd56d8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:27.204 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:55:32.210 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:55:32.211 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:55:32.211 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:32.211 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:55:32.211 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:32.211 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:55:32.212 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:55:32.212 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:32.212 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:55:32.212 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:32.212 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:32.212 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:55:32.212 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:36.214 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:55:41.218 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:55:41.219 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:55:41.219 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:41.219 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:55:41.219 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:41.220 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:55:41.220 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:55:41.220 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:41.223 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:55:41.224 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:41.224 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:41.224 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:55:41.224 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:45.223 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:55:50.229 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:55:50.230 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:55:50.230 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:50.230 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:55:50.230 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:50.230 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:55:50.231 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:55:50.231 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:50.231 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:55:50.231 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:50.231 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:50.231 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:55:50.231 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:54.233 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:55:59.239 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:55:59.240 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:55:59.240 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:59.240 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:55:59.240 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:59.240 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:55:59.241 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:55:59.241 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:59.241 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:55:59.241 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:59.241 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:59.241 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:55:59.241 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:03.243 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:56:08.250 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:56:08.250 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:56:08.250 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:08.250 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:56:08.250 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:08.250 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:56:08.251 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:56:08.251 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:08.251 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:56:08.251 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:08.251 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:08.252 7f3b7acd40f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:56:08.252 7f3b7acd40f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:12.252 7f3b7ac7b0f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:56:14.417 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:56:17.254 7f3b7ac7b0f0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:56:17.254 7f3b7ac7b0f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:56:17.255 7f3b7ac7b0f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:17.255 7f3b7ac7b0f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:56:17.255 7f3b7ac7b0f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:17.255 7f3b7ac7b0f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:17.255 7f3b7ac7b0f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 database_usage = [ << disk_space_used_1k = 51026132 disk_space_available_1k = 30064352 config_db_size_1k = 1028 >>, ] >> 03/31/2025 02:56:17.255 7f3b7ac7b0f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:17.255 7f3b7ac7b0f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:56:17.256 7f3b7ac7b0f0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:56:17.256 7f3b7ac7b0f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:17.256 7f3b7ac7b0f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:56:17.256 7f3b7ac7b0f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:17.256 7f3b7ac7b0f0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:17.256 7f3b7ac7b0f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:56:17.256 7f3b7ac7b0f0 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:21.255 7f3b7860feb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:56:22.888 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51026348, partition_space_available_1k=30064136, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1334556, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=210632, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43736, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15073212, free=8856008, buffers=844620, cached=7891272, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.79, five_min_avg=3.23, fifteen_min_avg=3.0, cpu_share=3.88, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:56:26.258 7f3b7860feb8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:56:26.259 7f3b7860feb8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:56:26.259 7f3b7860feb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:26.259 7f3b7860feb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:56:26.259 7f3b7860feb8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:26.260 7f3b7860feb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:26.260 7f3b7860feb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51026348 partition_space_available_1k = 30064136 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1334556 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 210632 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 43736 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15073212 free = 8856008 buffers = 844620 cached = 7891272 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 4.79 five_min_avg = 3.23 fifteen_min_avg = 3.0 cpu_share = 3.88 node_type = config-database-node >> >> 03/31/2025 02:56:26.260 7f3b7860feb8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:26.260 7f3b7860feb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:56:26.261 7f3b7860feb8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:56:26.261 7f3b7860feb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:26.261 7f3b7860feb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:56:26.262 7f3b7860feb8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:26.262 7f3b7860feb8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:26.262 7f3b7860feb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:56:26.262 7f3b7860feb8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:30.262 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:56:35.269 7f3b789db438 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:56:35.269 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:56:35.269 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:35.269 7f3b789db438 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:56:35.269 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:35.270 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:56:35.270 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:56:35.270 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:35.270 7f3b789db438 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:56:35.270 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:35.270 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:35.271 7f3b789db438 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:56:35.271 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:39.273 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:56:43.363 7f3b7a836d68 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/31/2025 02:56:44.275 7f3b789db438 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:56:44.275 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:56:44.275 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:44.275 7f3b789db438 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:56:44.275 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:44.276 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:56:44.276 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:56:44.276 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:44.276 7f3b789db438 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:56:44.276 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:44.277 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:44.277 7f3b789db438 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:56:44.277 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:48.279 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:56:53.285 7f3b789db438 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:56:53.285 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:56:53.286 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:53.286 7f3b789db438 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:56:53.286 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:53.286 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:56:53.286 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:56:53.287 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:53.287 7f3b789db438 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:56:53.287 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:53.287 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:53.287 7f3b789db438 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:56:53.287 7f3b789db438 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:57.286 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:57:02.289 7f3b80021780 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:57:02.289 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:57:02.289 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:02.289 7f3b80021780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:57:02.289 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:02.290 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:57:02.290 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:57:02.290 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:02.290 7f3b80021780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:57:02.291 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:02.291 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:02.291 7f3b80021780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:57:02.291 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:06.293 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:57:11.295 7f3b80021780 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:57:11.296 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:57:11.296 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:11.296 7f3b80021780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:57:11.296 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:11.296 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:57:11.297 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:57:11.297 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:11.297 7f3b80021780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:57:11.297 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:11.297 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:11.297 7f3b80021780 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:57:11.297 7f3b80021780 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:14.445 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/31/2025 02:57:14.790 7f3b7721e240 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:14.791 7f3b7721e240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 database_usage = [ << disk_space_used_1k = 51031020 disk_space_available_1k = 30059464 config_db_size_1k = 1028 >>, ] >> 03/31/2025 02:57:14.791 7f3b7721e240 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:15.296 7f3b768ac2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:57:20.300 7f3b768ac2e8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:57:20.300 7f3b768ac2e8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:57:20.300 7f3b768ac2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:20.300 7f3b768ac2e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:57:20.300 7f3b768ac2e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:20.301 7f3b768ac2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:57:20.301 7f3b768ac2e8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:57:20.301 7f3b768ac2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:20.302 7f3b768ac2e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:57:20.302 7f3b768ac2e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:20.302 7f3b768ac2e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:20.302 7f3b768ac2e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:57:20.302 7f3b768ac2e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:22.888 7f3b7ae92688 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51031252, partition_space_available_1k=30059232, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1317876, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=210936, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43752, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15048428, free=8873100, buffers=844620, cached=7898964, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.71, five_min_avg=2.91, fifteen_min_avg=2.9, cpu_share=2.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/31/2025 02:57:22.889 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:22.889 7f3b7bf2cac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51031252 partition_space_available_1k = 30059232 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1317876 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 210936 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 43752 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15048428 free = 8873100 buffers = 844620 cached = 7898964 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 2.71 five_min_avg = 2.91 fifteen_min_avg = 2.9 cpu_share = 2.25 node_type = config-database-node >> >> 03/31/2025 02:57:22.889 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:24.301 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:57:29.307 7f3b7bf2cac8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:57:29.307 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:57:29.307 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:29.307 7f3b7bf2cac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:57:29.307 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:29.307 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:57:29.308 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:57:29.308 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:29.308 7f3b7bf2cac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:57:29.308 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:29.308 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:29.308 7f3b7bf2cac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:57:29.308 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:33.310 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:57:38.313 7f3b7bf2cac8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:57:38.313 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:57:38.313 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:38.313 7f3b7bf2cac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:57:38.313 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:38.313 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:57:38.314 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:57:38.314 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:38.314 7f3b7bf2cac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:57:38.314 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:38.314 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:38.314 7f3b7bf2cac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1743387253723525 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/31/2025 02:57:38.314 7f3b7bf2cac8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:38.401 7f3b7bf2cac8 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map