02/13/2025 02:16:26.030 7f36e8f12308 [contrail-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 02/13/2025 02:16:26.030 7f36e8f12308 [contrail-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 02/13/2025 02:16:26.030 7f36e8f12308 [contrail-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 02/13/2025 02:16:26.044 7f36e5e7a978 [contrail-database-nodemgr] [ERROR]: Cannot write http_port 8103 to /tmp/contrail-database-nodemgr.0.http_port 02/13/2025 02:16:26.045 7f36e5e7a978 [contrail-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8103 02/13/2025 02:16:26.046 7f36e5e7f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 02/13/2025 02:16:26.047 7f36e5e7f2e8 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:16:26.048 7f36e5e7f2e8 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 02/13/2025 02:16:26.048 7f36e5e7f2e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 02/13/2025 02:16:26.048 7f36e5e7f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:26.049 7f36e5e7f2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 02/13/2025 02:16:26.049 7f36e5e7f2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:26.049 7f36e5e7f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:26.049 7f36e5e7f2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 02/13/2025 02:16:26.049 7f36e5e7f2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:26.217 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 110048700806286897962005952317187902027312504129996358863875257412805305224035, 'start': '1739412983000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-query-engine', 'group': 'contrail-query-engine', 'pid': 24754729347326258031505006104904516998836287077670814015440407264416965612859, 'start': '1739412983000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-database-nodemgr', 'group': 'contrail-database-nodemgr', 'pid': 14892272233064584495439754769783637781092610654387845038522079997002076996219, 'start': '1739412983000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 02/13/2025 02:16:26.369 7f36e5e936d8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:16:26.464 7f36ea3db390 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:26.465 7f36ea3db390 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-02-13 00:54:17.060692", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7769-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 8 num_core_per_socket = 8 num_thread_per_core = 1 >> running_package_version = nightly-nightly.el7 installed_package_version = nightly-nightly.el7 >> 02/13/2025 02:16:26.465 7f36ea3db390 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:26.466 7f36e8f12308 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-473-1', _node_type='Database', _instance_id='0', _timestamp=1739412986466388, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 02/13/2025 02:16:26.467 7f36e8f12308 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-473-1', _node_type='Database', _instance_id='0', _timestamp=1739412986467568, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1739412983000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1739412983000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1739412983000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-02-13 00:54:17.060692", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7769-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 02/13/2025 02:16:26.470 7f36e5e92cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:26.470 7f36e5e92cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 02/13/2025 02:16:26.470 7f36e5e92cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:26.470 7f36e5e92cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:26.471 7f36e5e92cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1739412983000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1739412983000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1739412983000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-02-13 00:54:17.060692", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7769-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 02/13/2025 02:16:26.471 7f36e5e92cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:26.857 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:16:27.233 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:DatabaseUsage] map 02/13/2025 02:16:27.235 7f36e8872588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:27.235 7f36e8872588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 database_usage = [ << disk_space_used_1k = 50436892 disk_space_available_1k = 30653592 analytics_db_size_1k = 4 >>, ] >> 02/13/2025 02:16:27.235 7f36e8872588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:28.024 7f36e8f12308 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL: Result: 1 02/13/2025 02:16:28.024 7f36e8f12308 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-473-1', _node_type='Database', _instance_id='0', _timestamp=1739412988024675, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 02/13/2025 02:16:28.026 7f36e541d6d8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:28.026 7f36e541d6d8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 02/13/2025 02:16:28.026 7f36e541d6d8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:29.583 7f36e8f12308 [contrail-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 2 02/13/2025 02:16:30.047 7f36e67edc18 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:16:30.048 7f36e67edc18 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:16:30.049 7f36e67edc18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:16:30.049 7f36e67edc18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:16:30.049 7f36e67edc18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:16:30.050 7f36e67edc18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:16:30.050 7f36e67edc18 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:16:30.051 7f36e67edc18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:16:30.052 7f36e67edc18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:30.052 7f36e67edc18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:16:30.052 7f36e67edc18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:30.053 7f36e67edc18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:30.053 7f36e67edc18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 02/13/2025 02:16:30.053 7f36e67edc18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:32.342 7f36e8f12308 [contrail-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7200 repair -pr ContrailAnalyticsCql'. Error is: Result: 1 02/13/2025 02:16:34.050 7f36e541d438 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:16:34.051 7f36e541d438 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:16:34.052 7f36e541d438 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:16:34.052 7f36e541d438 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:16:34.052 7f36e541d438 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:16:34.052 7f36e541d438 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:16:34.052 7f36e541d438 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:16:34.053 7f36e541d438 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:16:34.053 7f36e541d438 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:34.053 7f36e541d438 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:16:34.053 7f36e541d438 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:34.053 7f36e541d438 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:34.054 7f36e541d438 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 02/13/2025 02:16:34.054 7f36e541d438 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:37.410 7f36e8f12308 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-473-1', _node_type='Database', _instance_id='0', _timestamp=1739412997410523, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1739412983000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1739412983000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1739412983000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-02-13 00:54:17.060692", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7769-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 02/13/2025 02:16:37.411 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50439628, partition_space_available_1k=30650856, percentage_partition_space_used=62, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1252884, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17332, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41688, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=13315168, free=12502052, buffers=852400, cached=5995492, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=8.17, five_min_avg=4.85, fifteen_min_avg=3.09, cpu_share=0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:16:37.411 7f36e67ed0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:37.413 7f36e67ed0f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1739412983000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1739412983000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1739412983000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-02-13 00:54:17.060692", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7769-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 02/13/2025 02:16:37.413 7f36e67ed0f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:37.413 7f36e67ed0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:37.414 7f36e67ed0f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 50439628 partition_space_available_1k = 30650856 percentage_partition_space_used = 62 >> } all_core_file_list = [ ] process_mem_cpu_usage = { cassandra : << mem_virt = 1252884 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 17332 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 41688 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 13315168 free = 12502052 buffers = 852400 cached = 5995492 node_type = database-node >> system_cpu_usage = << one_min_avg = 8.17 five_min_avg = 4.85 fifteen_min_avg = 3.09 cpu_share = 0 node_type = database-node >> >> 02/13/2025 02:16:37.415 7f36e67ed0f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:16:38.053 7f36e8a5bac8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:16:38.062 7f36e8a5bac8 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 02/13/2025 02:16:38.063 7f36e8a5bac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:16:38.063 7f36e8a5bac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:16:38.063 7f36e8a5bac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:16:38.064 7f36e8a5bac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:16:38.064 7f36e8a5bac8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 02/13/2025 02:16:38.064 7f36e8a5bac8 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 02/13/2025 02:16:38.065 7f36e8a5bac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 02/13/2025 02:16:38.066 7f36e8a5bac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 02/13/2025 02:16:38.066 7f36e8a5bac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 02/13/2025 02:16:38.071 7f36e5429b70 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001011 02/13/2025 02:16:38.072 7f36e5429b70 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 02/13/2025 02:16:38.072 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 02/13/2025 02:16:38.073 7f36e5e92f60 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 02/13/2025 02:16:38.073 7f36e5e92f60 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 02/13/2025 02:16:38.073 7f36e5e92f60 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:16:38.073 7f36e5e92f60 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 02/13/2025 02:16:38.074 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 02/13/2025 02:16:38.074 7f36e8a5bac8 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 02/13/2025 02:16:38.076 7f36e8a5bac8 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 02/13/2025 02:16:51.628 7f36e8c9beb8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:17:11.950 7f36e87fa2e8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:17:26.832 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:17:27.919 7f36e8a5b828 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:17:28.076 7f36e8f12308 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-473-1', _node_type='Database', _instance_id='0', _timestamp=1739413048076642, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.17:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 02/13/2025 02:17:30.549 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-473-1', _node_type='Database', _instance_id='0', _timestamp=1739413050549404, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 02/13/2025 02:17:30.549 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 02/13/2025 02:17:35.591 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50981044, partition_space_available_1k=30109440, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1296916, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19124, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41740, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14521572, free=11082684, buffers=875284, cached=6185572, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=8.01, five_min_avg=5.55, fifteen_min_avg=3.45, cpu_share=7.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:18:07.545 7f36e5e76780 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:18:17.543 7f36e5429630 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:18:26.798 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:18:34.654 7f36e542e2e8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:18:35.533 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51001932, partition_space_available_1k=30088552, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1301728, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=20884, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42572, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14705876, free=10899328, buffers=881252, cached=6178656, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.49, five_min_avg=5.28, fifteen_min_avg=3.49, cpu_share=3.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:18:45.547 7f36e46d6198 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:19:10.598 7f36e8a51978 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:19:20.831 7f36e541deb8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:19:26.804 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:19:35.309 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51011896, partition_space_available_1k=30078588, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1297200, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=21364, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42568, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14698724, free=10812176, buffers=882392, cached=6271820, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.64, five_min_avg=5.05, fifteen_min_avg=3.52, cpu_share=3.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:19:55.076 7f36e5e93b70 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:20:26.853 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:20:35.298 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51006220, partition_space_available_1k=30084264, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1301612, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=21612, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42572, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15029312, free=9875168, buffers=905120, cached=6855512, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.32, five_min_avg=4.8, fifteen_min_avg=3.52, cpu_share=3.12, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:21:26.840 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:21:35.311 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51014764, partition_space_available_1k=30075720, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1328744, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=21812, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42584, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14688720, free=9387868, buffers=910332, cached=7678192, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.56, five_min_avg=5.0, fifteen_min_avg=3.69, cpu_share=4.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:22:05.928 7f36e8a3d2e8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:22:26.837 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:22:35.288 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51020332, partition_space_available_1k=30070152, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1298888, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22032, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42592, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14689272, free=9381864, buffers=910340, cached=7683636, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.86, five_min_avg=4.45, fifteen_min_avg=3.58, cpu_share=2.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:23:06.215 7f36ea391c18 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:23:17.322 7f36e8a080f0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:23:26.853 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:23:35.401 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51024100, partition_space_available_1k=30066384, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1297856, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22172, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42592, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14690684, free=9374644, buffers=910340, cached=7689444, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.35, five_min_avg=4.03, fifteen_min_avg=3.49, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:24:10.553 7f36e5e7c438 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:24:26.907 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:24:35.421 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51027968, partition_space_available_1k=30062516, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1319288, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=21540, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42596, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14706200, free=9353688, buffers=910416, cached=7694808, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.9, five_min_avg=3.59, fifteen_min_avg=3.37, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:25:26.859 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:25:35.347 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51031776, partition_space_available_1k=30058708, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1299552, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=21740, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42600, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14708436, free=9346448, buffers=910416, cached=7699812, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.29, five_min_avg=3.37, fifteen_min_avg=3.3, cpu_share=2.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:26:26.873 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:26:35.272 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51143300, partition_space_available_1k=29947184, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1302320, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=21912, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42604, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14785128, free=9328024, buffers=910424, cached=7641536, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.86, five_min_avg=3.79, fifteen_min_avg=3.46, cpu_share=4.12, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:27:03.652 7f36e98a3ac8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:27:26.896 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:27:35.269 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51046460, partition_space_available_1k=30044024, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1306908, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22016, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42616, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14797184, free=9309552, buffers=910428, cached=7647948, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.51, five_min_avg=3.4, fifteen_min_avg=3.35, cpu_share=2.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:28:18.105 7f36e6809780 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:28:26.890 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:28:29.259 7f36e5439390 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:28:35.587 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51051236, partition_space_available_1k=30039248, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1310472, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22276, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42612, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14797188, free=9304976, buffers=910428, cached=7652520, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.15, five_min_avg=3.13, fifteen_min_avg=3.25, cpu_share=2.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:29:13.476 7f36ee0d8f60 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:29:26.900 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:29:35.273 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51054988, partition_space_available_1k=30035496, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1301180, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22492, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42612, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14799948, free=9297520, buffers=910428, cached=7657216, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.95, five_min_avg=2.88, fifteen_min_avg=3.16, cpu_share=2.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:30:26.914 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:30:35.411 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51064228, partition_space_available_1k=30026256, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1306184, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22052, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42620, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14869956, free=9208732, buffers=910432, cached=7675992, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.9, five_min_avg=4.07, fifteen_min_avg=3.54, cpu_share=4.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:31:26.949 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:31:35.509 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51068040, partition_space_available_1k=30022444, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1311160, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22228, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42616, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14828112, free=9245680, buffers=910448, cached=7680872, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.12, five_min_avg=3.76, fifteen_min_avg=3.46, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:31:58.900 7f36ef179a20 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:32:26.919 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:32:35.431 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51072416, partition_space_available_1k=30018068, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1302968, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22388, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42628, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14830984, free=9237748, buffers=910460, cached=7685920, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.86, five_min_avg=3.44, fifteen_min_avg=3.37, cpu_share=2.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:32:57.815 7f36e53d1048 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:33:09.723 7f36efa42d68 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:33:26.928 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:33:35.344 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51076400, partition_space_available_1k=30014084, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1302956, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22496, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42624, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14826056, free=9237384, buffers=910484, cached=7691188, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.96, five_min_avg=3.33, fifteen_min_avg=3.34, cpu_share=3.38, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:33:59.708 7f36e8a51c18 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:34:26.915 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:34:35.513 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51080192, partition_space_available_1k=30010292, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1303360, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22624, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42632, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14827096, free=9231300, buffers=910492, cached=7696224, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.71, five_min_avg=3.16, fifteen_min_avg=3.27, cpu_share=3.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:35:26.963 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:35:35.452 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51089780, partition_space_available_1k=30000704, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1304816, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22772, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42632, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14871956, free=9148072, buffers=911364, cached=7733720, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.49, five_min_avg=3.94, fifteen_min_avg=3.54, cpu_share=4.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:36:16.860 7f36e5e93c18 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:36:26.930 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:36:35.435 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51094404, partition_space_available_1k=29996080, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1308972, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22664, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42632, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14881432, free=9132812, buffers=911364, cached=7739504, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.48, five_min_avg=3.61, fifteen_min_avg=3.45, cpu_share=2.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:37:09.782 7f36ef179a20 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:37:21.176 7f36e5e92cc0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:37:26.930 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:37:35.648 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51099064, partition_space_available_1k=29991420, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1307320, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22884, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42644, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14880964, free=9127300, buffers=911376, cached=7745472, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.72, five_min_avg=3.34, fifteen_min_avg=3.36, cpu_share=3.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:38:26.948 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:38:35.391 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51102676, partition_space_available_1k=29987808, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1309624, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23020, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42640, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14882788, free=9120592, buffers=911376, cached=7750356, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.23, five_min_avg=3.06, fifteen_min_avg=3.26, cpu_share=2.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:39:08.846 7f36e5eba240 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:39:28.449 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:39:41.150 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51111232, partition_space_available_1k=29979252, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1307268, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23172, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42644, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14967616, free=9013516, buffers=911384, cached=7772596, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.26, five_min_avg=3.84, fifteen_min_avg=3.5, cpu_share=4.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:40:27.041 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:40:35.414 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51116868, partition_space_available_1k=29973616, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1307004, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23300, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42652, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14882888, free=9093980, buffers=911384, cached=7776860, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.06, five_min_avg=3.58, fifteen_min_avg=3.43, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:41:27.036 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:41:29.904 7f36e54286d8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:41:35.583 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51120072, partition_space_available_1k=29970412, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1307032, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23464, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42656, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14934184, free=9038668, buffers=911384, cached=7780876, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.15, five_min_avg=3.43, fifteen_min_avg=3.39, cpu_share=2.38, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:41:58.074 7f36e5429b70 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 02/13/2025 02:41:58.075 7f36ea391c18 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[Established] 02/13/2025 02:42:03.078 7f36ea391c18 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:42:03.078 7f36ea391c18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[Established] -> State[Connect] 02/13/2025 02:42:03.078 7f36ea391c18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:42:03.079 7f36ea391c18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Established to Connect on EvTcpClose >>, ] description = Collector:Collector[Established to Connect on EvTcpClose] connection down >>, ] >> 02/13/2025 02:42:03.079 7f36ea391c18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:42:03.079 7f36ea391c18 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:42:03.079 7f36ea391c18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:42:03.079 7f36ea391c18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:03.080 7f36ea391c18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:42:03.080 7f36ea391c18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:03.080 7f36ea391c18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:03.080 7f36ea391c18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:42:03.080 7f36ea391c18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:07.082 7f36ea391c18 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:42:12.088 7f36ea391c18 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:42:12.088 7f36ea391c18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:42:12.088 7f36ea391c18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:42:12.088 7f36ea391c18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:42:12.089 7f36ea391c18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:42:12.089 7f36ea391c18 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:42:12.089 7f36ea391c18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:42:12.089 7f36ea391c18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:12.089 7f36ea391c18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:42:12.090 7f36ea391c18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:12.090 7f36ea391c18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:12.090 7f36ea391c18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:42:12.090 7f36ea391c18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:16.090 7f36e5428f60 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:42:21.092 7f36e5428f60 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:42:21.092 7f36e5428f60 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:42:21.092 7f36e5428f60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:42:21.093 7f36e5428f60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:42:21.093 7f36e5428f60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:42:21.093 7f36e5428f60 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:42:21.093 7f36e5428f60 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:42:21.093 7f36e5428f60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:21.094 7f36e5428f60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:42:21.094 7f36e5428f60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:21.094 7f36e5428f60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:21.094 7f36e5428f60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:42:21.094 7f36e5428f60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:25.096 7f36e5428f60 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:42:27.039 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:42:30.098 7f36e5428f60 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:42:30.098 7f36e5428f60 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:42:30.098 7f36e5428f60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:42:30.099 7f36e5428f60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:42:30.099 7f36e5428f60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:42:30.099 7f36e5428f60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:42:30.099 7f36e5428f60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 database_usage = [ << disk_space_used_1k = 51119528 disk_space_available_1k = 29970956 analytics_db_size_1k = 996 >>, ] >> 02/13/2025 02:42:30.099 7f36e5428f60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:42:30.099 7f36e5428f60 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:42:30.100 7f36e5428f60 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:42:30.100 7f36e5428f60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:30.100 7f36e5428f60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:42:30.100 7f36e5428f60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:30.100 7f36e5428f60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:30.100 7f36e5428f60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:42:30.101 7f36e5428f60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:34.099 7f36e65a6048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:42:35.418 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51119796, partition_space_available_1k=29970688, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1311160, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23044, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42704, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14905128, free=9067192, buffers=911384, cached=7781408, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.41, five_min_avg=3.13, fifteen_min_avg=3.28, cpu_share=2.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:42:39.104 7f36e65a6048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:42:39.104 7f36e65a6048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:42:39.104 7f36e65a6048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:42:39.105 7f36e65a6048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:42:39.105 7f36e65a6048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:42:39.105 7f36e65a6048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:42:39.105 7f36e65a6048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51119796 partition_space_available_1k = 29970688 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1311160 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 23044 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 42704 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 14905128 free = 9067192 buffers = 911384 cached = 7781408 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.41 five_min_avg = 3.13 fifteen_min_avg = 3.28 cpu_share = 2.5 node_type = database-node >> >> 02/13/2025 02:42:39.105 7f36e65a6048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:42:39.105 7f36e65a6048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:42:39.106 7f36e65a6048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:42:39.106 7f36e65a6048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:39.106 7f36e65a6048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:42:39.106 7f36e65a6048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:39.106 7f36e65a6048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:39.107 7f36e65a6048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:42:39.107 7f36e65a6048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:43.108 7f36e5e7c0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:42:44.623 7f36e6af8588 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:42:48.112 7f36e5e7c0f0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:42:48.112 7f36e5e7c0f0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:42:48.113 7f36e5e7c0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:42:48.113 7f36e5e7c0f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:42:48.113 7f36e5e7c0f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:42:48.113 7f36e5e7c0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:42:48.114 7f36e5e7c0f0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:42:48.114 7f36e5e7c0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:48.115 7f36e5e7c0f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:42:48.115 7f36e5e7c0f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:48.115 7f36e5e7c0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:48.115 7f36e5e7c0f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:42:48.115 7f36e5e7c0f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:52.116 7f36e8a34048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:42:55.564 7f36e46d6198 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:42:57.119 7f36e8a34048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:42:57.120 7f36e8a34048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:42:57.120 7f36e8a34048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:42:57.120 7f36e8a34048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:42:57.120 7f36e8a34048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:42:57.120 7f36e8a34048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:42:57.121 7f36e8a34048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:42:57.121 7f36e8a34048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:57.121 7f36e8a34048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:42:57.121 7f36e8a34048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:57.121 7f36e8a34048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:42:57.121 7f36e8a34048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:42:57.121 7f36e8a34048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:01.120 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:43:06.126 7f36e6af8630 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:43:06.126 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:43:06.126 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:43:06.127 7f36e6af8630 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:43:06.127 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:43:06.127 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:43:06.127 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:43:06.128 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:06.128 7f36e6af8630 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:43:06.128 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:06.128 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:06.128 7f36e6af8630 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:43:06.128 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:10.130 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:43:15.133 7f36e6af8630 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:43:15.133 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:43:15.133 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:43:15.133 7f36e6af8630 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:43:15.134 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:43:15.134 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:43:15.134 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:43:15.134 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:15.134 7f36e6af8630 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:43:15.135 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:15.135 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:15.135 7f36e6af8630 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:43:15.135 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:19.137 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:43:24.142 7f36e6af8630 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:43:24.142 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:43:24.142 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:43:24.142 7f36e6af8630 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:43:24.143 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:43:24.143 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:43:24.143 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:43:24.143 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:24.143 7f36e6af8630 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:43:24.143 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:24.144 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:24.144 7f36e6af8630 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:43:24.144 7f36e6af8630 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:27.019 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:43:27.380 7f36e6af8198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:27.380 7f36e6af8198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 database_usage = [ << disk_space_used_1k = 51123400 disk_space_available_1k = 29967084 analytics_db_size_1k = 996 >>, ] >> 02/13/2025 02:43:27.380 7f36e6af8198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:28.144 7f36e8a51c18 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:43:33.150 7f36e8a51c18 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:43:33.150 7f36e8a51c18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:43:33.151 7f36e8a51c18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:43:33.151 7f36e8a51c18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:43:33.151 7f36e8a51c18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:43:33.151 7f36e8a51c18 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:43:33.152 7f36e8a51c18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:43:33.152 7f36e8a51c18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:33.152 7f36e8a51c18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:43:33.152 7f36e8a51c18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:33.152 7f36e8a51c18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:33.153 7f36e8a51c18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:43:33.153 7f36e8a51c18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:35.431 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51123700, partition_space_available_1k=29966784, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1312364, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23272, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42720, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14946088, free=9021296, buffers=911384, cached=7786344, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.36, five_min_avg=3.04, fifteen_min_avg=3.24, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:43:35.431 7f36e5429828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:35.432 7f36e5429828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51123700 partition_space_available_1k = 29966784 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1312364 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 23272 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 42720 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 14946088 free = 9021296 buffers = 911384 cached = 7786344 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.36 five_min_avg = 3.04 fifteen_min_avg = 3.24 cpu_share = 2.62 node_type = database-node >> >> 02/13/2025 02:43:35.432 7f36e5429828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:37.151 7f36e5429828 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:43:42.157 7f36e5429828 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:43:42.157 7f36e5429828 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:43:42.157 7f36e5429828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:43:42.158 7f36e5429828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:43:42.158 7f36e5429828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:43:42.158 7f36e5429828 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:43:42.158 7f36e5429828 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:43:42.158 7f36e5429828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:42.158 7f36e5429828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:43:42.159 7f36e5429828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:42.159 7f36e5429828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:42.159 7f36e5429828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:43:42.159 7f36e5429828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:46.158 7f36e67f0eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:43:51.164 7f36e67f0eb8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:43:51.164 7f36e67f0eb8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:43:51.164 7f36e67f0eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:43:51.164 7f36e67f0eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:43:51.165 7f36e67f0eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:43:51.165 7f36e67f0eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:43:51.165 7f36e67f0eb8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:43:51.165 7f36e67f0eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:51.165 7f36e67f0eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:43:51.166 7f36e67f0eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:51.166 7f36e67f0eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:51.166 7f36e67f0eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:43:51.166 7f36e67f0eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:43:55.168 7f36e67f0eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:43:55.954 7f36e5428b70 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:44:00.170 7f36e67f0eb8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:44:00.170 7f36e67f0eb8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:44:00.170 7f36e67f0eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:44:00.170 7f36e67f0eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:44:00.170 7f36e67f0eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:44:00.171 7f36e67f0eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:44:00.171 7f36e67f0eb8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:44:00.171 7f36e67f0eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:00.171 7f36e67f0eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:44:00.172 7f36e67f0eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:00.172 7f36e67f0eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:00.172 7f36e67f0eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:44:00.172 7f36e67f0eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:04.175 7f36e67ed8d0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:44:09.181 7f36e67ed8d0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:44:09.181 7f36e67ed8d0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:44:09.181 7f36e67ed8d0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:44:09.181 7f36e67ed8d0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:44:09.182 7f36e67ed8d0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:44:09.182 7f36e67ed8d0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:44:09.182 7f36e67ed8d0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:44:09.183 7f36e67ed8d0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:09.183 7f36e67ed8d0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:44:09.183 7f36e67ed8d0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:09.183 7f36e67ed8d0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:09.183 7f36e67ed8d0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:44:09.184 7f36e67ed8d0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:13.185 7f36e67ed8d0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:44:18.190 7f36e67ed8d0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:44:18.190 7f36e67ed8d0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:44:18.191 7f36e67ed8d0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:44:18.191 7f36e67ed8d0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:44:18.191 7f36e67ed8d0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:44:18.191 7f36e67ed8d0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:44:18.191 7f36e67ed8d0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:44:18.192 7f36e67ed8d0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:18.192 7f36e67ed8d0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:44:18.192 7f36e67ed8d0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:18.192 7f36e67ed8d0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:18.192 7f36e67ed8d0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:44:18.192 7f36e67ed8d0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:22.192 7f36e542c828 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:44:27.038 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:44:27.196 7f36e542c828 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:44:27.196 7f36e542c828 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:44:27.196 7f36e542c828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:44:27.196 7f36e542c828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:44:27.197 7f36e542c828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:44:27.197 7f36e542c828 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:44:27.197 7f36e542c828 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:44:27.198 7f36e542c828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:27.198 7f36e542c828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:44:27.198 7f36e542c828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:27.198 7f36e542c828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:27.198 7f36e542c828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:44:27.198 7f36e542c828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:27.430 7f36e5eba438 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:27.430 7f36e5eba438 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 database_usage = [ << disk_space_used_1k = 51133120 disk_space_available_1k = 29957364 analytics_db_size_1k = 996 >>, ] >> 02/13/2025 02:44:27.430 7f36e5eba438 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:31.197 7f36efd23ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:44:35.358 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51133368, partition_space_available_1k=29957116, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1307000, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23468, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42792, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14987268, free=8961192, buffers=911388, cached=7805264, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.06, five_min_avg=3.7, fifteen_min_avg=3.46, cpu_share=4.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:44:36.201 7f36efd23ac8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:44:36.201 7f36efd23ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:44:36.201 7f36efd23ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:44:36.201 7f36efd23ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:44:36.202 7f36efd23ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:44:36.202 7f36efd23ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:44:36.202 7f36efd23ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51133368 partition_space_available_1k = 29957116 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1307000 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 23468 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 42792 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 14987268 free = 8961192 buffers = 911388 cached = 7805264 node_type = database-node >> system_cpu_usage = << one_min_avg = 5.06 five_min_avg = 3.7 fifteen_min_avg = 3.46 cpu_share = 4.5 node_type = database-node >> >> 02/13/2025 02:44:36.202 7f36efd23ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:44:36.202 7f36efd23ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:44:36.203 7f36efd23ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:44:36.203 7f36efd23ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:36.203 7f36efd23ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:44:36.203 7f36efd23ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:36.203 7f36efd23ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:36.204 7f36efd23ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:44:36.204 7f36efd23ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:40.204 7f36efd23ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:44:45.210 7f36efd23ac8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:44:45.211 7f36efd23ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:44:45.211 7f36efd23ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:44:45.211 7f36efd23ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:44:45.211 7f36efd23ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:44:45.211 7f36efd23ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:44:45.212 7f36efd23ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:44:45.212 7f36efd23ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:45.212 7f36efd23ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:44:45.212 7f36efd23ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:45.212 7f36efd23ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:45.212 7f36efd23ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:44:45.213 7f36efd23ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:49.214 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:44:54.220 7f36e88e8828 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:44:54.221 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:44:54.221 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:44:54.221 7f36e88e8828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:44:54.221 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:44:54.221 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:44:54.222 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:44:54.222 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:54.222 7f36e88e8828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:44:54.222 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:54.222 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:54.222 7f36e88e8828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:44:54.222 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:44:58.224 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:45:03.229 7f36e88e8828 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:45:03.229 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:45:03.229 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:45:03.229 7f36e88e8828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:45:03.229 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:45:03.229 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:45:03.230 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:45:03.230 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:03.230 7f36e88e8828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:45:03.230 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:03.230 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:03.231 7f36e88e8828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:45:03.231 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:07.232 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:45:12.238 7f36e88e8828 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:45:12.238 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:45:12.238 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:45:12.238 7f36e88e8828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:45:12.239 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:45:12.239 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:45:12.239 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:45:12.239 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:12.239 7f36e88e8828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:45:12.240 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:12.240 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:12.240 7f36e88e8828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:45:12.240 7f36e88e8828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:16.239 7f36e46d6198 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:45:21.244 7f36e46d6198 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:45:21.244 7f36e46d6198 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:45:21.245 7f36e46d6198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:45:21.245 7f36e46d6198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:45:21.245 7f36e46d6198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:45:21.245 7f36e46d6198 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:45:21.246 7f36e46d6198 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:45:21.246 7f36e46d6198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:21.246 7f36e46d6198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:45:21.247 7f36e46d6198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:21.247 7f36e46d6198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:21.247 7f36e46d6198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:45:21.247 7f36e46d6198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:25.247 7f36e46d6198 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:45:27.056 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:45:30.248 7f36e46d6198 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:45:30.249 7f36e46d6198 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:45:30.249 7f36e46d6198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:45:30.249 7f36e46d6198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:45:30.249 7f36e46d6198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:45:30.249 7f36e46d6198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:45:30.250 7f36e46d6198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 database_usage = [ << disk_space_used_1k = 51137964 disk_space_available_1k = 29952520 analytics_db_size_1k = 996 >>, ] >> 02/13/2025 02:45:30.250 7f36e46d6198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:45:30.250 7f36e46d6198 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:45:30.250 7f36e46d6198 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:45:30.251 7f36e46d6198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:30.251 7f36e46d6198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:45:30.251 7f36e46d6198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:30.251 7f36e46d6198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:30.251 7f36e46d6198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:45:30.251 7f36e46d6198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:34.249 7f36e542e630 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:45:35.561 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51142676, partition_space_available_1k=29947808, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1305736, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23604, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42832, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15043704, free=8895144, buffers=911388, cached=7814876, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.0, five_min_avg=3.39, fifteen_min_avg=3.36, cpu_share=2.12, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:45:36.002 7f36e6594b70 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:45:39.255 7f36e542e630 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:45:39.255 7f36e542e630 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:45:39.255 7f36e542e630 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:45:39.256 7f36e542e630 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:45:39.256 7f36e542e630 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:45:39.256 7f36e542e630 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:45:39.256 7f36e542e630 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51142676 partition_space_available_1k = 29947808 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1305736 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 23604 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 42832 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15043704 free = 8895144 buffers = 911388 cached = 7814876 node_type = database-node >> system_cpu_usage = << one_min_avg = 3.0 five_min_avg = 3.39 fifteen_min_avg = 3.36 cpu_share = 2.12 node_type = database-node >> >> 02/13/2025 02:45:39.256 7f36e542e630 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:45:39.256 7f36e542e630 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:45:39.257 7f36e542e630 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:45:39.257 7f36e542e630 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:39.257 7f36e542e630 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:45:39.257 7f36e542e630 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:39.257 7f36e542e630 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:39.257 7f36e542e630 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:45:39.257 7f36e542e630 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:43.260 7f36e5e7c588 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:45:48.263 7f36e5e7c588 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:45:48.263 7f36e5e7c588 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:45:48.263 7f36e5e7c588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:45:48.264 7f36e5e7c588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:45:48.264 7f36e5e7c588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:45:48.264 7f36e5e7c588 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:45:48.264 7f36e5e7c588 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:45:48.265 7f36e5e7c588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:48.265 7f36e5e7c588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:45:48.265 7f36e5e7c588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:48.265 7f36e5e7c588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:48.265 7f36e5e7c588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:45:48.265 7f36e5e7c588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:52.268 7f36e5e7c588 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:45:57.274 7f36e5e7c588 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:45:57.274 7f36e5e7c588 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:45:57.274 7f36e5e7c588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:45:57.275 7f36e5e7c588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:45:57.275 7f36e5e7c588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:45:57.275 7f36e5e7c588 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:45:57.275 7f36e5e7c588 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:45:57.275 7f36e5e7c588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:57.276 7f36e5e7c588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:45:57.276 7f36e5e7c588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:57.276 7f36e5e7c588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:45:57.276 7f36e5e7c588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:45:57.276 7f36e5e7c588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:01.274 7f36efd230f0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:46:06.279 7f36efd230f0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:46:06.279 7f36efd230f0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:46:06.280 7f36efd230f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:46:06.280 7f36efd230f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:46:06.280 7f36efd230f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:46:06.280 7f36efd230f0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:46:06.281 7f36efd230f0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:46:06.281 7f36efd230f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:06.281 7f36efd230f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:46:06.281 7f36efd230f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:06.281 7f36efd230f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:06.281 7f36efd230f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:46:06.281 7f36efd230f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:10.282 7f36efd230f0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:46:15.288 7f36efd230f0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:46:15.288 7f36efd230f0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:46:15.288 7f36efd230f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:46:15.289 7f36efd230f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:46:15.289 7f36efd230f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:46:15.289 7f36efd230f0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:46:15.289 7f36efd230f0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:46:15.289 7f36efd230f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:15.290 7f36efd230f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:46:15.290 7f36efd230f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:15.290 7f36efd230f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:15.290 7f36efd230f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:46:15.290 7f36efd230f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:19.291 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:46:24.295 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:46:24.295 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:46:24.295 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:46:24.295 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:46:24.295 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:46:24.296 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:46:24.296 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:46:24.296 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:24.296 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:46:24.297 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:24.297 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:24.297 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:46:24.297 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:27.051 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:46:27.443 7f36e5411780 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:27.443 7f36e5411780 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 database_usage = [ << disk_space_used_1k = 51136780 disk_space_available_1k = 29953704 analytics_db_size_1k = 996 >>, ] >> 02/13/2025 02:46:27.444 7f36e5411780 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:28.295 7f36e6af8198 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:46:33.299 7f36e6af8198 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:46:33.299 7f36e6af8198 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:46:33.299 7f36e6af8198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:46:33.300 7f36e6af8198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:46:33.300 7f36e6af8198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:46:33.300 7f36e6af8198 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:46:33.301 7f36e6af8198 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:46:33.301 7f36e6af8198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:33.301 7f36e6af8198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:46:33.301 7f36e6af8198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:33.301 7f36e6af8198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:33.301 7f36e6af8198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:46:33.302 7f36e6af8198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:35.500 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51137044, partition_space_available_1k=29953440, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1306148, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23744, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42844, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14961804, free=8981668, buffers=911388, cached=7810252, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.41, five_min_avg=3.15, fifteen_min_avg=3.28, cpu_share=1.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:46:35.500 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:35.501 7f36e8f41ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51137044 partition_space_available_1k = 29953440 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1306148 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 23744 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 42844 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 14961804 free = 8981668 buffers = 911388 cached = 7810252 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.41 five_min_avg = 3.15 fifteen_min_avg = 3.28 cpu_share = 1.88 node_type = database-node >> >> 02/13/2025 02:46:35.501 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:37.301 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:46:42.306 7f36e8f41ac8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:46:42.306 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:46:42.306 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:46:42.306 7f36e8f41ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:46:42.306 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:46:42.307 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:46:42.307 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:46:42.307 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:42.307 7f36e8f41ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:46:42.307 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:42.308 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:42.308 7f36e8f41ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:46:42.308 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:46.308 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:46:49.248 7f36eedda048 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:46:51.312 7f36e8f41ac8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:46:51.313 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:46:51.313 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:46:51.313 7f36e8f41ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:46:51.313 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:46:51.313 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:46:51.314 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:46:51.314 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:51.314 7f36e8f41ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:46:51.314 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:51.314 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:51.314 7f36e8f41ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:46:51.314 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:46:55.317 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:47:00.072 7f36e881c978 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:47:00.319 7f36e8f41ac8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:47:00.319 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:47:00.319 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:47:00.319 7f36e8f41ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:47:00.320 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:47:00.320 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:47:00.320 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:47:00.321 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:00.321 7f36e8f41ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:47:00.321 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:00.321 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:00.321 7f36e8f41ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:47:00.321 7f36e8f41ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:04.321 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:47:09.326 7f36e98aec18 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:47:09.326 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:47:09.326 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:47:09.327 7f36e98aec18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:47:09.327 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:47:09.327 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:47:09.327 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:47:09.327 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:09.328 7f36e98aec18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:47:09.328 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:09.328 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:09.328 7f36e98aec18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:47:09.328 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:13.330 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:47:18.334 7f36e98aec18 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:47:18.334 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:47:18.334 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:47:18.334 7f36e98aec18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:47:18.335 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:47:18.335 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:47:18.335 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:47:18.335 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:18.336 7f36e98aec18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:47:18.336 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:18.336 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:18.336 7f36e98aec18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:47:18.336 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:22.338 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:47:27.051 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:47:27.342 7f36e98aec18 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:47:27.342 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:47:27.342 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:47:27.342 7f36e98aec18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:47:27.342 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:47:27.342 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:47:27.343 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:47:27.343 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:27.343 7f36e98aec18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:47:27.343 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:27.343 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:27.343 7f36e98aec18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:47:27.343 7f36e98aec18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:27.443 7f36e8cee240 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:27.443 7f36e8cee240 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 database_usage = [ << disk_space_used_1k = 51141508 disk_space_available_1k = 29948976 analytics_db_size_1k = 996 >>, ] >> 02/13/2025 02:47:27.443 7f36e8cee240 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:31.343 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:47:35.536 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51141760, partition_space_available_1k=29948724, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1307272, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23976, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42924, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14971288, free=8965752, buffers=911396, cached=7816676, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.08, five_min_avg=2.92, fifteen_min_avg=3.19, cpu_share=2.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:47:36.346 7f36e5e92f60 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:47:36.347 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:47:36.347 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:47:36.347 7f36e5e92f60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:47:36.347 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:47:36.347 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:47:36.347 7f36e5e92f60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51141760 partition_space_available_1k = 29948724 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1307272 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 23976 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 42924 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 14971288 free = 8965752 buffers = 911396 cached = 7816676 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.08 five_min_avg = 2.92 fifteen_min_avg = 3.19 cpu_share = 2.5 node_type = database-node >> >> 02/13/2025 02:47:36.348 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:47:36.348 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:47:36.348 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:47:36.348 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:36.348 7f36e5e92f60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:47:36.349 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:36.349 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:36.349 7f36e5e92f60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:47:36.349 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:40.350 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:47:45.356 7f36e5e92f60 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:47:45.357 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:47:45.357 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:47:45.357 7f36e5e92f60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:47:45.357 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:47:45.357 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:47:45.358 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:47:45.358 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:45.358 7f36e5e92f60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:47:45.358 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:45.359 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:45.359 7f36e5e92f60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:47:45.359 7f36e5e92f60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:49.358 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:47:54.362 7f36e6648eb8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:47:54.362 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:47:54.363 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:47:54.363 7f36e6648eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:47:54.363 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:47:54.363 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:47:54.364 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:47:54.364 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:54.364 7f36e6648eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:47:54.364 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:54.364 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:54.364 7f36e6648eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:47:54.364 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:47:58.366 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:48:03.369 7f36e6648eb8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:48:03.369 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:48:03.369 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:48:03.370 7f36e6648eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:48:03.370 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:48:03.370 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:48:03.371 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:48:03.371 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:03.371 7f36e6648eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:48:03.371 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:03.371 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:03.371 7f36e6648eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:48:03.371 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:07.372 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:48:12.376 7f36e6648eb8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:48:12.377 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:48:12.377 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:48:12.377 7f36e6648eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:48:12.377 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:48:12.377 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:48:12.378 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:48:12.378 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:12.378 7f36e6648eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:48:12.378 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:12.379 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:12.379 7f36e6648eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:48:12.379 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:16.381 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:48:21.387 7f36e6648eb8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:48:21.387 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:48:21.387 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:48:21.387 7f36e6648eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:48:21.387 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:48:21.388 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:48:21.388 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:48:21.388 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:21.388 7f36e6648eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:48:21.389 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:21.389 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:21.389 7f36e6648eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:48:21.389 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:25.388 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:48:27.044 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:48:30.390 7f36e6648eb8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:48:30.390 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:48:30.391 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:48:30.391 7f36e6648eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:48:30.391 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:48:30.391 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:48:30.391 7f36e6648eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 database_usage = [ << disk_space_used_1k = 51145208 disk_space_available_1k = 29945276 analytics_db_size_1k = 996 >>, ] >> 02/13/2025 02:48:30.391 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:48:30.391 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:48:30.392 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:48:30.392 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:30.393 7f36e6648eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:48:30.393 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:30.393 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:30.393 7f36e6648eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:48:30.393 7f36e6648eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:34.391 7f36e5449390 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:48:35.399 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51145444, partition_space_available_1k=29945040, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1307156, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23564, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42964, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14965872, free=8966864, buffers=911396, cached=7820980, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.04, five_min_avg=2.73, fifteen_min_avg=3.11, cpu_share=2.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:48:39.395 7f36e5449390 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:48:39.396 7f36e5449390 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:48:39.396 7f36e5449390 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:48:39.396 7f36e5449390 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:48:39.396 7f36e5449390 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:48:39.397 7f36e5449390 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:48:39.397 7f36e5449390 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51145444 partition_space_available_1k = 29945040 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1307156 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 23564 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 42964 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 14965872 free = 8966864 buffers = 911396 cached = 7820980 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.04 five_min_avg = 2.73 fifteen_min_avg = 3.11 cpu_share = 2.0 node_type = database-node >> >> 02/13/2025 02:48:39.397 7f36e5449390 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:48:39.397 7f36e5449390 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:48:39.397 7f36e5449390 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:48:39.398 7f36e5449390 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:39.398 7f36e5449390 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:48:39.398 7f36e5449390 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:39.398 7f36e5449390 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:39.398 7f36e5449390 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:48:39.398 7f36e5449390 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:43.400 7f36e65a6cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:48:48.402 7f36e65a6cc0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:48:48.403 7f36e65a6cc0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:48:48.403 7f36e65a6cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:48:48.403 7f36e65a6cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:48:48.403 7f36e65a6cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:48:48.403 7f36e65a6cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:48:48.404 7f36e65a6cc0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:48:48.404 7f36e65a6cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:48.404 7f36e65a6cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:48:48.405 7f36e65a6cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:48.405 7f36e65a6cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:48.405 7f36e65a6cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:48:48.405 7f36e65a6cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:52.406 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:48:57.412 7f36e54492e8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:48:57.412 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:48:57.413 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:48:57.413 7f36e54492e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:48:57.413 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:48:57.413 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:48:57.414 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:48:57.414 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:57.414 7f36e54492e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:48:57.414 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:57.414 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:48:57.415 7f36e54492e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:48:57.415 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:01.415 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:49:06.419 7f36e54492e8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:49:06.419 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:49:06.419 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:49:06.420 7f36e54492e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:49:06.420 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:49:06.420 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:49:06.420 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:49:06.420 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:06.421 7f36e54492e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:49:06.421 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:06.421 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:06.421 7f36e54492e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:49:06.421 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:10.423 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:49:15.429 7f36e54492e8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:49:15.429 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:49:15.429 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:49:15.429 7f36e54492e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:49:15.429 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:49:15.429 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:49:15.430 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:49:15.430 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:15.430 7f36e54492e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:49:15.430 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:15.430 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:15.430 7f36e54492e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:49:15.431 7f36e54492e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:19.430 7f36e5429cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:49:24.436 7f36e5429cc0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:49:24.437 7f36e5429cc0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:49:24.437 7f36e5429cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:49:24.437 7f36e5429cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:49:24.437 7f36e5429cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:49:24.437 7f36e5429cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:49:24.438 7f36e5429cc0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:49:24.438 7f36e5429cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:24.438 7f36e5429cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:49:24.438 7f36e5429cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:24.438 7f36e5429cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:24.438 7f36e5429cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:49:24.438 7f36e5429cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:27.064 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:49:27.448 7f36e5429198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:27.449 7f36e5429198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 database_usage = [ << disk_space_used_1k = 51148136 disk_space_available_1k = 29942348 analytics_db_size_1k = 996 >>, ] >> 02/13/2025 02:49:27.449 7f36e5429198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:28.437 7f36e65a62e8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:49:33.423 7f36e5e93a20 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:49:33.440 7f36e65a62e8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:49:33.440 7f36e65a62e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:49:33.440 7f36e65a62e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:49:33.440 7f36e65a62e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:49:33.440 7f36e65a62e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:49:33.440 7f36e65a62e8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:49:33.441 7f36e65a62e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:49:33.441 7f36e65a62e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:33.441 7f36e65a62e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:49:33.441 7f36e65a62e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:33.441 7f36e65a62e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:33.442 7f36e65a62e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:49:33.442 7f36e65a62e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:35.486 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51148664, partition_space_available_1k=29941820, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1307288, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23988, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42980, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14988360, free=8944764, buffers=911396, cached=7820592, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.89, five_min_avg=2.54, fifteen_min_avg=3.02, cpu_share=2.12, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:49:35.486 7f36e8872588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:35.487 7f36e8872588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51148664 partition_space_available_1k = 29941820 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1307288 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 23988 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 42980 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 14988360 free = 8944764 buffers = 911396 cached = 7820592 node_type = database-node >> system_cpu_usage = << one_min_avg = 1.89 five_min_avg = 2.54 fifteen_min_avg = 3.02 cpu_share = 2.12 node_type = database-node >> >> 02/13/2025 02:49:35.487 7f36e8872588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:37.442 7f36e8872588 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:49:42.448 7f36e8872588 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:49:42.449 7f36e8872588 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:49:42.449 7f36e8872588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:49:42.449 7f36e8872588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:49:42.449 7f36e8872588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:49:42.449 7f36e8872588 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:49:42.450 7f36e8872588 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:49:42.450 7f36e8872588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:42.450 7f36e8872588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:49:42.450 7f36e8872588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:42.450 7f36e8872588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:42.450 7f36e8872588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:49:42.450 7f36e8872588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:44.139 7f36e8872588 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:49:46.450 7f36e8872588 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:49:51.454 7f36e8872588 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:49:51.454 7f36e8872588 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:49:51.455 7f36e8872588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:49:51.455 7f36e8872588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:49:51.455 7f36e8872588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:49:51.455 7f36e8872588 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:49:51.456 7f36e8872588 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:49:51.456 7f36e8872588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:51.456 7f36e8872588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:49:51.456 7f36e8872588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:51.456 7f36e8872588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:51.456 7f36e8872588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:49:51.457 7f36e8872588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:49:55.458 7f36e8872588 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:50:00.462 7f36e8872588 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:50:00.462 7f36e8872588 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:50:00.462 7f36e8872588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:50:00.463 7f36e8872588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:50:00.463 7f36e8872588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:50:00.463 7f36e8872588 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:50:00.463 7f36e8872588 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:50:00.463 7f36e8872588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:00.463 7f36e8872588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:50:00.464 7f36e8872588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:00.464 7f36e8872588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:00.464 7f36e8872588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:50:00.464 7f36e8872588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:04.463 7f36e5411390 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:50:09.468 7f36e5411390 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:50:09.469 7f36e5411390 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:50:09.469 7f36e5411390 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:50:09.469 7f36e5411390 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:50:09.469 7f36e5411390 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:50:09.469 7f36e5411390 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:50:09.470 7f36e5411390 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:50:09.470 7f36e5411390 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:09.470 7f36e5411390 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:50:09.470 7f36e5411390 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:09.470 7f36e5411390 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:09.470 7f36e5411390 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:50:09.471 7f36e5411390 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:13.472 7f36e5411390 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:50:18.478 7f36e5411390 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:50:18.479 7f36e5411390 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:50:18.479 7f36e5411390 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:50:18.479 7f36e5411390 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:50:18.479 7f36e5411390 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:50:18.479 7f36e5411390 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:50:18.480 7f36e5411390 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:50:18.480 7f36e5411390 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:18.480 7f36e5411390 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:50:18.480 7f36e5411390 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:18.480 7f36e5411390 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:18.480 7f36e5411390 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:50:18.480 7f36e5411390 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:22.482 7f36eeb8c240 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:50:27.060 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:50:27.486 7f36eeb8c240 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:50:27.486 7f36eeb8c240 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:50:27.487 7f36eeb8c240 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:50:27.487 7f36eeb8c240 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:50:27.487 7f36eeb8c240 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:50:27.487 7f36eeb8c240 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:50:27.487 7f36eeb8c240 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 database_usage = [ << disk_space_used_1k = 51153932 disk_space_available_1k = 29936552 analytics_db_size_1k = 996 >>, ] >> 02/13/2025 02:50:27.487 7f36eeb8c240 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:50:27.487 7f36eeb8c240 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:50:27.488 7f36eeb8c240 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:50:27.488 7f36eeb8c240 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:27.488 7f36eeb8c240 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:50:27.488 7f36eeb8c240 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:27.488 7f36eeb8c240 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:27.488 7f36eeb8c240 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:50:27.489 7f36eeb8c240 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:31.488 7f36e6809978 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:50:35.497 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51154168, partition_space_available_1k=29936316, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1307588, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23904, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43064, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15011728, free=8902400, buffers=911400, cached=7839584, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.13, five_min_avg=3.28, fifteen_min_avg=3.25, cpu_share=4.12, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:50:36.492 7f36e6809978 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:50:36.492 7f36e6809978 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:50:36.492 7f36e6809978 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:50:36.492 7f36e6809978 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:50:36.492 7f36e6809978 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:50:36.492 7f36e6809978 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:50:36.493 7f36e6809978 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51154168 partition_space_available_1k = 29936316 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1307588 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 23904 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 43064 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15011728 free = 8902400 buffers = 911400 cached = 7839584 node_type = database-node >> system_cpu_usage = << one_min_avg = 4.13 five_min_avg = 3.28 fifteen_min_avg = 3.25 cpu_share = 4.12 node_type = database-node >> >> 02/13/2025 02:50:36.493 7f36e6809978 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:50:36.493 7f36e6809978 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:50:36.493 7f36e6809978 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:50:36.494 7f36e6809978 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:36.494 7f36e6809978 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:50:36.494 7f36e6809978 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:36.494 7f36e6809978 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:36.494 7f36e6809978 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:50:36.494 7f36e6809978 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:40.493 7f36e8a5bac8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:50:45.499 7f36e8a5bac8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:50:45.500 7f36e8a5bac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:50:45.500 7f36e8a5bac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:50:45.500 7f36e8a5bac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:50:45.500 7f36e8a5bac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:50:45.500 7f36e8a5bac8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:50:45.501 7f36e8a5bac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:50:45.501 7f36e8a5bac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:45.501 7f36e8a5bac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:50:45.501 7f36e8a5bac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:45.501 7f36e8a5bac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:45.501 7f36e8a5bac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:50:45.501 7f36e8a5bac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:49.500 7f36e5437a20 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:50:54.504 7f36e5437a20 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:50:54.505 7f36e5437a20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:50:54.505 7f36e5437a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:50:54.505 7f36e5437a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:50:54.505 7f36e5437a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:50:54.505 7f36e5437a20 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:50:54.506 7f36e5437a20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:50:54.506 7f36e5437a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:54.506 7f36e5437a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:50:54.506 7f36e5437a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:54.506 7f36e5437a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:54.506 7f36e5437a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:50:54.506 7f36e5437a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:50:58.509 7f36e8874f60 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:51:03.515 7f36e8874f60 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:51:03.515 7f36e8874f60 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:51:03.515 7f36e8874f60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:51:03.516 7f36e8874f60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:51:03.516 7f36e8874f60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:51:03.516 7f36e8874f60 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:51:03.516 7f36e8874f60 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:51:03.516 7f36e8874f60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:03.517 7f36e8874f60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:51:03.517 7f36e8874f60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:03.517 7f36e8874f60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:03.517 7f36e8874f60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:51:03.517 7f36e8874f60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:07.518 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:51:12.524 7f36e8cee2e8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:51:12.525 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:51:12.525 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:51:12.525 7f36e8cee2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:51:12.525 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:51:12.525 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:51:12.526 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:51:12.526 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:12.526 7f36e8cee2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:51:12.526 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:12.527 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:12.527 7f36e8cee2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:51:12.527 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:16.527 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:51:21.531 7f36e8cee2e8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:51:21.531 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:51:21.531 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:51:21.532 7f36e8cee2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:51:21.532 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:51:21.532 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:51:21.532 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:51:21.532 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:21.533 7f36e8cee2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:51:21.533 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:21.533 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:21.533 7f36e8cee2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:51:21.533 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:25.535 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:51:27.095 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:51:30.537 7f36e8cee2e8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:51:30.537 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:51:30.537 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:51:30.538 7f36e8cee2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:51:30.538 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:51:30.538 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:51:30.538 7f36e8cee2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 database_usage = [ << disk_space_used_1k = 51154440 disk_space_available_1k = 29936044 analytics_db_size_1k = 996 >>, ] >> 02/13/2025 02:51:30.538 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:51:30.538 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:51:30.539 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:51:30.539 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:30.539 7f36e8cee2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:51:30.539 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:30.540 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:30.540 7f36e8cee2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:51:30.540 7f36e8cee2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:34.403 7f36e54376d8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:51:34.537 7f36e67ed198 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:51:35.603 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51158488, partition_space_available_1k=29931996, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1307864, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=24072, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43104, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15030204, free=8883076, buffers=911404, cached=7840428, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.31, five_min_avg=2.91, fifteen_min_avg=3.12, cpu_share=1.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:51:39.545 7f36e67ed198 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:51:39.545 7f36e67ed198 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:51:39.545 7f36e67ed198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:51:39.545 7f36e67ed198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:51:39.545 7f36e67ed198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:51:39.545 7f36e67ed198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:51:39.546 7f36e67ed198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51158488 partition_space_available_1k = 29931996 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1307864 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 24072 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 43104 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15030204 free = 8883076 buffers = 911404 cached = 7840428 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.31 five_min_avg = 2.91 fifteen_min_avg = 3.12 cpu_share = 1.88 node_type = database-node >> >> 02/13/2025 02:51:39.546 7f36e67ed198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:51:39.546 7f36e67ed198 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:51:39.546 7f36e67ed198 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:51:39.547 7f36e67ed198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:39.547 7f36e67ed198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:51:39.547 7f36e67ed198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:39.547 7f36e67ed198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:39.547 7f36e67ed198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:51:39.547 7f36e67ed198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:43.546 7f36e54114e0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:51:45.489 7f36e98aec18 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:51:48.550 7f36e54114e0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:51:48.550 7f36e54114e0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:51:48.550 7f36e54114e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:51:48.550 7f36e54114e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:51:48.551 7f36e54114e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:51:48.551 7f36e54114e0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:51:48.551 7f36e54114e0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:51:48.551 7f36e54114e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:48.551 7f36e54114e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:51:48.552 7f36e54114e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:48.552 7f36e54114e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:48.552 7f36e54114e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:51:48.552 7f36e54114e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:52.553 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:51:57.558 7f36e98a3ac8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:51:57.558 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:51:57.558 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:51:57.558 7f36e98a3ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:51:57.559 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:51:57.559 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:51:57.559 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:51:57.559 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:57.559 7f36e98a3ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:51:57.559 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:57.560 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:51:57.560 7f36e98a3ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:51:57.561 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:01.562 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:52:06.565 7f36e98a3ac8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:52:06.565 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:52:06.565 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:52:06.565 7f36e98a3ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:52:06.566 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:52:06.566 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:52:06.566 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:52:06.566 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:06.567 7f36e98a3ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:52:06.567 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:06.567 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:06.567 7f36e98a3ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:52:06.567 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:10.569 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:52:15.575 7f36e98a3ac8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:52:15.575 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:52:15.576 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:52:15.576 7f36e98a3ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:52:15.576 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:52:15.576 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:52:15.577 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:52:15.577 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:15.577 7f36e98a3ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:52:15.577 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:15.577 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:15.577 7f36e98a3ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:52:15.578 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:19.580 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:52:24.586 7f36e98a3ac8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:52:24.586 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:52:24.586 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:52:24.587 7f36e98a3ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:52:24.587 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:52:24.587 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:52:24.587 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:52:24.587 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:24.588 7f36e98a3ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:52:24.588 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:24.588 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:24.588 7f36e98a3ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:52:24.588 7f36e98a3ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:27.071 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:52:27.450 7f36efa76438 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:27.450 7f36efa76438 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 database_usage = [ << disk_space_used_1k = 51158384 disk_space_available_1k = 29932100 analytics_db_size_1k = 996 >>, ] >> 02/13/2025 02:52:27.450 7f36efa76438 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:28.587 7f36e5442f60 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:52:33.589 7f36e5442f60 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:52:33.590 7f36e5442f60 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:52:33.590 7f36e5442f60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:52:33.590 7f36e5442f60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:52:33.590 7f36e5442f60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:52:33.590 7f36e5442f60 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:52:33.591 7f36e5442f60 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:52:33.591 7f36e5442f60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:33.591 7f36e5442f60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:52:33.591 7f36e5442f60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:33.591 7f36e5442f60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:33.592 7f36e5442f60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:52:33.592 7f36e5442f60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:35.431 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51158640, partition_space_available_1k=29931844, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1308336, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=24188, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43116, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14990936, free=8916868, buffers=911408, cached=7845900, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.5, five_min_avg=2.56, fifteen_min_avg=2.99, cpu_share=2.38, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:52:35.432 7f36e67f0a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:35.432 7f36e67f0a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51158640 partition_space_available_1k = 29931844 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1308336 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 24188 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 43116 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 14990936 free = 8916868 buffers = 911408 cached = 7845900 node_type = database-node >> system_cpu_usage = << one_min_avg = 1.5 five_min_avg = 2.56 fifteen_min_avg = 2.99 cpu_share = 2.38 node_type = database-node >> >> 02/13/2025 02:52:35.432 7f36e67f0a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:37.591 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:52:42.597 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:52:42.597 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:52:42.597 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:52:42.597 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:52:42.597 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:52:42.598 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:52:42.598 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:52:42.598 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:42.598 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:52:42.598 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:42.598 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:42.599 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:52:42.599 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:46.599 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:52:51.602 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:52:51.602 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:52:51.602 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:52:51.602 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:52:51.602 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:52:51.602 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:52:51.603 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:52:51.603 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:51.603 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:52:51.603 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:51.603 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:51.604 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:52:51.604 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:52:55.606 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:53:00.612 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:53:00.612 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:53:00.613 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:53:00.613 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:53:00.613 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:53:00.613 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:53:00.614 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:53:00.614 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:00.614 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:53:00.614 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:00.614 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:00.614 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:53:00.614 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:04.614 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:53:09.619 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:53:09.619 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:53:09.620 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:53:09.620 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:53:09.620 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:53:09.620 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:53:09.621 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:53:09.621 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:09.621 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:53:09.622 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:09.622 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:09.622 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:53:09.622 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:13.623 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:53:18.629 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:53:18.629 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:53:18.630 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:53:18.630 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:53:18.630 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:53:18.630 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:53:18.630 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:53:18.631 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:18.632 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:53:18.632 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:18.632 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:18.632 7f36e8cc2a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:53:18.632 7f36e8cc2a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:22.630 7f36e5411e10 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:53:27.110 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:53:27.632 7f36e5411e10 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:53:27.632 7f36e5411e10 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:53:27.632 7f36e5411e10 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:53:27.633 7f36e5411e10 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:53:27.633 7f36e5411e10 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:53:27.633 7f36e5411e10 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:53:27.633 7f36e5411e10 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 database_usage = [ << disk_space_used_1k = 51157216 disk_space_available_1k = 29933268 analytics_db_size_1k = 996 >>, ] >> 02/13/2025 02:53:27.633 7f36e5411e10 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:53:27.633 7f36e5411e10 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:53:27.634 7f36e5411e10 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:53:27.634 7f36e5411e10 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:27.634 7f36e5411e10 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:53:27.634 7f36e5411e10 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:27.634 7f36e5411e10 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:27.634 7f36e5411e10 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:53:27.634 7f36e5411e10 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:31.633 7f36e5ebaac8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:53:35.559 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51157464, partition_space_available_1k=29933020, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1308452, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=24348, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43184, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15001016, free=8907460, buffers=911408, cached=7845228, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.48, five_min_avg=2.38, fifteen_min_avg=2.9, cpu_share=1.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:53:36.636 7f36e5ebaac8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:53:36.636 7f36e5ebaac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:53:36.636 7f36e5ebaac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:53:36.637 7f36e5ebaac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:53:36.637 7f36e5ebaac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:53:36.637 7f36e5ebaac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:53:36.637 7f36e5ebaac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51157464 partition_space_available_1k = 29933020 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1308452 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 24348 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 43184 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15001016 free = 8907460 buffers = 911408 cached = 7845228 node_type = database-node >> system_cpu_usage = << one_min_avg = 1.48 five_min_avg = 2.38 fifteen_min_avg = 2.9 cpu_share = 1.88 node_type = database-node >> >> 02/13/2025 02:53:36.637 7f36e5ebaac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:53:36.637 7f36e5ebaac8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:53:36.638 7f36e5ebaac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:53:36.638 7f36e5ebaac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:36.638 7f36e5ebaac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:53:36.638 7f36e5ebaac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:36.638 7f36e5ebaac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:36.638 7f36e5ebaac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:53:36.638 7f36e5ebaac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:40.638 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:53:45.644 7f36e8a3d2e8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:53:45.645 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:53:45.645 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:53:45.645 7f36e8a3d2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:53:45.645 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:53:45.645 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:53:45.646 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:53:45.646 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:45.646 7f36e8a3d2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:53:45.646 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:45.646 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:45.646 7f36e8a3d2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:53:45.647 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:49.647 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:53:54.653 7f36e8a3d2e8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:53:54.653 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:53:54.653 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:53:54.653 7f36e8a3d2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:53:54.654 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:53:54.654 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:53:54.654 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:53:54.654 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:54.655 7f36e8a3d2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:53:54.655 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:54.655 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:54.655 7f36e8a3d2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:53:54.655 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:53:58.657 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:54:00.164 7f36e53d1e10 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:54:03.662 7f36e8a3d2e8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:54:03.663 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:54:03.663 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:54:03.663 7f36e8a3d2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:54:03.663 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:54:03.663 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:54:03.664 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:54:03.664 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:03.664 7f36e8a3d2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:54:03.665 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:03.665 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:03.665 7f36e8a3d2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:54:03.665 7f36e8a3d2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:07.664 7f36e8f4f780 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:54:12.670 7f36e8f4f780 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:54:12.670 7f36e8f4f780 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:54:12.671 7f36e8f4f780 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:54:12.671 7f36e8f4f780 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:54:12.671 7f36e8f4f780 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:54:12.671 7f36e8f4f780 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:54:12.672 7f36e8f4f780 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:54:12.672 7f36e8f4f780 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:12.672 7f36e8f4f780 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:54:12.672 7f36e8f4f780 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:12.672 7f36e8f4f780 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:12.672 7f36e8f4f780 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:54:12.673 7f36e8f4f780 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:16.674 7f36e8f4f780 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:54:21.677 7f36e8f4f780 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:54:21.677 7f36e8f4f780 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:54:21.677 7f36e8f4f780 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:54:21.677 7f36e8f4f780 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:54:21.677 7f36e8f4f780 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:54:21.677 7f36e8f4f780 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:54:21.678 7f36e8f4f780 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:54:21.678 7f36e8f4f780 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:21.678 7f36e8f4f780 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:54:21.679 7f36e8f4f780 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:21.679 7f36e8f4f780 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:21.679 7f36e8f4f780 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:54:21.679 7f36e8f4f780 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:25.681 7f36e5e93a20 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:54:27.067 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:54:30.683 7f36e5e93a20 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:54:30.684 7f36e5e93a20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:54:30.684 7f36e5e93a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:54:30.684 7f36e5e93a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:54:30.684 7f36e5e93a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:54:30.685 7f36e5e93a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:54:30.685 7f36e5e93a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 database_usage = [ << disk_space_used_1k = 51160956 disk_space_available_1k = 29929528 analytics_db_size_1k = 996 >>, ] >> 02/13/2025 02:54:30.685 7f36e5e93a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:54:30.685 7f36e5e93a20 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:54:30.686 7f36e5e93a20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:54:30.686 7f36e5e93a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:30.686 7f36e5e93a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:54:30.686 7f36e5e93a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:30.687 7f36e5e93a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:30.687 7f36e5e93a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:54:30.687 7f36e5e93a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:34.684 7f36e5437cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:54:35.523 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51161180, partition_space_available_1k=29929304, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1308360, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=24188, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43224, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14979820, free=8923360, buffers=911408, cached=7850524, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.05, five_min_avg=2.1, fifteen_min_avg=2.77, cpu_share=2.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:54:39.688 7f36e5437cc0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:54:39.688 7f36e5437cc0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:54:39.688 7f36e5437cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:54:39.688 7f36e5437cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:54:39.689 7f36e5437cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:54:39.689 7f36e5437cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:54:39.689 7f36e5437cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51161180 partition_space_available_1k = 29929304 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1308360 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 24188 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 43224 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 14979820 free = 8923360 buffers = 911408 cached = 7850524 node_type = database-node >> system_cpu_usage = << one_min_avg = 1.05 five_min_avg = 2.1 fifteen_min_avg = 2.77 cpu_share = 2.0 node_type = database-node >> >> 02/13/2025 02:54:39.689 7f36e5437cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:54:39.689 7f36e5437cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:54:39.690 7f36e5437cc0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:54:39.690 7f36e5437cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:39.690 7f36e5437cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:54:39.690 7f36e5437cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:39.690 7f36e5437cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:39.690 7f36e5437cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:54:39.690 7f36e5437cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:43.692 7f36e5437cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:54:48.699 7f36e5437cc0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:54:48.699 7f36e5437cc0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:54:48.699 7f36e5437cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:54:48.699 7f36e5437cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:54:48.700 7f36e5437cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:54:48.700 7f36e5437cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:54:48.700 7f36e5437cc0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:54:48.700 7f36e5437cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:48.701 7f36e5437cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:54:48.701 7f36e5437cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:48.701 7f36e5437cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:48.701 7f36e5437cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:54:48.701 7f36e5437cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:52.700 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:54:57.706 7f36e8f4f198 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:54:57.706 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:54:57.706 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:54:57.706 7f36e8f4f198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:54:57.707 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:54:57.707 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:54:57.707 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:54:57.708 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:57.708 7f36e8f4f198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:54:57.708 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:57.708 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:54:57.708 7f36e8f4f198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:54:57.708 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:01.710 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:55:06.712 7f36e8f4f198 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:55:06.712 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:55:06.712 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:55:06.712 7f36e8f4f198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:55:06.713 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:55:06.713 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:55:06.713 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:55:06.713 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:06.714 7f36e8f4f198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:55:06.714 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:06.714 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:06.714 7f36e8f4f198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:55:06.714 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:06.893 7f36e8f4f198 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:55:10.716 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:55:15.722 7f36e8f4f198 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:55:15.722 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:55:15.722 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:55:15.723 7f36e8f4f198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:55:15.723 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:55:15.723 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:55:15.723 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:55:15.724 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:15.724 7f36e8f4f198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:55:15.724 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:15.724 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:15.724 7f36e8f4f198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:55:15.724 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:19.726 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:55:24.730 7f36e8f4f198 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:55:24.730 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:55:24.730 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:55:24.730 7f36e8f4f198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:55:24.731 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:55:24.731 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:55:24.731 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:55:24.731 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:24.732 7f36e8f4f198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:55:24.732 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:24.732 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:24.732 7f36e8f4f198 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:55:24.732 7f36e8f4f198 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:27.063 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:55:27.456 7f36e46fbb70 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:27.456 7f36e46fbb70 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 database_usage = [ << disk_space_used_1k = 51165024 disk_space_available_1k = 29925460 analytics_db_size_1k = 996 >>, ] >> 02/13/2025 02:55:27.457 7f36e46fbb70 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:28.731 7f36e67f02e8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:55:33.733 7f36e67f02e8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:55:33.733 7f36e67f02e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:55:33.733 7f36e67f02e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:55:33.734 7f36e67f02e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:55:33.734 7f36e67f02e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:55:33.734 7f36e67f02e8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:55:33.735 7f36e67f02e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:55:33.735 7f36e67f02e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:33.735 7f36e67f02e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:55:33.735 7f36e67f02e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:33.735 7f36e67f02e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:33.735 7f36e67f02e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:55:33.735 7f36e67f02e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:35.511 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51165292, partition_space_available_1k=29925192, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1308172, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=24396, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43240, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14991784, free=8905564, buffers=911408, cached=7856356, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.17, five_min_avg=1.99, fifteen_min_avg=2.69, cpu_share=2.12, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:55:35.511 7f36e54372e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:35.512 7f36e54372e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51165292 partition_space_available_1k = 29925192 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1308172 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 24396 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 43240 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 14991784 free = 8905564 buffers = 911408 cached = 7856356 node_type = database-node >> system_cpu_usage = << one_min_avg = 1.17 five_min_avg = 1.99 fifteen_min_avg = 2.69 cpu_share = 2.12 node_type = database-node >> >> 02/13/2025 02:55:35.512 7f36e54372e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:37.734 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:55:42.740 7f36ea3db0f0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:55:42.741 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:55:42.741 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:55:42.741 7f36ea3db0f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:55:42.741 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:55:42.741 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:55:42.742 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:55:42.742 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:42.742 7f36ea3db0f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:55:42.742 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:42.742 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:42.743 7f36ea3db0f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:55:42.743 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:46.745 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:55:51.750 7f36ea3db0f0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:55:51.750 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:55:51.750 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:55:51.751 7f36ea3db0f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:55:51.751 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:55:51.751 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:55:51.751 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:55:51.752 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:51.752 7f36ea3db0f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:55:51.752 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:51.752 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:51.752 7f36ea3db0f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:55:51.752 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:55:55.754 7f36e65a64e0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:56:00.761 7f36e65a64e0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:56:00.761 7f36e65a64e0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:56:00.763 7f36e65a64e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:56:00.763 7f36e65a64e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:56:00.763 7f36e65a64e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:56:00.763 7f36e65a64e0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:56:00.764 7f36e65a64e0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:56:00.764 7f36e65a64e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:00.764 7f36e65a64e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:56:00.764 7f36e65a64e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:00.764 7f36e65a64e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:00.764 7f36e65a64e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:56:00.765 7f36e65a64e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:04.765 7f36e65a64e0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:56:06.661 7f36e543a6d8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:56:09.768 7f36e65a64e0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:56:09.769 7f36e65a64e0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:56:09.769 7f36e65a64e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:56:09.769 7f36e65a64e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:56:09.769 7f36e65a64e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:56:09.769 7f36e65a64e0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:56:09.770 7f36e65a64e0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:56:09.770 7f36e65a64e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:09.770 7f36e65a64e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:56:09.770 7f36e65a64e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:09.770 7f36e65a64e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:09.771 7f36e65a64e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:56:09.771 7f36e65a64e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:13.770 7f36e543aa20 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:56:17.495 7f36e6809e10 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 02/13/2025 02:56:18.773 7f36e543aa20 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:56:18.773 7f36e543aa20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:56:18.774 7f36e543aa20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:56:18.774 7f36e543aa20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:56:18.774 7f36e543aa20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:56:18.774 7f36e543aa20 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:56:18.775 7f36e543aa20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:56:18.775 7f36e543aa20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:18.775 7f36e543aa20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:56:18.775 7f36e543aa20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:18.775 7f36e543aa20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:18.775 7f36e543aa20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:56:18.775 7f36e543aa20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:22.774 7f36e67edb70 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:56:27.066 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:56:27.777 7f36e67edb70 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:56:27.777 7f36e67edb70 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:56:27.777 7f36e67edb70 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:56:27.777 7f36e67edb70 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:56:27.777 7f36e67edb70 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:56:27.778 7f36e67edb70 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:56:27.778 7f36e67edb70 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 database_usage = [ << disk_space_used_1k = 51170156 disk_space_available_1k = 29920328 analytics_db_size_1k = 996 >>, ] >> 02/13/2025 02:56:27.778 7f36e67edb70 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:56:27.778 7f36e67edb70 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:56:27.778 7f36e67edb70 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:56:27.779 7f36e67edb70 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:27.779 7f36e67edb70 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:56:27.779 7f36e67edb70 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:27.779 7f36e67edb70 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:27.779 7f36e67edb70 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:56:27.779 7f36e67edb70 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:31.778 7f36e8872588 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:56:35.507 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51170416, partition_space_available_1k=29920068, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1307524, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=24564, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43316, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15059200, free=8824036, buffers=911412, cached=7870464, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.44, five_min_avg=2.92, fifteen_min_avg=2.97, cpu_share=4.38, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:56:36.781 7f36e8872588 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:56:36.781 7f36e8872588 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:56:36.781 7f36e8872588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:56:36.781 7f36e8872588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:56:36.782 7f36e8872588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:56:36.782 7f36e8872588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:56:36.782 7f36e8872588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51170416 partition_space_available_1k = 29920068 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1307524 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 24564 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 43316 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15059200 free = 8824036 buffers = 911412 cached = 7870464 node_type = database-node >> system_cpu_usage = << one_min_avg = 4.44 five_min_avg = 2.92 fifteen_min_avg = 2.97 cpu_share = 4.38 node_type = database-node >> >> 02/13/2025 02:56:36.782 7f36e8872588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:56:36.782 7f36e8872588 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:56:36.783 7f36e8872588 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:56:36.783 7f36e8872588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:36.783 7f36e8872588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:56:36.783 7f36e8872588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:36.783 7f36e8872588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:36.783 7f36e8872588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:56:36.783 7f36e8872588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:40.782 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:56:45.786 7f36e8cd7ac8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:56:45.786 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:56:45.786 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:56:45.787 7f36e8cd7ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:56:45.787 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:56:45.787 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:56:45.788 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:56:45.788 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:45.788 7f36e8cd7ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:56:45.789 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:45.789 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:45.789 7f36e8cd7ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:56:45.790 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:49.790 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:56:54.793 7f36e8cd7ac8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:56:54.793 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:56:54.793 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:56:54.793 7f36e8cd7ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:56:54.794 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:56:54.794 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:56:54.794 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:56:54.794 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:54.795 7f36e8cd7ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:56:54.795 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:54.795 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:54.795 7f36e8cd7ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:56:54.795 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:56:58.796 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:57:03.802 7f36e8cd7ac8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:57:03.802 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:57:03.802 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:57:03.803 7f36e8cd7ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:57:03.803 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:57:03.803 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:57:03.803 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:57:03.804 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:03.804 7f36e8cd7ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:57:03.804 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:03.804 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:03.804 7f36e8cd7ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:57:03.804 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:07.806 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:57:12.812 7f36e8cd7ac8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:57:12.813 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:57:12.813 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:57:12.813 7f36e8cd7ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:57:12.813 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:57:12.813 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:57:12.814 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:57:12.814 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:12.814 7f36e8cd7ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:57:12.814 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:12.814 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:12.814 7f36e8cd7ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:57:12.815 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:16.816 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:57:21.822 7f36e8cd7ac8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:57:21.822 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:57:21.822 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:57:21.822 7f36e8cd7ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:57:21.822 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:57:21.823 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:57:21.823 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:57:21.823 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:21.823 7f36e8cd7ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:57:21.823 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:21.823 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:21.824 7f36e8cd7ac8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:57:21.824 7f36e8cd7ac8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:25.825 7f36e5e93588 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:57:27.091 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:57:30.828 7f36e5e93588 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:57:30.828 7f36e5e93588 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:57:30.828 7f36e5e93588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:57:30.829 7f36e5e93588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:57:30.829 7f36e5e93588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:57:30.829 7f36e5e93588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:57:30.829 7f36e5e93588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 database_usage = [ << disk_space_used_1k = 51175916 disk_space_available_1k = 29914568 analytics_db_size_1k = 996 >>, ] >> 02/13/2025 02:57:30.829 7f36e5e93588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:57:30.829 7f36e5e93588 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:57:30.830 7f36e5e93588 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:57:30.830 7f36e5e93588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:30.831 7f36e5e93588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:57:30.831 7f36e5e93588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:30.831 7f36e5e93588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:30.831 7f36e5e93588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:57:30.831 7f36e5e93588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:34.829 7f36e5437240 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:57:35.463 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51176176, partition_space_available_1k=29914308, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1309124, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=24680, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43352, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15031808, free=8844852, buffers=911412, cached=7877040, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.86, five_min_avg=2.75, fifteen_min_avg=2.91, cpu_share=2.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:57:39.833 7f36e5437240 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:57:39.833 7f36e5437240 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:57:39.833 7f36e5437240 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:57:39.834 7f36e5437240 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:57:39.834 7f36e5437240 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:57:39.834 7f36e5437240 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:57:39.834 7f36e5437240 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51176176 partition_space_available_1k = 29914308 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1309124 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 24680 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 43352 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15031808 free = 8844852 buffers = 911412 cached = 7877040 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.86 five_min_avg = 2.75 fifteen_min_avg = 2.91 cpu_share = 2.25 node_type = database-node >> >> 02/13/2025 02:57:39.834 7f36e5437240 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:57:39.834 7f36e5437240 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:57:39.835 7f36e5437240 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:57:39.835 7f36e5437240 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:39.835 7f36e5437240 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:57:39.835 7f36e5437240 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:39.835 7f36e5437240 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:39.835 7f36e5437240 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:57:39.835 7f36e5437240 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:43.838 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:57:48.844 7f36ea3db0f0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:57:48.845 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:57:48.845 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:57:48.845 7f36ea3db0f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:57:48.845 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:57:48.845 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:57:48.846 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:57:48.846 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:48.846 7f36ea3db0f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:57:48.846 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:48.846 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:48.847 7f36ea3db0f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:57:48.847 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:52.848 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:57:57.854 7f36ea3db0f0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:57:57.854 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:57:57.854 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:57:57.854 7f36ea3db0f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:57:57.855 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:57:57.855 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:57:57.855 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:57:57.855 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:57.855 7f36ea3db0f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:57:57.856 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:57.856 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:57:57.856 7f36ea3db0f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:57:57.856 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:58:01.858 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:58:06.863 7f36ea3db0f0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:58:06.864 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:58:06.864 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:58:06.864 7f36ea3db0f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:58:06.864 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:58:06.864 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:58:06.865 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:58:06.865 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:58:06.865 7f36ea3db0f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:58:06.865 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:58:06.866 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:58:06.866 7f36ea3db0f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:58:06.866 7f36ea3db0f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:58:10.864 7f36e8a5b828 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:58:15.870 7f36e8a5b828 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:58:15.870 7f36e8a5b828 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:58:15.870 7f36e8a5b828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:58:15.871 7f36e8a5b828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:58:15.871 7f36e8a5b828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:58:15.871 7f36e8a5b828 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:58:15.871 7f36e8a5b828 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:58:15.872 7f36e8a5b828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:58:15.872 7f36e8a5b828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:58:15.872 7f36e8a5b828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:58:15.872 7f36e8a5b828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:58:15.872 7f36e8a5b828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:58:15.872 7f36e8a5b828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:58:19.874 7f36e8a5b828 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:58:24.877 7f36e8a5b828 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:58:24.877 7f36e8a5b828 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:58:24.877 7f36e8a5b828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:58:24.878 7f36e8a5b828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:58:24.878 7f36e8a5b828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:58:24.878 7f36e8a5b828 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:58:24.878 7f36e8a5b828 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:58:24.879 7f36e8a5b828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:58:24.879 7f36e8a5b828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:58:24.879 7f36e8a5b828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:58:24.879 7f36e8a5b828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:58:24.879 7f36e8a5b828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:58:24.879 7f36e8a5b828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:58:27.063 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 02/13/2025 02:58:27.449 7f36e88722e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:58:27.450 7f36e88722e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 database_usage = [ << disk_space_used_1k = 51186156 disk_space_available_1k = 29904328 analytics_db_size_1k = 996 >>, ] >> 02/13/2025 02:58:27.450 7f36e88722e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:58:28.878 7f36e541dd68 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:58:33.880 7f36e541dd68 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 02/13/2025 02:58:33.880 7f36e541dd68 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:58:33.880 7f36e541dd68 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:58:33.880 7f36e541dd68 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 02/13/2025 02:58:33.881 7f36e541dd68 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:58:33.881 7f36e541dd68 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 02/13/2025 02:58:33.881 7f36e541dd68 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 02/13/2025 02:58:33.881 7f36e541dd68 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:58:33.882 7f36e541dd68 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 02/13/2025 02:58:33.882 7f36e541dd68 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:58:33.882 7f36e541dd68 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:58:33.882 7f36e541dd68 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-473-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1739412986048227 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 02/13/2025 02:58:33.882 7f36e541dd68 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:58:35.776 7f36e8f12308 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-473-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51191952, partition_space_available_1k=29898532, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1309164, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=24812, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=43368, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15094256, free=8586512, buffers=912912, cached=8071432, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.43, five_min_avg=2.61, fifteen_min_avg=2.85, cpu_share=2.38, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 02/13/2025 02:58:35.777 7f36e5e76780 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:58:35.777 7f36e5e76780 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-473-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51191952 partition_space_available_1k = 29898532 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1309164 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 24812 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 43368 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15094256 free = 8586512 buffers = 912912 cached = 8071432 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.43 five_min_avg = 2.61 fifteen_min_avg = 2.85 cpu_share = 2.38 node_type = database-node >> >> 02/13/2025 02:58:35.777 7f36e5e76780 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:58:37.882 7f36e5e76780 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:58:39.790 7f36e5e930f0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map