03/20/2025 00:44:19.941 7faa820aadf0 [contrail-config-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 03/20/2025 00:44:19.941 7faa820aadf0 [contrail-config-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 03/20/2025 00:44:19.941 7faa820aadf0 [contrail-config-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 03/20/2025 00:44:19.949 7faa80a28148 [contrail-config-database-nodemgr] [ERROR]: Cannot write http_port 8112 to /tmp/contrail-config-database-nodemgr.0.http_port 03/20/2025 00:44:19.949 7faa80a28148 [contrail-config-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8112 03/20/2025 00:44:19.950 7faa7c8a5648 [contrail-config-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 03/20/2025 00:44:19.950 7faa7c8a5648 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/20/2025 00:44:19.951 7faa7c8a5648 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/20/2025 00:44:19.951 7faa7c8a5648 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 03/20/2025 00:44:19.951 7faa7c8a5648 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:19.951 7faa7c8a5648 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/20/2025 00:44:19.952 7faa7c8a5648 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:19.952 7faa7c8a5648 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:19.952 7faa7c8a5648 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 03/20/2025 00:44:19.952 7faa7c8a5648 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:20.032 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 10873332665642725122558304532124286267164827865277466652510538593036487547962, 'start': '1742431457000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'zookeeper', 'group': 'zookeeper', 'pid': 5057221750418482296835987726371120978093503601993300154265813374933025949962, 'start': '1742431457000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-config-database-nodemgr', 'group': 'contrail-config-database-nodemgr', 'pid': 53321400583668198283938383852360463026999483032090278927571254795566302906205, 'start': '1742431458000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 03/20/2025 00:44:20.176 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:20.176 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. build_info = {"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-17 01:12:14.132392", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8139-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 4 num_core_per_socket = 4 num_thread_per_core = 1 >> running_package_version = nightly.r24.1.centos-nightly_r24.1_centos.el7 installed_package_version = nightly.r24.1.centos-nightly_r24.1_centos.el7 >> 03/20/2025 00:44:20.176 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:20.177 7faa820aadf0 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1742431460176907, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/20/2025 00:44:20.177 7faa820aadf0 [contrail-config-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1742431460177190, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742431457000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='zookeeper', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742431457000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-config-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742431458000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-17 01:12:14.132392", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8139-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/20/2025 00:44:20.179 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:20.179 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/20/2025 00:44:20.179 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:20.179 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:20.179 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742431457000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = zookeeper process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742431457000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-config-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742431458000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-17 01:12:14.132392", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8139-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]} >> 03/20/2025 00:44:20.179 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:20.376 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 00:44:20.581 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:DatabaseUsage] map 03/20/2025 00:44:20.582 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:20.582 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. database_usage = [ << disk_space_used_1k = 4585460 disk_space_available_1k = 76505024 config_db_size_1k = 4 >>, ] >> 03/20/2025 00:44:20.582 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:20.957 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to connect to database by CQL: Result: 1 03/20/2025 00:44:20.957 7faa820aadf0 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1742431460957693, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/20/2025 00:44:20.960 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:20.960 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/20/2025 00:44:20.961 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:22.869 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 00:44:23.951 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:44:23.953 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:44:23.953 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:44:23.953 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:44:23.954 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:44:23.954 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:44:23.954 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:44:23.954 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:44:23.955 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:23.955 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:44:23.955 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:23.955 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:23.956 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:44:23.956 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:24.737 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr config_db_uuid'. Error is: Result: 1 03/20/2025 00:44:25.739 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr to_bgp_keyspace'. Error is: Result: 1 03/20/2025 00:44:26.586 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr svc_monitor_keyspace'. Error is: Result: 1 03/20/2025 00:44:27.391 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr DISCOVERY_SERVER'. Error is: Result: 1 03/20/2025 00:44:27.953 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:44:27.955 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:44:27.956 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:44:27.956 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:44:27.956 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:44:27.956 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:44:27.956 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:44:27.957 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:44:27.957 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:27.957 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:44:27.957 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:27.957 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:27.958 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:44:27.958 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:28.702 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr useragent'. Error is: Result: 1 03/20/2025 00:44:29.851 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr dm_keyspace'. Error is: Result: 1 03/20/2025 00:44:31.956 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:44:31.958 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:44:31.958 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:44:31.958 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:44:31.959 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:44:31.959 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:44:31.959 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:44:31.960 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:44:31.960 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:31.960 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:44:31.960 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:31.961 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:31.961 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:44:31.961 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:34.875 7faa820aadf0 [contrail-config-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1742431474875522, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742431457000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='zookeeper', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742431457000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-config-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742431458000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-17 01:12:14.132392", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8139-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/20/2025 00:44:34.876 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=4950172, partition_space_available_1k=76140312, percentage_partition_space_used=6, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1235688, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=58172, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=50984, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=1775244, free=10040492, buffers=99712, cached=4260180, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.01, five_min_avg=0.82, fifteen_min_avg=0.3, cpu_share=0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 00:44:34.876 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:34.876 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742431457000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = zookeeper process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742431457000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-config-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742431458000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-17 01:12:14.132392", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8139-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]} >> 03/20/2025 00:44:34.876 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:34.877 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:34.877 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 4950172 partition_space_available_1k = 76140312 percentage_partition_space_used = 6 >> } all_core_file_list = [ ] process_mem_cpu_usage = { cassandra : << mem_virt = 1235688 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 58172 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 50984 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 16175628 used = 1775244 free = 10040492 buffers = 99712 cached = 4260180 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 3.01 five_min_avg = 0.82 fifteen_min_avg = 0.3 cpu_share = 0 node_type = config-database-node >> >> 03/20/2025 00:44:34.877 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:35.958 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:44:35.959 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:44:35.960 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:44:35.960 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:44:35.960 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:44:35.960 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:44:35.960 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:44:35.960 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:44:35.960 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:35.960 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:44:35.961 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:35.961 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:35.961 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:44:35.961 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:39.963 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:44:39.964 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:44:39.964 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:44:39.965 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:44:39.965 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:44:39.965 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:44:39.965 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:44:39.965 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:44:39.965 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:39.965 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:44:39.965 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:39.965 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:39.966 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:44:39.966 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:43.968 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:44:44.133 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:44:44.133 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:44:44.133 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:44:44.133 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:44:44.133 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:44:44.133 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:44:44.133 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:44:44.134 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:44.134 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:44:44.134 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:44.135 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:44.136 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:44:44.136 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:48.135 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:44:48.137 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:44:48.137 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:44:48.138 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:44:48.138 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:44:48.138 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:44:48.138 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:44:48.138 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:44:48.138 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:48.138 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:44:48.138 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:48.139 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:48.139 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:44:48.139 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:52.139 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:44:52.140 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:44:52.140 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:44:52.140 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:44:52.141 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:44:52.141 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:44:52.141 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:44:52.141 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:44:52.141 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:52.141 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:44:52.141 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:52.142 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:52.142 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:44:52.142 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:56.143 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:44:56.144 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:44:56.144 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:44:56.144 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:44:56.144 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:44:56.144 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:44:56.144 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:44:56.145 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:44:56.145 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:56.145 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:44:56.145 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:56.145 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:44:56.145 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:44:56.146 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:00.148 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:45:00.151 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:45:00.151 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:45:00.151 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:00.151 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:45:00.151 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:00.152 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:45:00.152 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:45:00.152 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:00.152 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:45:00.153 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:00.153 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:00.153 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:45:00.153 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:04.152 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:45:04.154 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:45:04.155 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:45:04.155 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:04.155 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:45:04.155 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:04.155 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:45:04.155 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:45:04.155 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:04.156 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:45:04.156 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:04.156 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:04.156 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:45:04.156 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:08.155 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:45:08.156 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:45:08.157 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:45:08.157 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:08.158 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:45:08.158 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:08.158 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:45:08.158 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:45:08.159 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:08.159 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:45:08.159 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:08.159 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:08.160 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:45:08.160 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:12.157 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:45:12.158 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:45:12.158 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:45:12.158 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:12.158 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:45:12.159 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:12.159 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:45:12.159 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:45:12.159 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:12.159 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:45:12.159 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:12.159 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:12.159 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:45:12.160 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:16.162 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:45:16.164 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:45:16.164 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:45:16.164 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:16.164 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:45:16.164 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:16.164 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:45:16.164 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:45:16.165 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:16.165 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:45:16.165 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:16.165 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:16.165 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:45:16.165 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:20.167 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:45:20.169 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:45:20.169 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:45:20.169 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:20.169 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:45:20.169 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:20.169 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:45:20.170 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:45:20.170 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:20.170 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:45:20.170 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:20.170 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:20.170 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:45:20.170 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:20.330 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 00:45:20.487 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:20.487 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. database_usage = [ << disk_space_used_1k = 5860848 disk_space_available_1k = 75229636 config_db_size_1k = 1624 >>, ] >> 03/20/2025 00:45:20.487 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:20.983 7faa820aadf0 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1742431520982932, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.12:8086'], status='Down', description='Connect to Idle on EvTcpConnectFail')]), flag_infos=None, description='Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/20/2025 00:45:20.984 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:20.984 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:45:20.984 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:21.751 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 00:45:24.169 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:45:24.171 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:45:24.172 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:45:24.172 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:24.172 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:45:24.172 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:24.172 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:45:24.172 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:45:24.172 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:24.173 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:45:24.173 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:24.173 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:24.173 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:45:24.173 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:26.774 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=5860820, partition_space_available_1k=75229664, percentage_partition_space_used=7, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1316596, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=57944, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51332, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=1809244, free=8988512, buffers=173684, cached=5204188, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.13, five_min_avg=0.94, fifteen_min_avg=0.37, cpu_share=7.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 00:45:26.774 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:26.774 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 5860820 partition_space_available_1k = 75229664 percentage_partition_space_used = 7 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1316596 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 57944 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 51332 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 16175628 used = 1809244 free = 8988512 buffers = 173684 cached = 5204188 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 2.13 five_min_avg = 0.94 fifteen_min_avg = 0.37 cpu_share = 7.75 node_type = config-database-node >> >> 03/20/2025 00:45:26.775 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:28.172 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:45:28.174 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:45:28.174 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:45:28.174 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:28.175 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:45:28.175 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:28.175 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:45:28.175 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:45:28.175 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:28.175 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:45:28.175 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:28.175 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:28.176 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:45:28.176 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:32.178 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:45:32.179 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:45:32.179 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:45:32.179 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:32.179 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:45:32.180 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:32.180 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:45:32.180 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:45:32.180 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:32.180 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:45:32.180 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:32.180 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:32.180 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:45:32.181 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:36.180 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:45:36.182 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:45:36.182 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:45:36.182 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:36.182 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:45:36.182 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:36.183 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:45:36.183 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:45:36.183 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:36.183 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:45:36.183 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:36.184 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:36.184 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:45:36.184 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:40.183 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:45:40.184 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:45:40.184 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:45:40.184 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:40.185 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:45:40.185 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:40.185 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:45:40.185 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:45:40.185 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:40.185 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:45:40.185 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:40.186 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:40.186 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:45:40.186 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:44.186 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:45:44.187 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:45:44.187 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:45:44.187 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:44.187 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:45:44.188 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:44.188 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:45:44.188 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:45:44.188 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:44.188 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:45:44.188 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:44.188 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:44.189 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:45:44.189 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:48.191 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:45:48.193 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:45:48.194 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:45:48.194 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:48.195 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:45:48.195 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:48.195 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:45:48.196 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:45:48.197 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:48.197 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:45:48.197 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:48.197 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:48.198 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:45:48.198 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:52.195 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:45:52.196 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:45:52.197 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:45:52.197 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:52.197 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:45:52.197 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:52.197 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:45:52.197 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:45:52.197 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:52.198 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:45:52.198 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:52.198 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:52.198 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:45:52.198 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:56.199 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:45:56.200 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:45:56.200 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:45:56.200 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:56.200 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:45:56.200 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:45:56.200 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:45:56.201 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:45:56.201 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:56.201 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:45:56.201 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:56.201 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:45:56.201 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:45:56.201 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:00.203 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:46:00.205 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:46:00.205 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:46:00.205 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:00.206 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:46:00.206 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:00.206 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:46:00.206 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:46:00.206 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:00.207 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:46:00.207 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:00.207 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:00.207 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:46:00.207 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:04.207 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:46:04.208 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:46:04.208 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:46:04.209 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:04.209 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:46:04.210 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:04.210 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:46:04.210 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:46:04.211 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:04.211 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:46:04.211 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:04.211 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:04.211 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:46:04.213 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:08.209 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:46:08.210 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:46:08.210 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:46:08.210 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:08.210 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:46:08.211 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:08.211 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:46:08.211 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:46:08.211 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:08.211 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:46:08.211 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:08.211 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:08.211 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:46:08.211 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:12.214 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:46:12.215 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:46:12.215 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:46:12.216 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:12.216 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:46:12.216 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:12.216 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:46:12.216 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:46:12.216 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:12.216 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:46:12.216 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:12.217 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:12.217 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:46:12.217 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:16.218 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:46:16.220 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:46:16.220 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:46:16.220 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:16.220 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:46:16.220 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:16.220 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:46:16.221 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:46:16.221 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:16.221 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:46:16.221 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:16.221 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:16.221 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:46:16.221 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:20.222 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:46:20.227 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:46:20.227 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:46:20.227 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:20.228 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:46:20.228 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:20.228 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:46:20.228 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:46:20.228 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:20.228 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:46:20.228 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:20.228 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:20.228 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:46:20.228 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:20.439 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 00:46:20.685 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:20.685 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. database_usage = [ << disk_space_used_1k = 6160368 disk_space_available_1k = 74930116 config_db_size_1k = 1812 >>, ] >> 03/20/2025 00:46:20.686 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:22.410 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 00:46:24.228 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:46:24.230 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:46:24.230 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:46:24.230 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:24.231 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:46:24.231 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:24.231 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:46:24.232 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:46:24.232 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:24.232 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:46:24.232 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:24.232 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:24.233 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:46:24.233 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:27.431 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=6160368, partition_space_available_1k=74930116, percentage_partition_space_used=8, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1614832, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=75520, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51556, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=2711896, free=7703324, buffers=205736, cached=5554672, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.55, five_min_avg=0.99, fifteen_min_avg=0.42, cpu_share=7.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 00:46:27.431 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:27.432 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 6160368 partition_space_available_1k = 74930116 percentage_partition_space_used = 8 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1614832 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 75520 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 51556 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 16175628 used = 2711896 free = 7703324 buffers = 205736 cached = 5554672 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 1.55 five_min_avg = 0.99 fifteen_min_avg = 0.42 cpu_share = 7.5 node_type = config-database-node >> >> 03/20/2025 00:46:27.432 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:28.231 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:46:28.233 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:46:28.233 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:46:28.233 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:28.234 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:46:28.234 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:28.234 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:46:28.234 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:46:28.235 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:28.235 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:46:28.235 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:28.235 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:28.236 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:46:28.236 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:32.235 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:46:32.236 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:46:32.236 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:46:32.236 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:32.236 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:46:32.236 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:32.236 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:46:32.237 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:46:32.237 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:32.237 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:46:32.237 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:32.237 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:32.237 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:46:32.237 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:36.236 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:46:36.239 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:46:36.239 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:46:36.239 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:36.239 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:46:36.239 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:36.239 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:46:36.240 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:46:36.240 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:36.240 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:46:36.240 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:36.240 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:36.240 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:46:36.241 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:40.242 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:46:40.243 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:46:40.244 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:46:40.244 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:40.244 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:46:40.244 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:40.245 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:46:40.245 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:46:40.245 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:40.245 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:46:40.246 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:40.246 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:40.246 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:46:40.246 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:44.247 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:46:44.248 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:46:44.248 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:46:44.248 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:44.248 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:46:44.249 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:44.249 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:46:44.249 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:46:44.249 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:44.249 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:46:44.249 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:44.249 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:44.249 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:46:44.249 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:48.251 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:46:48.253 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:46:48.253 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:46:48.253 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:48.253 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:46:48.253 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:48.253 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:46:48.254 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:46:48.254 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:48.254 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:46:48.254 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:48.254 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:48.254 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:46:48.254 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:52.254 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:46:52.255 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:46:52.256 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:46:52.256 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:52.256 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:46:52.256 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:52.256 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:46:52.256 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:46:52.256 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:52.256 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:46:52.257 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:52.257 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:52.257 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:46:52.257 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:56.256 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:46:56.257 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:46:56.257 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:46:56.257 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:56.258 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:46:56.258 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:46:56.258 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:46:56.258 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:46:56.258 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:56.258 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:46:56.258 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:56.259 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:46:56.259 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:46:56.259 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:00.259 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:47:00.260 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:47:00.261 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:47:00.261 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:00.261 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:47:00.261 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:00.261 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:47:00.261 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:47:00.261 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:00.262 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:47:00.262 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:00.262 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:00.262 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:47:00.262 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:04.263 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:47:04.264 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:47:04.264 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:47:04.264 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:04.265 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:47:04.265 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:04.265 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:47:04.265 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:47:04.265 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:04.265 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:47:04.265 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:04.266 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:04.266 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:47:04.266 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:08.266 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:47:08.267 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:47:08.268 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:47:08.268 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:08.268 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:47:08.268 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:08.268 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:47:08.269 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:47:08.269 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:08.269 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:47:08.269 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:08.269 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:08.269 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:47:08.269 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:12.271 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:47:12.272 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:47:12.272 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:47:12.272 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:12.273 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:47:12.273 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:12.273 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:47:12.273 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:47:12.273 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:12.273 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:47:12.273 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:12.274 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:12.274 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:47:12.274 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:16.276 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:47:16.278 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:47:16.278 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:47:16.278 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:16.279 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:47:16.279 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:16.279 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:47:16.279 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:47:16.279 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:16.279 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:47:16.279 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:16.279 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:16.280 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:47:16.280 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:20.279 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:47:20.281 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:47:20.281 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:47:20.281 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:20.282 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:47:20.282 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:20.282 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:47:20.282 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:47:20.283 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:20.283 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:47:20.283 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:20.283 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:20.283 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:47:20.283 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:20.379 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 00:47:20.547 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:20.547 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. database_usage = [ << disk_space_used_1k = 6278248 disk_space_available_1k = 74812236 config_db_size_1k = 1696 >>, ] >> 03/20/2025 00:47:20.547 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:21.651 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 00:47:24.283 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:47:24.285 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:47:24.285 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:47:24.285 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:24.285 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:47:24.285 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:24.285 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:47:24.286 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:47:24.286 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:24.286 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:47:24.286 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:24.286 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:24.286 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:47:24.286 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:26.673 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=6278256, partition_space_available_1k=74812228, percentage_partition_space_used=8, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1712692, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=77908, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51684, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=2945360, free=7341588, buffers=208204, cached=5680476, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.72, five_min_avg=0.86, fifteen_min_avg=0.41, cpu_share=4.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 00:47:26.673 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:26.674 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 6278256 partition_space_available_1k = 74812228 percentage_partition_space_used = 8 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1712692 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 77908 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 51684 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 16175628 used = 2945360 free = 7341588 buffers = 208204 cached = 5680476 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 0.72 five_min_avg = 0.86 fifteen_min_avg = 0.41 cpu_share = 4.25 node_type = config-database-node >> >> 03/20/2025 00:47:26.674 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:28.285 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:47:28.287 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:47:28.287 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:47:28.287 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:28.287 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:47:28.288 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:28.288 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:47:28.288 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:47:28.288 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:28.288 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:47:28.288 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:28.288 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:28.288 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:47:28.289 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:32.291 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:47:32.292 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:47:32.292 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:47:32.292 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:32.292 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:47:32.292 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:32.293 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:47:32.293 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:47:32.293 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:32.293 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:47:32.293 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:32.293 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:32.293 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:47:32.293 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:36.293 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:47:36.294 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:47:36.294 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:47:36.295 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:36.295 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:47:36.295 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:36.295 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:47:36.295 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:47:36.295 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:36.296 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:47:36.296 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:36.296 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:36.296 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:47:36.296 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:40.295 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:47:40.297 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:47:40.297 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:47:40.297 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:40.297 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:47:40.297 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:40.297 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:47:40.298 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:47:40.298 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:40.298 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:47:40.298 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:40.298 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:40.298 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:47:40.298 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:44.300 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:47:44.301 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:47:44.301 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:47:44.301 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:44.301 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:47:44.302 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:44.302 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:47:44.302 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:47:44.302 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:44.302 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:47:44.302 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:44.302 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:44.302 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:47:44.302 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:48.303 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:47:48.304 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:47:48.305 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:47:48.305 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:48.305 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:47:48.305 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:48.305 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:47:48.305 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:47:48.306 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:48.306 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:47:48.306 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:48.306 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:48.306 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:47:48.306 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:52.305 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:47:52.307 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:47:52.307 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:47:52.308 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:52.308 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:47:52.308 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:52.308 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:47:52.309 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:47:52.309 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:52.309 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:47:52.309 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:52.309 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:52.309 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:47:52.310 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:56.311 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:47:56.312 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:47:56.312 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:47:56.312 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:56.313 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:47:56.313 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:47:56.313 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:47:56.313 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:47:56.313 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:56.313 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:47:56.313 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:56.313 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:47:56.314 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:47:56.314 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:00.316 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:48:00.318 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:48:00.318 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:48:00.318 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:00.318 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:48:00.319 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:00.319 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:48:00.319 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:48:00.319 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:00.319 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:48:00.319 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:00.319 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:00.319 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:48:00.319 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:04.319 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:48:04.320 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:48:04.321 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:48:04.321 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:04.321 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:48:04.321 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:04.321 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:48:04.322 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:48:04.322 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:04.322 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:48:04.322 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:04.322 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:04.322 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:48:04.323 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:08.323 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:48:08.324 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:48:08.324 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:48:08.324 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:08.324 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:48:08.324 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:08.324 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:48:08.325 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:48:08.325 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:08.325 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:48:08.325 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:08.325 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:08.325 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:48:08.326 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:12.327 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:48:12.328 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:48:12.328 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:48:12.329 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:12.329 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:48:12.329 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:12.329 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:48:12.329 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:48:12.329 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:12.329 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:48:12.329 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:12.330 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:12.330 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:48:12.330 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:16.331 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:48:16.332 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:48:16.332 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:48:16.333 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:16.333 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:48:16.333 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:16.333 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:48:16.333 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:48:16.333 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:16.333 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:48:16.334 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:16.334 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:16.334 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:48:16.334 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:20.379 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 00:48:20.382 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:48:20.386 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:48:20.387 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:48:20.387 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:20.387 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:48:20.387 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:20.387 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:48:20.388 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:48:20.388 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:20.388 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:48:20.388 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:20.388 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:20.388 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:48:20.389 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:20.587 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:20.587 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. database_usage = [ << disk_space_used_1k = 6667780 disk_space_available_1k = 74422704 config_db_size_1k = 1696 >>, ] >> 03/20/2025 00:48:20.587 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:21.955 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 00:48:24.388 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:48:24.390 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:48:24.390 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:48:24.390 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:24.390 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:48:24.390 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:24.390 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:48:24.391 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:48:24.391 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:24.391 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:48:24.391 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:24.391 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:24.391 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:48:24.391 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:26.983 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=6667940, partition_space_available_1k=74422544, percentage_partition_space_used=8, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1715060, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=78256, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52100, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=3084400, free=6741928, buffers=234276, cached=6115024, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.59, five_min_avg=0.78, fifteen_min_avg=0.41, cpu_share=3.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 00:48:26.984 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:26.984 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 6667940 partition_space_available_1k = 74422544 percentage_partition_space_used = 8 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1715060 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 78256 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 52100 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 16175628 used = 3084400 free = 6741928 buffers = 234276 cached = 6115024 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 0.59 five_min_avg = 0.78 fifteen_min_avg = 0.41 cpu_share = 3.25 node_type = config-database-node >> >> 03/20/2025 00:48:26.984 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:28.391 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:48:28.393 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:48:28.393 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:48:28.393 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:28.393 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:48:28.393 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:28.393 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:48:28.393 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:48:28.394 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:28.394 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:48:28.394 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:28.394 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:28.394 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:48:28.394 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:32.395 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:48:32.397 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:48:32.397 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:48:32.397 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:32.397 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:48:32.397 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:32.397 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:48:32.398 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:48:32.398 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:32.398 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:48:32.398 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:32.398 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:32.398 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:48:32.398 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:36.397 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:48:36.398 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:48:36.399 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:48:36.399 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:36.399 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:48:36.399 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:36.399 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:48:36.399 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:48:36.400 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:36.400 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:48:36.400 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:36.400 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:36.400 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:48:36.400 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:40.400 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:48:40.402 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:48:40.402 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:48:40.402 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:40.402 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:48:40.402 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:40.402 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:48:40.403 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:48:40.403 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:40.403 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:48:40.403 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:40.403 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:40.403 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:48:40.403 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:44.403 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:48:44.404 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:48:44.404 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:48:44.404 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:44.404 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:48:44.405 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:44.405 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:48:44.405 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:48:44.405 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:44.405 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:48:44.405 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:44.405 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:44.406 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:48:44.406 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:48.408 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:48:48.410 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:48:48.410 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:48:48.411 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:48.411 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:48:48.411 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:48.411 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:48:48.411 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:48:48.412 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:48.412 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:48:48.412 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:48.412 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:48.412 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:48:48.412 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:52.411 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:48:52.412 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:48:52.413 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:48:52.413 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:52.413 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:48:52.414 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:52.414 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:48:52.415 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:48:52.415 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:52.415 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:48:52.415 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:52.415 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:52.415 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:48:52.416 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:56.416 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:48:56.417 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:48:56.417 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:48:56.417 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:56.417 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:48:56.417 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:48:56.418 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:48:56.418 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:48:56.418 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:56.418 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:48:56.418 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:56.418 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:48:56.418 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:48:56.418 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:00.419 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:49:00.420 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:49:00.421 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:49:00.421 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:00.421 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:49:00.421 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:00.421 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:49:00.421 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:49:00.421 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:00.421 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:49:00.422 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:00.422 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:00.422 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:49:00.422 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:04.423 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:49:04.424 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:49:04.424 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:49:04.425 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:04.425 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:49:04.425 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:04.425 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:49:04.425 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:49:04.425 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:04.425 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:49:04.426 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:04.426 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:04.426 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:49:04.426 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:08.427 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:49:08.428 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:49:08.428 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:49:08.428 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:08.428 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:49:08.429 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:08.429 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:49:08.429 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:49:08.429 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:08.429 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:49:08.429 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:08.429 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:08.429 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:49:08.430 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:12.430 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:49:12.432 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:49:12.432 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:49:12.432 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:12.432 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:49:12.433 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:12.433 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:49:12.433 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:49:12.433 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:12.433 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:49:12.433 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:12.433 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:12.433 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:49:12.433 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:16.436 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:49:16.437 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:49:16.438 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:49:16.438 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:16.438 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:49:16.438 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:16.438 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:49:16.439 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:49:16.439 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:16.439 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:49:16.439 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:16.439 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:16.439 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:49:16.439 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:20.375 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 00:49:20.440 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:49:20.443 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:49:20.443 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:49:20.443 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:20.444 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:49:20.444 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:20.444 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:49:20.444 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:49:20.444 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:20.444 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:49:20.444 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:20.444 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:20.444 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:49:20.444 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:20.638 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:20.638 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. database_usage = [ << disk_space_used_1k = 6681124 disk_space_available_1k = 74409360 config_db_size_1k = 1696 >>, ] >> 03/20/2025 00:49:20.639 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:22.066 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 00:49:24.444 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:49:24.446 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 00:49:24.446 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:49:24.446 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:24.446 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:49:24.446 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:24.447 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 00:49:24.447 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 00:49:24.447 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:24.447 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 00:49:24.447 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:24.447 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:24.447 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 00:49:24.447 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:27.101 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=6681132, partition_space_available_1k=74409352, percentage_partition_space_used=8, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1715832, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=78444, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52248, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=4515448, free=5279016, buffers=239348, cached=6141816, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.94, five_min_avg=0.88, fifteen_min_avg=0.47, cpu_share=4.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 00:49:27.102 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:27.102 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 6681132 partition_space_available_1k = 74409352 percentage_partition_space_used = 8 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1715832 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 78444 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 52248 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 16175628 used = 4515448 free = 5279016 buffers = 239348 cached = 6141816 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 0.94 five_min_avg = 0.88 fifteen_min_avg = 0.47 cpu_share = 4.75 node_type = config-database-node >> >> 03/20/2025 00:49:27.102 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 00:49:28.449 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 00:49:28.450 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Session Event: TCP Connected 03/20/2025 00:49:28.451 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 00:49:28.451 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:28.451 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 00:49:28.451 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 00:49:28.451 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 03/20/2025 00:49:28.451 7faa7c726348 [contrail-config-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 03/20/2025 00:49:28.451 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 03/20/2025 00:49:28.452 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 00:49:28.452 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 00:49:40.056 7faa7c726048 [contrail-config-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001016 03/20/2025 00:49:40.056 7faa7c726048 [contrail-config-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 03/20/2025 00:49:40.057 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 03/20/2025 00:49:40.057 7faa7c726348 [contrail-config-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 03/20/2025 00:49:40.057 7faa7c726348 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:DatabaseUsage] map 03/20/2025 00:49:40.057 7faa7c726348 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/20/2025 00:49:40.057 7faa7c726348 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/20/2025 00:49:40.058 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 03/20/2025 00:49:40.058 7faa7c8a5e48 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/20/2025 00:49:40.059 7faa7c8a5e48 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/20/2025 00:49:50.814 7faa7c726048 [contrail-config-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001439 03/20/2025 00:49:50.815 7faa7c726048 [contrail-config-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 03/20/2025 00:49:50.815 7faa7c8a5e48 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 03/20/2025 00:49:50.815 7faa7c8a5e48 [contrail-config-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 3 03/20/2025 00:49:50.815 7faa7c8a5e48 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:DatabaseUsage] map 03/20/2025 00:49:50.816 7faa7c8a5e48 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/20/2025 00:49:50.816 7faa7c8a5e48 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/20/2025 00:49:50.816 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/20/2025 00:49:50.816 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/20/2025 00:50:20.496 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 00:50:22.720 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 00:50:27.747 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7108120, partition_space_available_1k=73982364, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1720912, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=158820, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52468, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5598296, free=3702356, buffers=261408, cached=6613568, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.99, five_min_avg=1.22, fifteen_min_avg=0.61, cpu_share=10.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 00:51:20.447 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 00:51:22.462 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 00:51:27.490 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7171112, partition_space_available_1k=73919372, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1720156, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=162948, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52532, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5612184, free=3621548, buffers=262352, cached=6679544, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.22, five_min_avg=1.45, fifteen_min_avg=0.73, cpu_share=10.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 00:52:16.209 7faa7c726348 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/20/2025 00:52:20.445 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 00:52:22.247 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 00:52:27.272 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7646808, partition_space_available_1k=73443676, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1723472, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=148644, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52580, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5627620, free=3113924, buffers=268724, cached=7165360, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.48, five_min_avg=1.67, fifteen_min_avg=0.86, cpu_share=11.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 00:52:59.197 7faa7c726348 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/20/2025 00:53:20.442 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 00:53:22.387 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 00:53:27.413 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7660732, partition_space_available_1k=73429752, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1772544, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=139028, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52616, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5674312, free=3041136, buffers=270600, cached=7189580, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.54, five_min_avg=1.84, fifteen_min_avg=0.97, cpu_share=11.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 00:54:20.465 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 00:54:22.404 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 00:54:27.437 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7673920, partition_space_available_1k=73416564, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1817832, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=138452, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52648, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5741780, free=2968016, buffers=272192, cached=7193640, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.43, five_min_avg=2.18, fifteen_min_avg=1.14, cpu_share=11.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 00:55:20.459 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 00:55:22.360 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 00:55:27.386 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7668500, partition_space_available_1k=73421984, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1876640, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=138640, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52668, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5829320, free=2874376, buffers=273668, cached=7198264, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.31, five_min_avg=2.42, fifteen_min_avg=1.29, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 00:56:20.527 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 00:56:22.527 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 00:56:27.555 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7686780, partition_space_available_1k=73403704, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1895308, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=142316, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52688, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5866020, free=2827400, buffers=274644, cached=7207564, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.15, five_min_avg=2.29, fifteen_min_avg=1.32, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 00:57:20.520 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 00:57:22.553 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 00:57:27.586 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7690312, partition_space_available_1k=73400172, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1911340, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=146012, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52696, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5860888, free=2818116, buffers=275532, cached=7221092, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.67, five_min_avg=2.44, fifteen_min_avg=1.44, cpu_share=10.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 00:58:20.477 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 00:58:22.377 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 00:58:27.401 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7705180, partition_space_available_1k=73385304, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1935844, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=150280, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52732, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5911728, free=2760700, buffers=275944, cached=7227256, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.08, five_min_avg=2.36, fifteen_min_avg=1.48, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 00:59:20.485 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 00:59:22.393 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 00:59:27.429 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7710820, partition_space_available_1k=73379664, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1967664, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=154036, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52728, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5931040, free=2729424, buffers=276304, cached=7238860, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.0, five_min_avg=2.27, fifteen_min_avg=1.5, cpu_share=10.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:00:20.529 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:00:22.717 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:00:27.749 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7722604, partition_space_available_1k=73367880, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1996412, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=157564, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52736, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5971544, free=2681920, buffers=276572, cached=7245592, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.94, five_min_avg=2.57, fifteen_min_avg=1.65, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:01:20.533 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:01:22.420 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:01:27.454 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7731628, partition_space_available_1k=73358856, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2029776, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=160972, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52744, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6031700, free=2612312, buffers=276756, cached=7254860, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.44, five_min_avg=2.76, fifteen_min_avg=1.78, cpu_share=11.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:02:20.601 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:02:22.832 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:02:27.860 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7741148, partition_space_available_1k=73349336, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2059368, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=164360, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52748, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6063912, free=2569256, buffers=276928, cached=7265532, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.44, five_min_avg=2.6, fifteen_min_avg=1.79, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:03:20.556 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:03:22.692 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:03:27.721 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7750744, partition_space_available_1k=73339740, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2113928, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=168328, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52752, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6118284, free=2504624, buffers=277020, cached=7275700, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.68, five_min_avg=2.62, fifteen_min_avg=1.85, cpu_share=11.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:04:20.606 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:04:22.741 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:04:27.775 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7755616, partition_space_available_1k=73334868, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2160004, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=171848, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52752, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6147200, free=2465220, buffers=277104, cached=7286104, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.47, five_min_avg=2.58, fifteen_min_avg=1.88, cpu_share=11.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:05:20.558 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:05:22.992 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:05:28.020 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7764904, partition_space_available_1k=73325580, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2179192, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=175320, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52756, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6182340, free=2415360, buffers=277140, cached=7300788, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.79, five_min_avg=2.64, fifteen_min_avg=1.95, cpu_share=11.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:06:20.575 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:06:22.890 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:06:27.919 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7767540, partition_space_available_1k=73322944, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2192272, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=178856, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52760, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6197120, free=2397476, buffers=277164, cached=7303868, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.47, five_min_avg=2.59, fifteen_min_avg=1.98, cpu_share=11.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:07:20.563 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:07:22.841 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:07:27.869 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7782352, partition_space_available_1k=73308132, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2192840, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=182672, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52760, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6208912, free=2370636, buffers=277164, cached=7318916, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.14, five_min_avg=2.48, fifteen_min_avg=1.99, cpu_share=11.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:08:20.575 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:08:22.716 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:08:27.747 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7786672, partition_space_available_1k=73303812, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2194376, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=186208, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52768, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6194532, free=2385400, buffers=277184, cached=7318512, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.69, five_min_avg=2.57, fifteen_min_avg=2.04, cpu_share=11.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:09:20.537 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:09:22.779 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:09:27.827 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7797888, partition_space_available_1k=73292596, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2196908, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=193088, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52768, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6214804, free=2358156, buffers=277224, cached=7325444, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.19, five_min_avg=2.45, fifteen_min_avg=2.04, cpu_share=11.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:10:20.612 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:10:22.835 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:10:27.866 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7784604, partition_space_available_1k=73305880, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2181960, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=196900, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52776, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6218532, free=2358324, buffers=277228, cached=7321544, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.59, five_min_avg=2.55, fifteen_min_avg=2.1, cpu_share=11.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:11:20.540 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:11:22.763 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:11:27.791 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7788140, partition_space_available_1k=73302344, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2186132, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=194620, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52780, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6236340, free=2341868, buffers=277244, cached=7320176, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.2, five_min_avg=2.69, fifteen_min_avg=2.17, cpu_share=11.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:12:20.549 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:12:22.436 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:12:27.468 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7801112, partition_space_available_1k=73289372, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2188768, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=198068, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52780, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6250964, free=2318780, buffers=277260, cached=7328624, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.61, five_min_avg=2.67, fifteen_min_avg=2.2, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:13:20.528 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:13:22.535 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:13:27.560 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7804708, partition_space_available_1k=73285776, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2192028, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=201932, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52784, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6252464, free=2308540, buffers=277264, cached=7337360, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.51, five_min_avg=2.63, fifteen_min_avg=2.21, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:14:20.540 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:14:22.609 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:14:27.644 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7810036, partition_space_available_1k=73280448, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2198360, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=206004, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52788, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6250784, free=2299164, buffers=277276, cached=7348404, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.47, five_min_avg=2.59, fifteen_min_avg=2.22, cpu_share=10.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:15:20.659 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:15:22.690 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:15:27.716 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7815292, partition_space_available_1k=73275192, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2203192, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=210600, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53216, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6263512, free=2281596, buffers=277284, cached=7353236, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.4, five_min_avg=2.57, fifteen_min_avg=2.24, cpu_share=10.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:16:20.552 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:16:22.633 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:16:27.665 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7821372, partition_space_available_1k=73269112, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2199528, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=214300, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53240, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6243532, free=2295172, buffers=277288, cached=7359636, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.4, five_min_avg=2.56, fifteen_min_avg=2.26, cpu_share=10.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:17:20.631 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:17:22.768 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:17:27.797 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7829216, partition_space_available_1k=73261268, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2214916, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=218180, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53236, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6268476, free=2267028, buffers=277304, cached=7362820, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.62, five_min_avg=2.61, fifteen_min_avg=2.3, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:18:20.635 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:18:22.919 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:18:27.952 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7838204, partition_space_available_1k=73252280, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2199888, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=222144, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53248, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6278328, free=2252804, buffers=277308, cached=7367188, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.86, five_min_avg=2.39, fifteen_min_avg=2.24, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:19:20.613 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:19:22.863 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:19:27.903 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7845204, partition_space_available_1k=73245280, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2199924, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=226004, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53244, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6289052, free=2235112, buffers=277312, cached=7374152, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.5, five_min_avg=2.46, fifteen_min_avg=2.28, cpu_share=11.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:20:20.622 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:20:22.879 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:20:27.914 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7852724, partition_space_available_1k=73237760, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2205132, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=230296, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53252, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6292540, free=2218976, buffers=277312, cached=7386800, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.79, five_min_avg=2.56, fifteen_min_avg=2.32, cpu_share=11.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:21:20.656 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:21:22.937 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:21:27.964 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7856516, partition_space_available_1k=73233968, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2212052, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=234200, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53256, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6285368, free=2222132, buffers=277320, cached=7390808, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.75, five_min_avg=2.64, fifteen_min_avg=2.37, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:22:20.660 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:22:22.691 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:22:27.715 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7860260, partition_space_available_1k=73230224, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2202696, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=238264, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53256, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6289676, free=2208668, buffers=277324, cached=7399960, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.6, five_min_avg=2.65, fifteen_min_avg=2.39, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:23:20.651 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:23:22.793 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:23:27.822 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7865588, partition_space_available_1k=73224896, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2202308, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=242184, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53264, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6272200, free=2221044, buffers=277324, cached=7405060, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.73, five_min_avg=2.72, fifteen_min_avg=2.43, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:24:20.672 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:24:22.504 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:24:27.537 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7869596, partition_space_available_1k=73220888, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2203076, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=246160, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53264, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6293072, free=2195552, buffers=277324, cached=7409680, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.05, five_min_avg=2.79, fifteen_min_avg=2.47, cpu_share=10.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:25:20.686 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:25:22.898 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:25:27.925 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7881124, partition_space_available_1k=73209360, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2205592, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=250632, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53272, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6308480, free=2173648, buffers=277328, cached=7416172, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.81, five_min_avg=2.82, fifteen_min_avg=2.51, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:26:20.681 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:26:22.697 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:26:27.724 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7885984, partition_space_available_1k=73204500, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2205068, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=254612, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53272, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6302844, free=2174060, buffers=277328, cached=7421396, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.56, five_min_avg=2.73, fifteen_min_avg=2.49, cpu_share=10.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:27:20.731 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:27:22.924 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:27:27.956 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7892164, partition_space_available_1k=73198320, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2206508, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=261552, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53280, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6310164, free=2160404, buffers=277328, cached=7427732, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.36, five_min_avg=2.66, fifteen_min_avg=2.48, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:28:20.687 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:28:23.057 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:28:28.087 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7902372, partition_space_available_1k=73188112, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2205136, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=265720, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53280, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6313128, free=2151796, buffers=277328, cached=7433376, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.45, five_min_avg=2.6, fifteen_min_avg=2.47, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:29:20.668 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:29:22.635 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:29:27.669 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7924192, partition_space_available_1k=73166292, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2206164, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=270100, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53284, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6281736, free=2165328, buffers=277336, cached=7451228, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.04, five_min_avg=2.46, fifteen_min_avg=2.43, cpu_share=11.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:30:20.724 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:30:22.905 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:30:27.934 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7930308, partition_space_available_1k=73160176, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2207972, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=274152, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53288, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6283296, free=2157760, buffers=277340, cached=7457232, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.54, five_min_avg=2.52, fifteen_min_avg=2.45, cpu_share=10.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:31:20.708 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:31:22.663 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:31:27.692 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7936148, partition_space_available_1k=73154336, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2208352, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=278308, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53292, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6286016, free=2144148, buffers=277340, cached=7468124, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.63, five_min_avg=2.54, fifteen_min_avg=2.47, cpu_share=10.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:32:20.852 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:32:22.920 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:32:27.949 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7942016, partition_space_available_1k=73148468, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2208940, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=282260, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53296, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6285644, free=2133640, buffers=277340, cached=7479004, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.59, five_min_avg=2.54, fifteen_min_avg=2.47, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:33:20.773 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:33:22.882 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:33:27.912 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7947296, partition_space_available_1k=73143188, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2208860, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=286312, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53300, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6297844, free=2116064, buffers=277344, cached=7484376, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.9, five_min_avg=2.37, fifteen_min_avg=2.41, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:34:20.726 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:34:22.812 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:34:27.852 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7952008, partition_space_available_1k=73138476, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2209352, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=290432, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53300, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6295856, free=2112840, buffers=277348, cached=7489584, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.2, five_min_avg=2.34, fifteen_min_avg=2.4, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:35:20.732 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:35:22.946 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:35:27.977 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7961852, partition_space_available_1k=73128632, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2211560, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=294256, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53308, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6304792, free=2094388, buffers=277348, cached=7499100, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.61, five_min_avg=2.42, fifteen_min_avg=2.42, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:36:20.728 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:36:22.768 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:36:27.804 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7965640, partition_space_available_1k=73124844, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2211072, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=298740, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53308, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6308404, free=2085900, buffers=277348, cached=7503976, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.44, five_min_avg=2.38, fifteen_min_avg=2.4, cpu_share=10.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:37:20.709 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:37:22.868 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:37:27.894 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7969928, partition_space_available_1k=73120556, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2212632, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=302856, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53316, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6317992, free=2072508, buffers=277352, cached=7507776, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.4, five_min_avg=2.4, fifteen_min_avg=2.41, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:38:20.747 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:38:22.906 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:38:27.931 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7974056, partition_space_available_1k=73116428, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2211568, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=306836, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53316, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6322316, free=2059264, buffers=277352, cached=7516696, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.19, five_min_avg=2.33, fifteen_min_avg=2.38, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:39:20.741 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:39:22.745 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:39:27.782 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7983800, partition_space_available_1k=73106684, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2212628, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=310852, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53320, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6316892, free=2058860, buffers=277352, cached=7522524, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.57, five_min_avg=2.43, fifteen_min_avg=2.41, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:40:20.721 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:40:22.982 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:40:28.010 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7989872, partition_space_available_1k=73100612, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2214760, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=315192, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53324, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6327068, free=2042960, buffers=277352, cached=7528248, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.49, five_min_avg=2.41, fifteen_min_avg=2.41, cpu_share=10.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:41:20.772 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:41:23.002 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:41:28.036 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7999780, partition_space_available_1k=73090704, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2215184, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=319252, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53328, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6327700, free=2037208, buffers=277352, cached=7533368, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.34, five_min_avg=2.38, fifteen_min_avg=2.4, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:42:20.770 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:42:23.116 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:42:28.143 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8009004, partition_space_available_1k=73081480, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2215808, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=323248, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53332, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6335440, free=2020212, buffers=277352, cached=7542624, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.17, five_min_avg=2.31, fifteen_min_avg=2.37, cpu_share=10.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:43:20.769 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:43:23.071 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:43:28.100 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8012724, partition_space_available_1k=73077760, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2215508, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=327188, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53336, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6337244, free=2014156, buffers=277352, cached=7546876, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.22, five_min_avg=2.33, fifteen_min_avg=2.37, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:44:20.775 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:44:23.259 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:44:28.300 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8017556, partition_space_available_1k=73072928, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2216252, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=331256, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53344, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6340468, free=2000972, buffers=277352, cached=7556836, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.46, five_min_avg=2.36, fifteen_min_avg=2.38, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:45:20.748 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:45:23.008 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:45:28.035 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8024360, partition_space_available_1k=73066124, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2189928, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=335764, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53356, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6307412, free=2026916, buffers=277372, cached=7563928, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.37, five_min_avg=2.34, fifteen_min_avg=2.37, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:46:20.814 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:46:23.275 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:46:28.303 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8029032, partition_space_available_1k=73061452, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2202124, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=339816, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53356, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6326528, free=1997828, buffers=277372, cached=7573900, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.37, five_min_avg=2.38, fifteen_min_avg=2.38, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:47:20.822 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:47:23.211 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:47:28.240 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8037548, partition_space_available_1k=73052936, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2202444, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=343780, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53360, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6330844, free=1990124, buffers=277372, cached=7577288, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.13, five_min_avg=2.31, fifteen_min_avg=2.35, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:48:20.842 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:48:23.015 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:48:28.052 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8054104, partition_space_available_1k=73036380, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2202428, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=347660, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53364, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6268620, free=2035484, buffers=277380, cached=7594144, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.06, five_min_avg=2.24, fifteen_min_avg=2.33, cpu_share=11.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:49:20.820 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:49:23.154 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:49:28.206 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8061832, partition_space_available_1k=73028652, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2202428, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=351584, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53368, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6268228, free=2027860, buffers=277396, cached=7602144, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.93, five_min_avg=2.16, fifteen_min_avg=2.29, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:50:20.825 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:50:23.107 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:50:28.134 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8070136, partition_space_available_1k=73020348, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2204876, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=356052, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53372, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6259964, free=2032196, buffers=277400, cached=7606068, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.39, five_min_avg=2.23, fifteen_min_avg=2.3, cpu_share=11.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:51:20.860 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:51:23.240 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:51:28.266 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8071664, partition_space_available_1k=73018820, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2205104, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=359808, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53380, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6263800, free=2017320, buffers=277400, cached=7617108, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.68, five_min_avg=2.29, fifteen_min_avg=2.32, cpu_share=11.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:52:20.874 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:52:23.370 7faa820aadf0 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:52:28.399 7faa820aadf0 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8081620, partition_space_available_1k=73008864, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=2205080, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=363116, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53384, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6327372, free=1948972, buffers=277404, cached=7621880, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.95, five_min_avg=2.37, fifteen_min_avg=2.34, cpu_share=10.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:52:46.844 7faa7c726048 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 03/20/2025 01:52:46.844 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[Established] 03/20/2025 01:52:46.846 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Session Event: TCP Connected 03/20/2025 01:52:46.846 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[Established] -> State[Connect] 03/20/2025 01:52:46.846 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:52:46.846 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Established to Connect on EvTcpClose >>, ] description = Collector:Collector[Established to Connect on EvTcpClose] connection down >>, ] >> 03/20/2025 01:52:46.846 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:52:46.846 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 03/20/2025 01:52:46.847 7faa7c726348 [contrail-config-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 03/20/2025 01:52:46.847 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 03/20/2025 01:52:46.847 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:52:46.847 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:52:46.879 7faa7c726048 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 03/20/2025 01:52:46.880 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 03/20/2025 01:52:46.880 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 03/20/2025 01:52:46.880 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:46.881 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 03/20/2025 01:52:46.881 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:46.881 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:46.881 7faa7c726348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 2 pid = 1 http_port = 8112 start_time = 1742431459951388 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:52:46.881 7faa7c726348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:57.123 7feb4290ff20 [contrail-config-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 03/20/2025 01:52:57.124 7feb4290ff20 [contrail-config-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 03/20/2025 01:52:57.124 7feb4290ff20 [contrail-config-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 03/20/2025 01:52:57.139 7feb3cafd648 [contrail-config-database-nodemgr] [ERROR]: Cannot write http_port 8112 to /tmp/contrail-config-database-nodemgr.0.http_port 03/20/2025 01:52:57.143 7feb3cafd648 [contrail-config-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8112 03/20/2025 01:52:57.147 7feb3cafde48 [contrail-config-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 03/20/2025 01:52:57.151 7feb3cafde48 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/20/2025 01:52:57.151 7feb3cafde48 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/20/2025 01:52:57.151 7feb3cafde48 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 03/20/2025 01:52:57.152 7feb3cafde48 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:57.152 7feb3cafde48 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/20/2025 01:52:57.152 7feb3cafde48 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:57.161 7feb3cafde48 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:57.162 7feb3cafde48 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 03/20/2025 01:52:57.162 7feb3cafde48 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:57.798 7feb4290ff20 [contrail-config-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 10873332665642725122558304532124286267164827865277466652510538593036487547962, 'start': '1742435569000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'zookeeper', 'group': 'zookeeper', 'pid': 5057221750418482296835987726371120978093503601993300154265813374933025949962, 'start': '1742435569000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-config-database-nodemgr', 'group': 'contrail-config-database-nodemgr', 'pid': 53321400583668198283938383852360463026999483032090278927571254795566302906205, 'start': '1742435569000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 03/20/2025 01:52:58.307 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:58.308 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. build_info = {"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-17 01:12:14.132392", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8139-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 4 num_core_per_socket = 4 num_thread_per_core = 1 >> running_package_version = nightly.r24.1.centos-nightly_r24.1_centos.el7 installed_package_version = nightly.r24.1.centos-nightly_r24.1_centos.el7 >> 03/20/2025 01:52:58.308 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:58.308 7feb4290ff20 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1742435578308762, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/20/2025 01:52:58.309 7feb4290ff20 [contrail-config-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1742435578309097, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742435569000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='zookeeper', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742435569000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-config-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742435569000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-17 01:12:14.132392", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8139-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/20/2025 01:52:58.311 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:58.312 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/20/2025 01:52:58.312 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:58.312 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:58.312 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742435569000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = zookeeper process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742435569000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-config-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742435569000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-17 01:12:14.132392", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8139-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]} >> 03/20/2025 01:52:58.312 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:53:00.052 7feb4290ff20 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:53:01.078 7feb4290ff20 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:DatabaseUsage] map 03/20/2025 01:53:01.081 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:53:01.087 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. database_usage = [ << disk_space_used_1k = 7919184 disk_space_available_1k = 73171300 config_db_size_1k = 4048 >>, ] >> 03/20/2025 01:53:01.087 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:53:01.151 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:53:01.159 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Session Event: TCP Connected 03/20/2025 01:53:01.159 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:53:01.159 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:53:01.159 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:53:01.159 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:53:01.159 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 03/20/2025 01:53:01.160 7feb3b94b348 [contrail-config-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 03/20/2025 01:53:01.160 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 03/20/2025 01:53:01.160 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:53:01.160 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:53:03.207 7feb3b913348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 03/20/2025 01:53:03.207 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 03/20/2025 01:53:03.208 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 03/20/2025 01:53:03.208 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:53:03.208 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 03/20/2025 01:53:03.208 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:53:03.208 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:53:03.208 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 01:53:03.208 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:53:03.706 7feb4290ff20 [contrail-config-database-nodemgr] [ERROR]: Failed to connect to database by CQL: Result: 1 03/20/2025 01:53:03.706 7feb4290ff20 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1742435583706455, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.47:8086'], status='Down', description='ClientInit to Idle on EvTcpClose')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[ClientInit to Idle on EvTcpClose] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/20/2025 01:53:03.715 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:53:03.716 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Cassandra state detected DOWN. Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 03/20/2025 01:53:03.716 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:53:07.208 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:53:07.210 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Session Event: TCP Connected 03/20/2025 01:53:07.210 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:53:07.210 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:53:07.210 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:53:07.210 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:53:07.210 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 03/20/2025 01:53:07.211 7feb3b94b348 [contrail-config-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 03/20/2025 01:53:07.211 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 03/20/2025 01:53:07.211 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:53:07.211 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:53:07.232 7feb3b913348 [contrail-config-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001017 03/20/2025 01:53:07.233 7feb3b913348 [contrail-config-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 03/20/2025 01:53:07.234 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 03/20/2025 01:53:07.234 7feb3b94b348 [contrail-config-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 03/20/2025 01:53:07.235 7feb3b94b348 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:DatabaseUsage] map 03/20/2025 01:53:07.235 7feb3b94b348 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/20/2025 01:53:07.236 7feb3b94b348 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/20/2025 01:53:07.236 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 03/20/2025 01:53:07.236 7feb3b94b048 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/20/2025 01:53:07.238 7feb3b94b048 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/20/2025 01:53:13.426 7feb4290ff20 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:53:24.470 7feb4290ff20 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr config_db_uuid'. Error is: Result: 1 03/20/2025 01:53:34.803 7feb4290ff20 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr to_bgp_keyspace'. Error is: timed out 03/20/2025 01:53:39.623 7feb4290ff20 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr svc_monitor_keyspace'. Error is: Result: 1 03/20/2025 01:53:42.607 7feb4290ff20 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr DISCOVERY_SERVER'. Error is: Result: 1 03/20/2025 01:53:49.506 7feb4290ff20 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr useragent'. Error is: Result: 1 03/20/2025 01:53:57.867 7feb4290ff20 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr dm_keyspace'. Error is: Result: 1 03/20/2025 01:54:03.211 7feb4290ff20 [contrail-config-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1742435643211828, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742435569000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='zookeeper', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742435569000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-config-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742435569000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-17 01:12:14.132392", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8139-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/20/2025 01:54:03.212 7feb4290ff20 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8840376, partition_space_available_1k=72250108, percentage_partition_space_used=11, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1330772, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=250628, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44616, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5226716, free=2136228, buffers=315948, cached=8496736, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=17.93, five_min_avg=7.62, fifteen_min_avg=4.21, cpu_share=0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:54:03.212 7feb4290ff20 [contrail-config-database-nodemgr] [ERROR]: function do_periodic_events duration exceeded 60.000000 interval (took 64.903207) 03/20/2025 01:54:03.744 7feb4290ff20 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:54:05.883 7feb4290ff20 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1742435645883601, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.12:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/20/2025 01:54:11.742 7feb4290ff20 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:54:16.802 7feb4290ff20 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=9015244, partition_space_available_1k=72075240, percentage_partition_space_used=11, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1308480, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=356348, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44628, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5386636, free=1714964, buffers=343016, cached=8731012, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=17.21, five_min_avg=7.98, fifteen_min_avg=4.38, cpu_share=22.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:55:03.835 7feb4290ff20 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:55:08.252 7feb4290ff20 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:55:13.937 7feb4290ff20 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=9983056, partition_space_available_1k=71107428, percentage_partition_space_used=12, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1684816, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=432312, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44632, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5787408, free=230712, buffers=387660, cached=9769848, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=10.76, five_min_avg=7.69, fifteen_min_avg=4.49, cpu_share=17.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:55:46.185 7feb3b913348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 03/20/2025 01:55:46.185 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[Established] 03/20/2025 01:55:46.189 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Session Event: TCP Connected 03/20/2025 01:55:46.189 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[Established] -> State[Connect] 03/20/2025 01:55:46.189 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:55:46.190 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Established to Connect on EvTcpClose >>, ] description = Collector:Collector[Established to Connect on EvTcpClose] connection down >>, ] >> 03/20/2025 01:55:46.190 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:55:46.190 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 03/20/2025 01:55:46.190 7feb3b94b348 [contrail-config-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 03/20/2025 01:55:46.191 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 03/20/2025 01:55:46.191 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:55:46.191 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:55:46.218 7feb3b913348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 03/20/2025 01:55:46.218 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 03/20/2025 01:55:46.219 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 03/20/2025 01:55:46.219 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:55:46.219 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 03/20/2025 01:55:46.220 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:55:46.220 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:55:46.220 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:55:46.221 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:55:50.220 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:55:50.222 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Session Event: TCP Connected 03/20/2025 01:55:50.222 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:55:50.222 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:55:50.222 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:55:50.222 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:55:50.222 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 03/20/2025 01:55:50.223 7feb3b94b348 [contrail-config-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 03/20/2025 01:55:50.223 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 03/20/2025 01:55:50.223 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:55:50.223 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:55:58.780 7feb3b913348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 03/20/2025 01:55:58.780 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 03/20/2025 01:55:58.781 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 03/20/2025 01:55:58.781 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:55:58.781 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 03/20/2025 01:55:58.781 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:55:58.781 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:55:58.781 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:55:58.781 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:02.781 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:02.782 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:02.782 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:02.782 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:02.783 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:02.783 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:02.783 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:02.783 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:02.783 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:02.783 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:02.784 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:02.784 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:02.784 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:02.784 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:03.468 7feb4290ff20 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:56:03.782 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:03.782 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. database_usage = [ << disk_space_used_1k = 10342908 disk_space_available_1k = 70747576 config_db_size_1k = 3724 >>, ] >> 03/20/2025 01:56:03.782 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:05.540 7feb4290ff20 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:56:06.783 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:06.787 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:06.787 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:06.787 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:06.788 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:06.788 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:06.788 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:06.789 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:06.789 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:06.789 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:06.789 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:06.789 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:06.789 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:06.791 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:10.787 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:10.789 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:10.790 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:10.790 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:10.790 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:10.790 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:10.790 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:10.791 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:10.791 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:10.791 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:10.791 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:10.792 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:10.792 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:10.792 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:11.358 7feb4290ff20 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10344352, partition_space_available_1k=70746132, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1690316, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=435940, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44680, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=4953884, free=1037364, buffers=387392, cached=9796988, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.3, five_min_avg=7.04, fifteen_min_avg=4.45, cpu_share=10.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:56:11.358 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:11.359 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 10344352 partition_space_available_1k = 70746132 percentage_partition_space_used = 13 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1690316 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 435940 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 44680 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 16175628 used = 4953884 free = 1037364 buffers = 387392 cached = 9796988 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 6.3 five_min_avg = 7.04 fifteen_min_avg = 4.45 cpu_share = 10.0 node_type = config-database-node >> >> 03/20/2025 01:56:11.359 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:14.791 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:14.792 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:14.793 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:14.793 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:14.793 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:14.793 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:14.793 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:14.793 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:14.793 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:14.794 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:14.794 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:14.794 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:14.794 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:14.794 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:18.793 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:18.795 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:18.795 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:18.795 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:18.796 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:18.796 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:18.796 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:18.796 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:18.796 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:18.796 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:18.796 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:18.796 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:18.797 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:18.797 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:22.799 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:22.801 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:22.801 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:22.801 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:22.801 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:22.801 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:22.802 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:22.802 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:22.802 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:22.802 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:22.802 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:22.802 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:22.802 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:22.803 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:26.804 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:26.806 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:26.806 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:26.806 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:26.806 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:26.807 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:26.807 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:26.807 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:26.807 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:26.807 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:26.808 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:26.808 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:26.808 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:26.808 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:30.807 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:30.810 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:30.810 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:30.810 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:30.810 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:30.810 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:30.811 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:30.811 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:30.811 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:30.811 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:30.812 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:30.812 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:30.812 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:30.812 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:34.812 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:34.813 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:34.813 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:34.814 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:34.814 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:34.814 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:34.814 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:34.815 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:34.815 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:34.815 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:34.815 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:34.815 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:34.815 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:34.815 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:38.815 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:38.816 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:38.816 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:38.816 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:38.816 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:38.817 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:38.817 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:38.817 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:38.817 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:38.817 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:38.818 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:38.818 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:38.818 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:38.818 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:42.819 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:42.821 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:42.821 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:42.821 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:42.821 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:42.821 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:42.821 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:42.822 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:42.822 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:42.822 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:42.822 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:42.822 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:42.822 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:42.822 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:46.823 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:46.825 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:46.825 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:46.825 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:46.826 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:46.826 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:46.826 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:46.826 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:46.826 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:46.826 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:46.827 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:46.827 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:46.827 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:46.827 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:50.827 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:50.829 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:50.829 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:50.829 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:50.829 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:50.829 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:50.829 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:50.830 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:50.830 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:50.830 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:50.830 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:50.830 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:50.830 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:50.830 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:54.831 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:54.832 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:54.833 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:54.833 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:54.833 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:54.833 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:54.833 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:54.833 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:54.833 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:54.834 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:54.834 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:54.834 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:54.834 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:54.834 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:58.835 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:56:58.837 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:56:58.837 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:56:58.837 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:58.837 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:56:58.837 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:56:58.838 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:56:58.838 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:56:58.838 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:58.838 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:56:58.838 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:58.838 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:56:58.839 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:56:58.839 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:02.838 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:02.840 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:57:02.840 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:02.841 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:02.841 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:02.843 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:02.843 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:57:02.844 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:57:02.846 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:02.847 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:57:02.847 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:02.848 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:02.848 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:57:02.848 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:03.668 7feb4290ff20 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:57:04.133 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:04.134 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. database_usage = [ << disk_space_used_1k = 10267692 disk_space_available_1k = 70822792 config_db_size_1k = 3724 >>, ] >> 03/20/2025 01:57:04.134 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:06.840 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:06.842 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:57:06.842 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:06.842 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:06.842 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:06.842 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:06.842 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:57:06.843 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:57:06.843 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:06.843 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:57:06.843 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:06.843 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:06.843 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:57:06.843 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:07.172 7feb4290ff20 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:57:10.842 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:10.844 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:57:10.844 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:10.845 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:10.845 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:10.845 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:10.845 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:57:10.845 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:57:10.845 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:10.846 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:57:10.846 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:10.846 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:10.846 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:57:10.846 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:12.763 7feb4290ff20 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10275648, partition_space_available_1k=70814836, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1689172, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=435980, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44760, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=4592320, free=1449432, buffers=392324, cached=9741552, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.58, five_min_avg=6.01, fifteen_min_avg=4.27, cpu_share=6.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:57:12.764 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:12.764 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 10275648 partition_space_available_1k = 70814836 percentage_partition_space_used = 13 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1689172 cpu_share = 0 mem_res = 0 >>zookeeper : << mem_virt = 435980 cpu_share = 0 mem_res = 0 >>contrail-config-database-nodemgr : << mem_virt = 44760 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 16175628 used = 4592320 free = 1449432 buffers = 392324 cached = 9741552 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 3.58 five_min_avg = 6.01 fifteen_min_avg = 4.27 cpu_share = 6.5 node_type = config-database-node >> >> 03/20/2025 01:57:12.764 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:14.846 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:14.848 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:57:14.848 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:14.848 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:14.848 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:14.848 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:14.848 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:57:14.849 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:57:14.849 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:14.849 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:57:14.849 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:14.849 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:14.849 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:57:14.849 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:18.849 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:18.853 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:57:18.853 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:18.854 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:18.854 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:18.854 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:18.854 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:57:18.855 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:57:18.855 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:18.855 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:57:18.855 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:18.855 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:18.855 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:57:18.855 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:22.854 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:22.855 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:57:22.856 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:22.856 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:22.856 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:22.856 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:22.856 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:57:22.856 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:57:22.857 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:22.857 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:57:22.858 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:22.858 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:22.858 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:57:22.858 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:26.856 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:26.857 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:57:26.857 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:26.858 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:26.858 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:26.858 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:26.858 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:57:26.859 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:57:26.859 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:26.859 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:57:26.859 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:26.859 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:26.860 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:57:26.860 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:30.859 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:30.864 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:57:30.864 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:30.864 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:30.865 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:30.865 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:30.865 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:57:30.866 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:57:30.866 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:30.866 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:57:30.866 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:30.866 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:30.866 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:57:30.866 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:34.866 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:34.868 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:57:34.868 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:34.868 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:34.868 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:34.868 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:34.868 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:57:34.869 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:57:34.869 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:34.869 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:57:34.869 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:34.869 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:34.869 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:57:34.869 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:38.871 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:38.872 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:57:38.872 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:38.873 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:38.873 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:38.873 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:38.873 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:57:38.874 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:57:38.874 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:38.874 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:57:38.874 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:38.874 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:38.874 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:57:38.874 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:42.875 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:42.877 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/20/2025 01:57:42.877 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:42.877 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:42.877 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:42.877 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:42.877 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/20/2025 01:57:42.877 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/20/2025 01:57:42.878 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:42.878 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/20/2025 01:57:42.878 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:42.878 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:42.878 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:57:42.878 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:46.879 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:46.880 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Session Event: TCP Connected 03/20/2025 01:57:46.881 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:46.881 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:46.881 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:46.882 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:46.882 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 03/20/2025 01:57:46.882 7feb3b94b348 [contrail-config-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 03/20/2025 01:57:46.883 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 03/20/2025 01:57:46.883 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:57:46.883 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:57:47.721 7feb3b913348 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 03/20/2025 01:57:47.722 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 03/20/2025 01:57:47.722 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 03/20/2025 01:57:47.722 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:47.722 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 03/20/2025 01:57:47.723 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:47.723 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:47.723 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8112 start_time = 1742435577151623 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.47:8086, 10.0.0.12:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 03/20/2025 01:57:47.723 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:57:51.724 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:57:51.725 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Session Event: TCP Connected 03/20/2025 01:57:51.725 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:57:51.725 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:51.725 7feb3b94b348 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 01:57:51.725 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:57:51.725 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 03/20/2025 01:57:51.725 7feb3b94b348 [contrail-config-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 03/20/2025 01:57:51.726 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 03/20/2025 01:57:51.726 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:57:51.726 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:57:51.737 7feb3b913348 [contrail-config-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001016 03/20/2025 01:57:51.738 7feb3b913348 [contrail-config-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 03/20/2025 01:57:51.738 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 03/20/2025 01:57:51.738 7feb3b94b348 [contrail-config-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 03/20/2025 01:57:51.739 7feb3b94b348 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:DatabaseUsage] map 03/20/2025 01:57:51.739 7feb3b94b348 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/20/2025 01:57:51.739 7feb3b94b348 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/20/2025 01:57:51.740 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 03/20/2025 01:57:51.740 7feb3b94b048 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/20/2025 01:57:51.741 7feb3b94b048 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/20/2025 01:58:03.617 7feb4290ff20 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:58:06.853 7feb4290ff20 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:58:08.749 7feb3b913348 [contrail-config-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001438 03/20/2025 01:58:08.751 7feb3b913348 [contrail-config-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 03/20/2025 01:58:08.755 7feb3b94b348 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 03/20/2025 01:58:08.757 7feb3b94b348 [contrail-config-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 3 03/20/2025 01:58:08.762 7feb3b94b348 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:DatabaseUsage] map 03/20/2025 01:58:08.764 7feb3b94b348 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/20/2025 01:58:08.764 7feb3b94b348 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/20/2025 01:58:08.765 7feb3b94b048 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/20/2025 01:58:08.765 7feb3b94b048 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/20/2025 01:58:12.808 7feb4290ff20 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10406360, partition_space_available_1k=70684124, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1691696, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=436724, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44912, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5906308, free=388364, buffers=396784, cached=9484172, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.24, five_min_avg=5.64, fifteen_min_avg=4.24, cpu_share=11.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 01:59:03.478 7feb4290ff20 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 01:59:05.588 7feb4290ff20 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 01:59:10.734 7feb4290ff20 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10831748, partition_space_available_1k=70258736, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1698404, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=436324, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=45088, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5906360, free=308260, buffers=406352, cached=9554656, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.54, five_min_avg=5.19, fifteen_min_avg=4.17, cpu_share=8.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:00:03.522 7feb4290ff20 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:00:05.602 7feb4290ff20 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:00:10.733 7feb4290ff20 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10845640, partition_space_available_1k=70244844, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1712984, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=436236, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=45096, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6002168, free=261104, buffers=406308, cached=9506048, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.03, five_min_avg=4.48, fifteen_min_avg=3.99, cpu_share=7.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:01:03.496 7feb4290ff20 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:01:05.559 7feb4290ff20 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:01:10.733 7feb4290ff20 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10844992, partition_space_available_1k=70245492, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1732432, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=435992, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=45100, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6074056, free=247700, buffers=405388, cached=9448484, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.97, five_min_avg=3.72, fifteen_min_avg=3.76, cpu_share=5.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:02:03.513 7feb4290ff20 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:02:05.841 7feb4290ff20 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:02:11.749 7feb4290ff20 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11295568, partition_space_available_1k=69794916, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1737716, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=435360, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=45100, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6120352, free=250980, buffers=413992, cached=9390304, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.35, five_min_avg=3.31, fifteen_min_avg=3.61, cpu_share=7.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:02:20.384 7feb3b913148 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_process_state:cassandra,group:cassandra,state:PROCESS_STATE_EXITED 03/20/2025 02:02:20.385 7feb3b913148 [contrail-config-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1742436140385051, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_EXITED', start_count=1, stop_count=0, exit_count=1, last_start_time='1742435569000000', last_stop_time='', last_exit_time='1742436140384982', core_file_list=[]), ProcessInfo(process_name='zookeeper', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742435569000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-config-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742435569000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-17 01:12:14.132392", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8139-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/20/2025 02:02:20.390 7feb3b913148 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_process_state:zookeeper,group:zookeeper,state:PROCESS_STATE_EXITED 03/20/2025 02:02:20.390 7feb3b913148 [contrail-config-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1742436140390680, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_EXITED', start_count=1, stop_count=0, exit_count=1, last_start_time='1742435569000000', last_stop_time='', last_exit_time='1742436140384982', core_file_list=[]), ProcessInfo(process_name='zookeeper', process_state='PROCESS_STATE_EXITED', start_count=1, stop_count=0, exit_count=1, last_start_time='1742435569000000', last_stop_time='', last_exit_time='1742436140390625', core_file_list=[]), ProcessInfo(process_name='contrail-config-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742435569000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-17 01:12:14.132392", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8139-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/20/2025 02:02:49.258 7f4b7cd0a648 [contrail-config-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 03/20/2025 02:02:49.258 7f4b7cd0a648 [contrail-config-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 03/20/2025 02:02:49.258 7f4b7cd0a648 [contrail-config-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 03/20/2025 02:02:49.273 7f4b79ddc6d8 [contrail-config-database-nodemgr] [ERROR]: Cannot write http_port 8112 to /tmp/contrail-config-database-nodemgr.0.http_port 03/20/2025 02:02:49.273 7f4b79ddc6d8 [contrail-config-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8112 03/20/2025 02:02:49.274 7f4b79de6048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 03/20/2025 02:02:49.275 7f4b79de6048 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/20/2025 02:02:49.276 7f4b79de6048 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/20/2025 02:02:49.276 7f4b79de6048 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 03/20/2025 02:02:49.276 7f4b79de6048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:02:49.276 7f4b79de6048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/20/2025 02:02:49.276 7f4b79de6048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:02:49.276 7f4b79de6048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:02:49.276 7f4b79de6048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2.:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1742436169275884 collector_name = collector_ip = collector_list = [ 10.0.0.42:8086, 10.0.0.47:8086, 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 03/20/2025 02:02:49.277 7f4b79de6048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:02:49.528 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 84647635515795833309694210071645913935051325243430599997319332756686105461129, 'start': '1742436165000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'zookeeper', 'group': 'zookeeper', 'pid': 43071756843141314609064596208870132913500355993397075270507544513155487419466, 'start': '1742436165000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-config-database-nodemgr', 'group': 'contrail-config-database-nodemgr', 'pid': 58366961115846348777484632136820695827094854884525315789988083134291993050765, 'start': '1742436165000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 03/20/2025 02:02:49.836 7f4b82fa1a20 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:02:49.836 7f4b82fa1a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-03-20 00:47:32.457292", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8220-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 4 num_core_per_socket = 4 num_thread_per_core = 1 >> running_package_version = nightly-nightly.el7 installed_package_version = nightly-nightly.el7 >> 03/20/2025 02:02:49.836 7f4b82fa1a20 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:02:49.837 7f4b7cd0a648 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1742436169837269, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/20/2025 02:02:49.837 7f4b7cd0a648 [contrail-config-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1742436169837684, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742436165000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='zookeeper', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742436165000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-config-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742436165000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-03-20 00:47:32.457292", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8220-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/20/2025 02:02:49.842 7f4b79de6d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:02:49.843 7f4b79de6d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/20/2025 02:02:49.843 7f4b79de6d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:02:49.843 7f4b79de6d68 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:02:49.843 7f4b79de6d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742436165000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = zookeeper process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742436165000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-config-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1742436165000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-03-20 00:47:32.457292", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8220-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 03/20/2025 02:02:49.843 7f4b79de6d68 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:02:50.530 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:02:51.134 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:DatabaseUsage] map 03/20/2025 02:02:51.136 7f4b7a6472e8 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:02:51.136 7f4b7a6472e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. database_usage = [ << disk_space_used_1k = 11747144 disk_space_available_1k = 69343340 config_db_size_1k = 4676 >>, ] >> 03/20/2025 02:02:51.136 7f4b7a6472e8 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:02:52.189 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to connect to database by CQL: Result: 1 03/20/2025 02:02:52.189 7f4b7cd0a648 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1742436172189389, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/20/2025 02:02:52.190 7f4b7938d198 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:02:52.191 7f4b7938d198 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/20/2025 02:02:52.191 7f4b7938d198 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 02:02:53.276 7f4b7a79a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 02:02:53.277 7f4b7a79a048 [contrail-config-database-nodemgr] [INFO]: Session Event: TCP Connected 03/20/2025 02:02:53.277 7f4b7a79a048 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 02:02:53.277 7f4b7a79a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 02:02:53.277 7f4b7a79a048 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/20/2025 02:02:53.277 7f4b7a79a048 [contrail-config-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 02:02:53.277 7f4b7a79a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 03/20/2025 02:02:53.278 7f4b7a79a048 [contrail-config-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 03/20/2025 02:02:53.278 7f4b7a79a048 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 03/20/2025 02:02:53.278 7f4b7a79a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 02:02:53.278 7f4b7a79a048 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 02:02:53.286 7f4b79de4d68 [contrail-config-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001017 03/20/2025 02:02:53.286 7f4b79de4d68 [contrail-config-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 03/20/2025 02:02:53.287 7f4b7938d4e0 [contrail-config-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 03/20/2025 02:02:53.288 7f4b7938d4e0 [contrail-config-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 03/20/2025 02:02:53.288 7f4b7938d4e0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:DatabaseUsage] map 03/20/2025 02:02:53.288 7f4b7938d4e0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/20/2025 02:02:53.288 7f4b7938d4e0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/20/2025 02:02:53.288 7f4b7938d4e0 [contrail-config-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 03/20/2025 02:02:53.289 7f4b7e1b6c18 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/20/2025 02:02:53.290 7f4b7e1b6c18 [contrail-config-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/20/2025 02:02:55.903 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:02:58.497 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr config_db_uuid'. Error is: Result: 1 03/20/2025 02:03:02.384 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr to_bgp_keyspace'. Error is: Result: 1 03/20/2025 02:03:03.959 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr svc_monitor_keyspace'. Error is: Result: 1 03/20/2025 02:03:05.353 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr DISCOVERY_SERVER'. Error is: Result: 1 03/20/2025 02:03:07.323 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr useragent'. Error is: Result: 1 03/20/2025 02:03:08.752 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr dm_keyspace'. Error is: Result: 1 03/20/2025 02:03:13.817 7f4b7cd0a648 [contrail-config-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1742436193817414, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742436165000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='zookeeper', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742436165000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-config-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1742436165000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-03-20 00:47:32.457292", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8220-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/20/2025 02:03:13.818 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12052052, partition_space_available_1k=69038432, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1248648, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=227988, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51644, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5591100, free=319160, buffers=428752, cached=9836616, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.91, five_min_avg=3.58, fifteen_min_avg=3.67, cpu_share=0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:03:50.113 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:03:51.158 7f4b7cd0a648 [contrail-config-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-config-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2923-2.', _node_type='ConfigDatabase', _instance_id='0', _timestamp=1742436231158620, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-config-database-nodemgr', instance_id='0', state='Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.42:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo'), _seqnum=0) 03/20/2025 02:03:52.767 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:03:58.562 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12052912, partition_space_available_1k=69037572, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1309584, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=240832, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51688, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5252860, free=645808, buffers=431660, cached=9845300, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.84, five_min_avg=3.59, fifteen_min_avg=3.67, cpu_share=13.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:04:50.134 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:04:52.855 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:04:58.086 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12059140, partition_space_available_1k=69031344, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1683504, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=245188, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51712, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5790208, free=248472, buffers=431232, cached=9705716, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.49, five_min_avg=3.49, fifteen_min_avg=3.63, cpu_share=8.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:05:50.127 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:05:52.145 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:05:57.935 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12063368, partition_space_available_1k=69027116, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1691576, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=245236, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51700, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5819136, free=262920, buffers=430692, cached=9662880, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.92, five_min_avg=3.05, fifteen_min_avg=3.47, cpu_share=5.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:06:50.114 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:06:52.134 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:06:57.611 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12066756, partition_space_available_1k=69023728, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1700428, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=245252, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51692, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5837724, free=268332, buffers=430688, cached=9638884, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.06, five_min_avg=2.6, fifteen_min_avg=3.29, cpu_share=4.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:07:50.110 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:07:52.017 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:07:57.681 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12070184, partition_space_available_1k=69020300, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1703004, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=245248, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51692, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5837456, free=291252, buffers=430612, cached=9616308, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.26, five_min_avg=2.39, fifteen_min_avg=3.17, cpu_share=4.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:08:50.158 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:08:51.985 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:08:57.626 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12076508, partition_space_available_1k=69013976, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1704292, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=245256, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51696, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5852484, free=286140, buffers=430456, cached=9606548, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.44, five_min_avg=2.24, fifteen_min_avg=3.07, cpu_share=5.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:09:50.142 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:09:52.107 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:09:57.913 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12075188, partition_space_available_1k=69015296, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1710744, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=245312, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51700, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5840888, free=293704, buffers=430456, cached=9610580, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.85, five_min_avg=1.92, fifteen_min_avg=2.91, cpu_share=4.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:10:50.171 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:10:52.224 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:10:57.743 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12078652, partition_space_available_1k=69011832, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1716148, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=248144, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51700, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5861624, free=269800, buffers=430460, cached=9613744, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.84, five_min_avg=1.7, fifteen_min_avg=2.76, cpu_share=4.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:11:50.198 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:11:52.087 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:11:57.842 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12082104, partition_space_available_1k=69008380, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1707164, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=248152, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51704, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5858664, free=272052, buffers=430432, cached=9614480, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.89, five_min_avg=1.54, fifteen_min_avg=2.64, cpu_share=4.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:12:50.196 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:12:52.295 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:12:57.651 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12085828, partition_space_available_1k=69004656, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1708988, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=248164, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51704, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5877116, free=266492, buffers=430384, cached=9601636, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.04, five_min_avg=1.43, fifteen_min_avg=2.53, cpu_share=4.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:13:50.177 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:13:52.124 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:13:57.708 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12089744, partition_space_available_1k=69000740, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1711116, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=248164, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51712, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5877252, free=275168, buffers=430108, cached=9593100, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.17, five_min_avg=1.39, fifteen_min_avg=2.44, cpu_share=4.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:14:50.174 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:14:52.098 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:14:57.689 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12093372, partition_space_available_1k=68997112, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1714612, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=248168, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51712, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5875244, free=289412, buffers=430088, cached=9580884, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.79, five_min_avg=1.23, fifteen_min_avg=2.33, cpu_share=4.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:15:50.173 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:15:52.168 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:15:57.744 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12096544, partition_space_available_1k=68993940, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1722716, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=250704, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51704, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5890560, free=289524, buffers=429940, cached=9565604, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.72, five_min_avg=1.13, fifteen_min_avg=2.22, cpu_share=5.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:16:50.203 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:16:52.127 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:16:57.620 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12099616, partition_space_available_1k=68990868, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1713556, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=250712, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51712, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5894360, free=286432, buffers=430104, cached=9564732, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.7, five_min_avg=1.05, fifteen_min_avg=2.13, cpu_share=4.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:17:12.134 7f4b7c6446d8 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/20/2025 02:17:36.737 7f4b7a5f1048 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/20/2025 02:17:50.202 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:17:52.199 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:17:52.682 7f4b79201b70 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/20/2025 02:17:57.795 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12459968, partition_space_available_1k=68630516, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1715292, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=250708, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51644, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5946756, free=296808, buffers=433556, cached=9498508, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.6, five_min_avg=1.24, fifteen_min_avg=2.12, cpu_share=6.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:18:07.949 7f4b7cad60f0 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/20/2025 02:18:41.336 7f4b79ddd198 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map 03/20/2025 02:18:50.217 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:18:52.107 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:18:57.311 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12464604, partition_space_available_1k=68625880, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1719732, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=250712, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51648, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5910692, free=349740, buffers=433548, cached=9481648, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.13, five_min_avg=1.2, fifteen_min_avg=2.05, cpu_share=5.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:19:50.224 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:19:52.110 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:19:57.718 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12468428, partition_space_available_1k=68622056, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1718504, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=250712, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51644, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5914284, free=339204, buffers=433616, cached=9488524, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.66, five_min_avg=1.04, fifteen_min_avg=1.94, cpu_share=4.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:20:50.228 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:20:52.120 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:20:57.254 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12472160, partition_space_available_1k=68618324, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1726048, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=251116, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51652, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5969496, free=287024, buffers=433244, cached=9485864, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.65, five_min_avg=1.33, fifteen_min_avg=1.99, cpu_share=6.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:21:50.268 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:21:52.435 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:21:57.997 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12486452, partition_space_available_1k=68604032, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1745320, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=251244, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51644, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=5998440, free=272824, buffers=432692, cached=9471672, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.98, five_min_avg=1.96, fifteen_min_avg=2.16, cpu_share=10.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:22:50.212 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:22:52.511 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:22:57.827 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12501736, partition_space_available_1k=68588748, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1738452, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=251656, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51640, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6015216, free=274168, buffers=432540, cached=9453704, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.66, five_min_avg=1.99, fifteen_min_avg=2.16, cpu_share=9.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:23:50.208 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:23:52.195 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:23:57.798 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12511084, partition_space_available_1k=68579400, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1739396, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=251756, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51640, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6028168, free=275872, buffers=432520, cached=9439068, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.43, five_min_avg=1.76, fifteen_min_avg=2.06, cpu_share=6.75, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:24:50.244 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:24:52.250 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:24:57.825 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12524284, partition_space_available_1k=68566200, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1741812, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=251900, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51800, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6043804, free=287104, buffers=432464, cached=9412256, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.29, five_min_avg=1.7, fifteen_min_avg=2.03, cpu_share=7.25, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:25:50.242 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:25:52.101 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:25:57.688 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12533668, partition_space_available_1k=68556816, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1743460, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=251916, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51888, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6047604, free=286716, buffers=432460, cached=9408848, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.96, five_min_avg=1.54, fifteen_min_avg=1.95, cpu_share=6.5, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:26:50.282 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:26:52.282 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:26:57.767 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12556264, partition_space_available_1k=68534220, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1743776, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=251944, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51876, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6134296, free=263156, buffers=432432, cached=9345744, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.22, five_min_avg=1.56, fifteen_min_avg=1.94, cpu_share=7.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:27:50.270 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: dir for config is /var/lib/cassandra/data 03/20/2025 02:27:52.232 7f4b7cd0a648 [contrail-config-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 1 03/20/2025 02:27:57.857 7f4b7cd0a648 [contrail-config-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2923-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12563224, partition_space_available_1k=68527260, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1743240, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'zookeeper': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=251940, cpu_share=0, mem_res=0, _EventManager__key='zookeeper'), 'contrail-config-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51864, cpu_share=0, mem_res=0, _EventManager__key='contrail-config-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175628, used=6155748, free=238892, buffers=432516, cached=9348472, node_type='config-database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.29, five_min_avg=1.86, fifteen_min_avg=2.02, cpu_share=9.0, node_type='config-database-node'), running_package_version=None, installed_package_version=None, _table='ObjectConfigDatabaseInfo') 03/20/2025 02:28:39.508 7f4b7cd6d390 [contrail-config-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectConfigDatabaseInfo:NodeStatusUVE] map