INFO: =================== Thu Sep 11 01:27:44 AM UTC 2025 =================== /contrail-functions.sh: line 167: /etc/contrail/vnc_api_lib.ini.tmp: No such file or directory /contrail-functions.sh: line 202: /etc/contrail/vnc_api_lib.ini.tmp: No such file or directory mv: cannot stat '/etc/contrail/vnc_api_lib.ini.tmp': No such file or directory [DEFAULTS] http_server_ip=0.0.0.0 log_file=/var/log/contrail/config-database-nodemgr/contrail-config-database-nodemgr.log log_level=SYS_NOTICE log_local=1 hostip=10.0.0.48 db_port=9041 db_jmx_port=7201 db_use_ssl=False [COLLECTOR] server_list=10.0.0.48:8086 [SANDESH] introspect_ssl_enable=False sandesh_ssl_enable=False [DEFAULTS] minimum_diskGB = 2 09/11/2025 01:27:45.756 7fafeaafe780 [contrail-config-database-nodemgr] [INFO]: SANDESH: CONNECT TO COLLECTOR: True 09/11/2025 01:27:45.956 7fafeaafe780 [contrail-config-database-nodemgr] [ERROR]: Failed to import package "database.sandesh" 09/11/2025 01:27:45.972 7fafeaafe780 [contrail-config-database-nodemgr] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8112 09/11/2025 01:27:45.975 7fafeaafe780 [contrail-config-database-nodemgr] [ERROR]: Failed to import package "database.sandesh" 09/11/2025 01:27:45.987 7fafeaafe780 [contrail-config-database-nodemgr] [INFO]: SANDESH: Logging: LEVEL: [SYS_INFO] -> [SYS_NOTICE] CRITICAL:root:Exit data: {'ID': '8055f5c4b4f3edd8b0bfcf5f658b4a705509cb40a58b0cd01c9ba70da65b8073', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'cqlsh', 'arguments': ['10.0.0.48', '9041', '-e', 'quit'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 65688} ERROR:contrail-config-database-nodemgr:Failed to connect to database by CQL and socket: Result: 1 ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> CRITICAL:root:Exit data: {'ID': 'a002b2f7bc34bec8d589a942c2f03012a0436927b613d8b37813a323338c1512', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 65717} ERROR:contrail-config-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 CRITICAL:root:Exit data: {'ID': '221c34622af9a41667ee8b8ccd7768e9e0d61f3805bb6a3e4846145fba1a5a8e', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'repair', '-pr', 'config_db_uuid'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 65958} ERROR:contrail-config-database-nodemgr:Failed to run cmd: 'nodetool -p 7201 repair -pr config_db_uuid'. Error is: Result: 1 ERROR:contrail-config-database-nodemgr:Session Event: TCP Connect Fail ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1757554066011301 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> CRITICAL:root:Exit data: {'ID': 'ef8829b35b8ba4721834a9bf63c7cb16074ea1f73330ee57c41c6eed2d453989', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'repair', '-pr', 'to_bgp_keyspace'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 66048} ERROR:contrail-config-database-nodemgr:Failed to run cmd: 'nodetool -p 7201 repair -pr to_bgp_keyspace'. Error is: Result: 1 CRITICAL:root:Exit data: {'ID': 'cf42d53dc4cad36e35348abe96c48cac65760dba8b7d9f491b8d7e66e2379736', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'repair', '-pr', 'svc_monitor_keyspace'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 66121} ERROR:contrail-config-database-nodemgr:Failed to run cmd: 'nodetool -p 7201 repair -pr svc_monitor_keyspace'. Error is: Result: 1 CRITICAL:root:Exit data: {'ID': 'b0aa4eb6b27c173e6bb99dac1f5255c15e08a43c928fcb058ae2bf91041c05c5', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'repair', '-pr', 'DISCOVERY_SERVER'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 66222} ERROR:contrail-config-database-nodemgr:Failed to run cmd: 'nodetool -p 7201 repair -pr DISCOVERY_SERVER'. Error is: Result: 1 CRITICAL:root:Exit data: {'ID': '1661a4c4c350833447651951330bd6c00446a0d874d1820ec8ae216b44ba6818', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'repair', '-pr', 'useragent'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 66293} ERROR:contrail-config-database-nodemgr:Failed to run cmd: 'nodetool -p 7201 repair -pr useragent'. Error is: Result: 1 CRITICAL:root:Exit data: {'ID': '6a530c374098b8ed12a878ddbfd5e4d912c809912954a57c24338f919a7e1031', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'repair', '-pr', 'dm_keyspace'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 66742} ERROR:contrail-config-database-nodemgr:Failed to run cmd: 'nodetool -p 7201 repair -pr dm_keyspace'. Error is: Result: 1 ERROR:contrail-config-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' ERROR:contrail-config-database-nodemgr:Session Event: TCP Connect Fail ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1757554066011301 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> ERROR:contrail-config-database-nodemgr:Session Event: TCP Connect Fail ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1757554066011301 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local disk_usage_info = { /dev/vda4 : << partition_type = xfs partition_space_used_1k = 9897132 partition_space_available_1k = 72793920 percentage_partition_space_used = 12 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1225520 cpu_share = 0.0 mem_res = 1216608 >>zookeeper : << mem_virt = 59712 cpu_share = 0.0 mem_res = 57388 >>contrail-config-database-nodemgr : << mem_virt = 53140 cpu_share = 0.0 mem_res = 44900 >> } system_mem_usage = << total = 16111712 used = 6486828 free = 434368 buffers = 5136 cached = 9185380 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 3.61 five_min_avg = 1.14 fifteen_min_avg = 0.45 cpu_share = 0 node_type = config-database-node >> >> ERROR:contrail-config-database-nodemgr:Session Event: TCP Connect Fail ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1757554066011301 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> ERROR:contrail-config-database-nodemgr:Session Event: TCP Connect Fail ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1757554066011301 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> ERROR:contrail-config-database-nodemgr:Session Event: TCP Connect Fail ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1757554066011301 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> ERROR:contrail-config-database-nodemgr:Session Event: TCP Connect Fail ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1757554066011301 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> ERROR:contrail-config-database-nodemgr:Session Event: TCP Connect Fail ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1757554066011301 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:Session Event: TCP Connection Closed ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Cassandra state detected DOWN. Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1757554066011301 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-248-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> CRITICAL:root:Exit data: {'ID': 'a011c5421bdfae992d54ee319226d334312e9ca9958f77726fb17970f86c12e2', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'cqlsh', 'arguments': ['10.0.0.48', '9041', '-e', 'quit'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 73753} WARNING:contrail-config-database-nodemgr:CQLSH failed, but Cassandra native transport port is open CRITICAL:root:Exit data: {'ID': '79d4b8f3fb9ae4d50365c2e2e78536d81e1efaadd310405587c72e6b84ea1e92', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 73781} ERROR:contrail-config-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-config-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' [Introspect:8112]10.0.0.48 - - [2025-09-11 01:29:44] "GET /Snh_SandeshUVECacheReq?x=NodeStatus HTTP/1.1" 200 6735 0.001166 CRITICAL:root:Exit data: {'ID': '93613bbffe2516b9ef55d39694dc7d082463c8eb02ab6409d90470a440254ee0', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'cqlsh', 'arguments': ['10.0.0.48', '9041', '-e', 'quit'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 76295} WARNING:contrail-config-database-nodemgr:CQLSH failed, but Cassandra native transport port is open [Introspect:8112]10.0.0.48 - - [2025-09-11 01:29:47] "GET /Snh_SandeshUVECacheReq?x=NodeStatus HTTP/1.1" 200 6735 0.005162 CRITICAL:root:Exit data: {'ID': '9dd72a5d516bb20cf9220107274880116438d82198ba3a6ec7f388454189f93c', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 76323} ERROR:contrail-config-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-config-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' CRITICAL:root:Exit data: {'ID': '243e85af749caedbe01b3fcb0ca0a434750258fde4e9406f3d620bb7819274d9', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'cqlsh', 'arguments': ['10.0.0.48', '9041', '-e', 'quit'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 78931} WARNING:contrail-config-database-nodemgr:CQLSH failed, but Cassandra native transport port is open CRITICAL:root:Exit data: {'ID': '4665d732711a3606493c296214da35b1c92f5f7e6b917477e11f74a255d7482d', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 78974} ERROR:contrail-config-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-config-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' CRITICAL:root:Exit data: {'ID': '89e6633a45b0f4118ce2bf755f62fec8e44f87beb2cb6597d66426ad47c45c06', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'cqlsh', 'arguments': ['10.0.0.48', '9041', '-e', 'quit'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 80819} WARNING:contrail-config-database-nodemgr:CQLSH failed, but Cassandra native transport port is open CRITICAL:root:Exit data: {'ID': 'bf735433a179ff6cde00efab8ce43b55949fa5445b8d67fdc41fb3502066e2be', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 80838} ERROR:contrail-config-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-config-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' CRITICAL:root:Exit data: {'ID': '9fb76a47bcd7375af4fca0d29b1b6d5b8320b7c8643206152492526aa70a4912', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'cqlsh', 'arguments': ['10.0.0.48', '9041', '-e', 'quit'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 83584} WARNING:contrail-config-database-nodemgr:CQLSH failed, but Cassandra native transport port is open CRITICAL:root:Exit data: {'ID': '564bc1aba3a5663eddb69fe9a0b4a075383d16db3499df41c73779dad4b0765b', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 83604} ERROR:contrail-config-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-config-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' CRITICAL:root:Exit data: {'ID': '1458c51089950e81d6372107e71377c6b37ecbcbbbef5bcc385d9febb4d010b2', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'cqlsh', 'arguments': ['10.0.0.48', '9041', '-e', 'quit'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 84734} WARNING:contrail-config-database-nodemgr:CQLSH failed, but Cassandra native transport port is open CRITICAL:root:Exit data: {'ID': 'cf48dea4265238f05857d0ef1bad587f355982dd225871f050741c34878ee0cd', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 84752} ERROR:contrail-config-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-config-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' CRITICAL:root:Exit data: {'ID': '9c6d9e4a06bc5c33e8342e3c2360c73c2a55e4763739405aaa278e2dd035d121', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'cqlsh', 'arguments': ['10.0.0.48', '9041', '-e', 'quit'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 85803} WARNING:contrail-config-database-nodemgr:CQLSH failed, but Cassandra native transport port is open CRITICAL:root:Exit data: {'ID': 'dd40b07a584190a3263cd309dfb2e0b179d54b4469fda1d2d2ef49a8253583ae', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 85829} ERROR:contrail-config-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-config-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' CRITICAL:root:Exit data: {'ID': '3561fc96f750d033c9057b5ab59efc82fc8494e4d77a6d1d578aea2fa391c45f', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'cqlsh', 'arguments': ['10.0.0.48', '9041', '-e', 'quit'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 87620} WARNING:contrail-config-database-nodemgr:CQLSH failed, but Cassandra native transport port is open CRITICAL:root:Exit data: {'ID': 'f3fbdb3872c0de5a9a8aef8061ed0c994db36f2ced1b9772a73a8bae44d269c1', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 87639} ERROR:contrail-config-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-config-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' CRITICAL:root:Exit data: {'ID': 'd8f9d91fd5c4d1e445627ffc12a8aa7092ce2fa19d10b5bcaffcdfe9eda9bd85', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'cqlsh', 'arguments': ['10.0.0.48', '9041', '-e', 'quit'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 89911} WARNING:contrail-config-database-nodemgr:CQLSH failed, but Cassandra native transport port is open CRITICAL:root:Exit data: {'ID': 'a2112111071e95715ab0c359c0351c7d169ecbc33e4bb61741b14808ba6d7533', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': '3449f8efe889864d5c5b744143afc81d01669479cb2fc1342186275a6b935df9', 'DetachKeys': '', 'Pid': 89930} ERROR:contrail-config-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-config-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' [Introspect:8112]10.0.0.48 - - [2025-09-11 01:37:01] "GET /Snh_SandeshUVECacheReq?x=NodeStatus HTTP/1.1" 200 6735 0.001700