03/12/2026 01:33:54.176 7efe9fc2b4a0 [contrail-config-database-nodemgr] [ERROR]: Cannot write http_port 8112 to /tmp/contrail-config-database-nodemgr.0.http_port 03/12/2026 01:33:54.176 7efe9fc2b4a0 [contrail-config-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8112 03/12/2026 01:33:54.179 7efe9fc2b860 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/12/2026 01:33:54.179 7efe9fc2b860 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1773279234178719 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 03/12/2026 01:33:54.425 7efe9fc2bea0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local build_info = {"build-info": [{"build-version": "v0+nightly", "build-time": "2026-03-12 00:49:37.110999", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-14348-1.novalocal", "build-id": "v0+nightly", "build-number": "202603120007"}]} system_cpu_info = << num_socket = 1 num_cpu = 4 num_core_per_socket = 4 num_thread_per_core = 1 >> >> 03/12/2026 01:33:54.429 7efe9fc2bea0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/12/2026 01:33:54.429 7efe9fc2bea0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1773279192000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = zookeeper process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1773279198000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-config-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1773279232000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info": [{"build-version": "v0+nightly", "build-time": "2026-03-12 00:49:37.110999", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-14348-1.novalocal", "build-id": "v0+nightly", "build-number": "202603120007"}]} >> 03/12/2026 01:33:55.852 7efe9fc2bea0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local database_usage = [ << disk_space_used_1k = 10613004 disk_space_available_1k = 72078048 config_db_size_1k = 604 >>, ] >> 03/12/2026 01:33:56.646 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to connect to database by CQL and socket: Result: 1 03/12/2026 01:33:56.684 7efe9fb50220 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/12/2026 01:33:58.185 7efe9fb50680 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/12/2026 01:33:58.189 7efe9fb50680 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/12/2026 01:33:58.189 7efe9fb50680 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/12/2026 01:33:58.190 7efe9fb50680 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1773279234178719 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/12/2026 01:34:02.193 7efe9ff6b720 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/12/2026 01:34:04.812 7efe9ff6bc20 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 03/12/2026 01:34:04.837 7efe9fe71b80 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Cassandra state detected DOWN. Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 03/12/2026 01:34:04.838 7efe9fe71b80 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1773279234178719 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/12/2026 01:34:06.290 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr config_db_uuid'. Error is: Result: 1 03/12/2026 01:34:08.840 7efea08b15e0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/12/2026 01:34:09.493 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr to_bgp_keyspace'. Error is: Result: 1 03/12/2026 01:34:13.129 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr svc_monitor_keyspace'. Error is: Result: 1 03/12/2026 01:34:15.776 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr DISCOVERY_SERVER'. Error is: Result: 1 03/12/2026 01:34:27.116 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr dm_keyspace'. Error is: Result: 1 03/12/2026 01:34:27.135 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 03/12/2026 01:34:58.725 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 03/12/2026 01:35:57.898 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 03/12/2026 01:36:58.209 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 03/12/2026 01:37:57.502 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 03/12/2026 01:38:58.282 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 03/12/2026 01:39:57.645 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 03/12/2026 01:40:57.870 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 03/12/2026 01:41:57.709 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 03/12/2026 01:42:57.584 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 03/12/2026 01:43:57.704 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 03/12/2026 01:44:57.860 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 03/12/2026 01:45:57.826 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 03/12/2026 01:46:57.791 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 03/12/2026 01:47:57.908 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 03/12/2026 01:48:57.624 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 03/12/2026 01:49:57.932 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 03/12/2026 01:50:57.772 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 03/12/2026 01:51:57.805 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 03/12/2026 01:52:57.685 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 03/12/2026 01:53:57.761 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 03/12/2026 01:54:58.369 7efea2bec680 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'