12/25/2025 01:48:57.420 7f6c39af9360 [contrail-config-database-nodemgr] [ERROR]: Cannot write http_port 8112 to /tmp/contrail-config-database-nodemgr.0.http_port
12/25/2025 01:48:57.420 7f6c39af9360 [contrail-config-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8112
12/25/2025 01:48:57.422 7f6c39af9720 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-282-1.cluster.local  process_status = [ <<  module_id = contrail-config-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
12/25/2025 01:48:57.423 7f6c39af9720 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-282-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0  client_info = <<  status = Idle  successful_connections = 0  pid = 1  http_port = 8112  start_time = 1766627337422390  collector_name =   collector_ip =   collector_list = [ 10.0.0.249:8086,  ]  >>  sm_queue_count = 1  max_sm_queue_count = 1  >>  
12/25/2025 01:48:57.666 7f6c39af9c20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-282-1.cluster.local  build_info = {"build-info": [{"build-version": "v0+nightly", "build-time": "2025-12-25 00:54:57.145448", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-13062-1.novalocal", "build-id": "v0+nightly", "build-number": "202512250007"}]}  system_cpu_info = <<  num_socket = 1  num_cpu = 4  num_core_per_socket = 4  num_thread_per_core = 1  >>  >>  
12/25/2025 01:48:57.670 7f6c39af9c20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-282-1.cluster.local  process_status = [ <<  module_id = contrail-config-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
12/25/2025 01:48:57.670 7f6c39af9c20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-282-1.cluster.local  deleted = False  process_info = [ <<  process_name = cassandra  process_state = PROCESS_STATE_RUNNING  start_count = 1  stop_count = 0  exit_count = 0  last_start_time = 1766627294000000  last_stop_time =   last_exit_time =   core_file_list = [  ]  >>, <<  process_name = zookeeper  process_state = PROCESS_STATE_RUNNING  start_count = 1  stop_count = 0  exit_count = 0  last_start_time = 1766627285000000  last_stop_time =   last_exit_time =   core_file_list = [  ]  >>, <<  process_name = contrail-config-database-nodemgr  process_state = PROCESS_STATE_RUNNING  start_count = 1  stop_count = 0  exit_count = 0  last_start_time = 1766627335000000  last_stop_time =   last_exit_time =   core_file_list = [  ]  >>,  ]  build_info = {"build-info": [{"build-version": "v0+nightly", "build-time": "2025-12-25 00:54:57.145448", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-13062-1.novalocal", "build-id": "v0+nightly", "build-number": "202512250007"}]}  >>  
12/25/2025 01:48:58.251 7f6c39987680 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-282-1.cluster.local  database_usage = [ <<  disk_space_used_1k = 12071084  disk_space_available_1k = 70619968  config_db_size_1k = 604  >>,  ]  >>  
12/25/2025 01:48:58.696 7f6c3c6a8940 [contrail-config-database-nodemgr] [ERROR]: Failed to connect to database by CQL and socket: Result: 1
12/25/2025 01:48:58.698 7f6c39987860 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-282-1.cluster.local  process_status = [ <<  module_id = contrail-config-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>,  ]  description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
12/25/2025 01:49:01.428 7f6c397b62c0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-282-1.cluster.local  process_status = [ <<  module_id = contrail-config-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ 10.0.0.249:8086,  ]  status = Initializing  description = Idle to Connect on EvIdleHoldTimerExpired  >>,  ]  description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down  >>,  ]  >>  
12/25/2025 01:49:04.387 7f6c397b6860 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed
12/25/2025 01:49:04.388 7f6c397c5b80 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-282-1.cluster.local  process_status = [ <<  module_id = contrail-config-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ 10.0.0.249:8086,  ]  status = Down  description = ClientInit to Idle on EvTcpClose  >>,  ]  description = Cassandra state detected DOWN. Collector:Collector[ClientInit to Idle on EvTcpClose] connection down  >>,  ]  >>  
12/25/2025 01:49:04.389 7f6c397c5b80 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-282-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0  client_info = <<  status = Idle  successful_connections = 1  pid = 1  http_port = 8112  start_time = 1766627337422390  collector_name =   collector_ip = 10.0.0.249:8086  collector_list = [ 10.0.0.249:8086,  ]  >>  sm_queue_count = 1  max_sm_queue_count = 2  >>  
12/25/2025 01:49:08.399 7f6c3a4d34a0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-282-1.cluster.local  process_status = [ <<  module_id = contrail-config-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ 10.0.0.249:8086,  ]  status = Initializing  description = Idle to Connect on EvIdleHoldTimerExpired  >>,  ]  description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down  >>,  ]  >>  
12/25/2025 01:49:10.302 7f6c3c6a8940 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr config_db_uuid'. Error is: Result: 1
12/25/2025 01:49:17.675 7f6c3c6a8940 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr to_bgp_keyspace'. Error is: Result: 1
12/25/2025 01:49:21.894 7f6c3c6a8940 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr svc_monitor_keyspace'. Error is: Result: 1
12/25/2025 01:49:29.247 7f6c3c6a8940 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr DISCOVERY_SERVER'. Error is: Result: 1
12/25/2025 01:49:40.020 7f6c3c6a8940 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
12/25/2025 01:50:02.851 7f6c3c6a8940 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
12/25/2025 01:51:02.922 7f6c3c6a8940 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
12/25/2025 01:52:05.684 7f6c3c6a8940 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
12/25/2025 01:53:03.071 7f6c3c6a8940 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
12/25/2025 01:54:01.798 7f6c3c6a8940 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
12/25/2025 01:55:02.020 7f6c3c6a8940 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
12/25/2025 01:56:01.735 7f6c3c6a8940 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
12/25/2025 01:57:01.897 7f6c3c6a8940 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
12/25/2025 01:58:01.384 7f6c3c6a8940 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
12/25/2025 01:59:02.457 7f6c3c6a8940 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'