08/14/2025 01:37:30.867 7fc5fd3ba048 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-236-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
08/14/2025 01:37:30.868 7fc5fd3ba048 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-236-1.cluster.local:KubernetesMgr:contrail-kube-manager:0  client_info = <<  status = Idle  successful_connections = 0  pid = 1  start_time = 1755135450867688  collector_name =   collector_ip =   collector_list = [ 10.0.0.33:8086,  ]  >>  sm_queue_count = 1  max_sm_queue_count = 1  >>  
08/14/2025 01:37:31.198 7fc600e2c488 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.33:5673//', 'pyamqp://guest:guest@:5673//']
08/14/2025 01:37:31.219 7fc600e2c488 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down
08/14/2025 01:37:31.220 7fc5faa1d588 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-236-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.33, ,  ]  status = Initializing  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down  >>,  ]  >>  
08/14/2025 01:37:31.220 7fc5faa1d588 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-236-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.33, ,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down  >>,  ]  >>  
08/14/2025 01:37:31.247 7fc600e2c488 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.33:5673// at 0x7fc5fd3b6dd8>
08/14/2025 01:37:31.247 7fc5faa3acc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-236-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.33, ,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
08/14/2025 01:37:31.273 7fc600e2c488 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down
08/14/2025 01:37:31.274 7fc5faa1db70 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-236-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.33, ,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down  >>,  ]  >>  
08/14/2025 01:37:31.306 7fc600e2c488 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.33:5673// at 0x7fc5faa12a20>
08/14/2025 01:37:31.306 7fc605dab048 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-236-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.33, ,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
08/14/2025 01:37:31.345 7fc5faa372e8 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-236-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.33, ,  ]  status = Up  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.33:2181,  ]  status = Initializing  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down  >>,  ]  >>  
08/14/2025 01:37:31.360 7fc600e2c488 [contrail-kube-manager] [WARNING]: Waiting to be elected as master...
08/14/2025 01:37:31.360 7fc600e2c488 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Waiting to be elected as master...  
08/14/2025 01:37:31.361 7fc5fc17e438 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-236-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.33, ,  ]  status = Up  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.33:2181,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
08/14/2025 01:37:31.545 7fc600e2c488 [contrail-kube-manager] [WARNING]: Elected master kube-manager node. Initializing...
08/14/2025 01:37:31.546 7fc600e2c488 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Elected master kube-manager node. Initializing...  
08/14/2025 01:37:31.627 7fc5fa9a3f60 [contrail-kube-manager] [ERROR]: Cannot write http_port 8108 to /tmp/contrail-kube-manager.0.http_port
08/14/2025 01:37:31.627 7fc5fa9a3f60 [contrail-kube-manager] [ERROR]: Starting Introspect on HTTP Port 8108
08/14/2025 01:37:31.765 7fc5fa9fea20 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-236-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.33, ,  ]  status = Up  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.33:2181,  ]  status = Up  description =   >>, <<  type = ApiServer  name = ApiServer  server_addrs = [ 10.0.0.33,:8082,  ]  status = Initializing  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[] connection down  >>,  ]  >>  
08/14/2025 01:37:31.773 7fc600e2c488 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 
08/14/2025 01:37:31.773 7fc600e2c488 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed:   
08/14/2025 01:37:31.773 7fc5fa9fea20 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-236-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.33, ,  ]  status = Up  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.33:2181,  ]  status = Up  description =   >>, <<  type = ApiServer  name = ApiServer  server_addrs = [ 10.0.0.33,:8082,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[] connection down  >>,  ]  >>  
08/14/2025 01:37:34.794 7fc600e2c488 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 
08/14/2025 01:37:34.794 7fc600e2c488 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed:   
08/14/2025 01:37:34.875 7fc5f91858d0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-236-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ 10.0.0.33:8086,  ]  status = Initializing  description = Idle to Connect on EvIdleHoldTimerExpired  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.33, ,  ]  status = Up  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.33:2181,  ]  status = Up  description =   >>, <<  type = ApiServer  name = ApiServer  server_addrs = [ 10.0.0.33,:8082,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[] connection down  >>,  ]  >>  
08/14/2025 01:37:37.824 7fc600e2c488 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 
08/14/2025 01:37:37.825 7fc600e2c488 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed:   
08/14/2025 01:37:40.845 7fc600e2c488 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 
08/14/2025 01:37:40.845 7fc600e2c488 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed:   
08/14/2025 01:37:43.903 7fc600e2c488 [contrail-kube-manager] [WARNING]: VNCCassandra started with driver cql
08/14/2025 01:37:43.903 7fc600e2c488 [contrail-kube-manager] [WARNING]: initializing cassandra cluster
08/14/2025 01:37:43.904 7fc600e2c488 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create
08/14/2025 01:37:43.906 7fc600e2c488 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created
08/14/2025 01:37:44.104 7fc600e2c488 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected
08/14/2025 01:37:44.118 7fc600e2c488 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS
08/14/2025 01:37:44.163 7fc600e2c488 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready
08/14/2025 01:37:44.306 7fc600e2c488 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready
08/14/2025 01:37:44.339 7fc600e2c488 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.33:5673//', 'pyamqp://guest:guest@:5673//']
08/14/2025 01:37:44.340 7fc600e2c488 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down
08/14/2025 01:37:44.371 7fc600e2c488 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.33:5673// at 0x7fc5fc16cf60>
08/14/2025 01:37:44.430 7fc600e2c488 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down
08/14/2025 01:37:44.444 7fc600e2c488 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.33:5673// at 0x7fc5f9169dd8>
08/14/2025 01:37:44.452 7fc5fa85c4e0 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down
08/14/2025 01:37:44.475 7fc5fa85c4e0 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.33:5673// at 0x7fc5f9169dd8>
08/14/2025 01:37:44.707 7fc5fa890198 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn'
08/14/2025 01:37:44.707 7fc5fa890198 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down
08/14/2025 01:37:44.724 7fc5fa890198 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.33:5673// at 0x7fc5fc16cf60>
08/14/2025 01:37:44.801 7fc5fa890198 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn'
08/14/2025 01:37:44.801 7fc5fa890198 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down
08/14/2025 01:37:44.815 7fc5fa890198 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.33:5673// at 0x7fc5fc16cf60>
08/14/2025 01:37:44.854 7fc5fa890198 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn'
08/14/2025 01:37:44.854 7fc5fa890198 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down
08/14/2025 01:37:44.870 7fc5fa890198 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.33:5673// at 0x7fc5fc16cf60>
08/14/2025 01:37:45.188 7fc5fa890198 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn'
08/14/2025 01:37:45.188 7fc5fa890198 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down
08/14/2025 01:37:45.198 7fc5fa890198 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.33:5673// at 0x7fc5fc16cf60>
08/14/2025 01:37:45.448 7fc5fa890198 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn'
08/14/2025 01:37:45.448 7fc5fa890198 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down
08/14/2025 01:37:45.474 7fc5fa890198 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.33:5673// at 0x7fc5fc16cf60>
08/14/2025 01:37:45.570 7fc5fa890198 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn'
08/14/2025 01:37:45.570 7fc5fa890198 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down
08/14/2025 01:37:45.588 7fc5fa890198 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.33:5673// at 0x7fc5fc16cf60>
08/14/2025 01:39:56.837 7fc5ff123780 [contrail-kube-manager] [ERROR]: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out.
08/14/2025 01:39:56.837 7fc5ff123780 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out.  
08/14/2025 01:39:56.840 7fc5fa9fe828 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out.
08/14/2025 01:39:56.841 7fc5fa9fe828 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out.  
08/14/2025 01:39:56.900 7fc5f92a2438 [contrail-kube-manager] [ERROR]: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out.
08/14/2025 01:39:56.900 7fc5f92a2438 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out.  
08/14/2025 01:39:56.904 7fc5f9253780 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out.
08/14/2025 01:39:56.905 7fc5f9253780 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out.  
08/14/2025 01:39:56.913 7fc5f92a3630 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out.
08/14/2025 01:39:56.914 7fc5f92a3630 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out.  
08/14/2025 01:41:56.917 7fc5f9253780 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out.
08/14/2025 01:41:56.918 7fc5f9253780 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out.  
08/14/2025 01:43:56.927 7fc5f9253780 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out.
08/14/2025 01:43:56.927 7fc5f9253780 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out.  
08/14/2025 01:45:39.830 7fc5f92a3780 [contrail-kube-manager] [WARNING]: Pods listed in the received event differ from actual pods attached to service ctest-nginx-svc-59705830
08/14/2025 01:45:39.830 7fc5f92a3780 [contrail-kube-manager] [WARNING]: __default__ [SYS_WARN]: KubeManagerWarningLog: Pods listed in the received event differ from actual pods attached to service ctest-nginx-svc-59705830  
08/14/2025 01:45:56.938 7fc5f9253780 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out.
08/14/2025 01:45:56.939 7fc5f9253780 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out.