08/04/2025 01:55:15.976 7f0636592048 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-233-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
08/04/2025 01:55:15.976 7f0636592048 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-233-1.cluster.local:KubernetesMgr:contrail-kube-manager:0  client_info = <<  status = Idle  successful_connections = 0  pid = 1  start_time = 1754272515976142  collector_name =   collector_ip =   collector_list = [ 10.0.0.48:8086,  ]  >>  sm_queue_count = 1  max_sm_queue_count = 1  >>  
08/04/2025 01:55:16.052 7f063a014ec8 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.48:5673//', 'pyamqp://guest:guest@:5673//']
08/04/2025 01:55:16.058 7f063a014ec8 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down
08/04/2025 01:55:16.059 7f0635e92588 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-233-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.48, ,  ]  status = Initializing  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down  >>,  ]  >>  
08/04/2025 01:55:16.059 7f0635e92588 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-233-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.48, ,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down  >>,  ]  >>  
08/04/2025 01:55:16.081 7f063a014ec8 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.48:5673// at 0x7f063346d860>
08/04/2025 01:55:16.081 7f0635ea5cc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-233-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.48, ,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
08/04/2025 01:55:16.095 7f063a014ec8 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down
08/04/2025 01:55:16.096 7f0635e92b70 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-233-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.48, ,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down  >>,  ]  >>  
08/04/2025 01:55:16.118 7f063a014ec8 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.48:5673// at 0x7f0635f3cd30>
08/04/2025 01:55:16.118 7f0635e92cc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-233-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.48, ,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
08/04/2025 01:55:16.389 7f0635eaa2e8 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-233-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.48, ,  ]  status = Up  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.48:2181,  ]  status = Initializing  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down  >>,  ]  >>  
08/04/2025 01:55:16.553 7f063a014ec8 [contrail-kube-manager] [WARNING]: Waiting to be elected as master...
08/04/2025 01:55:16.553 7f063a014ec8 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Waiting to be elected as master...  
08/04/2025 01:55:16.554 7f06352f1438 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-233-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.48, ,  ]  status = Up  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.48:2181,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
08/04/2025 01:55:17.766 7f063a014ec8 [contrail-kube-manager] [WARNING]: Elected master kube-manager node. Initializing...
08/04/2025 01:55:17.766 7f063a014ec8 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Elected master kube-manager node. Initializing...  
08/04/2025 01:55:17.808 7f063525ff60 [contrail-kube-manager] [ERROR]: Cannot write http_port 8108 to /tmp/contrail-kube-manager.0.http_port
08/04/2025 01:55:17.808 7f063525ff60 [contrail-kube-manager] [ERROR]: Starting Introspect on HTTP Port 8108
08/04/2025 01:55:18.009 7f06352c4a20 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-233-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.48, ,  ]  status = Up  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.48:2181,  ]  status = Up  description =   >>, <<  type = ApiServer  name = ApiServer  server_addrs = [ 10.0.0.48,:8082,  ]  status = Initializing  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[] connection down  >>,  ]  >>  
08/04/2025 01:55:18.014 7f063a014ec8 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 
08/04/2025 01:55:18.014 7f063a014ec8 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed:   
08/04/2025 01:55:18.014 7f06352c4a20 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-233-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.48, ,  ]  status = Up  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.48:2181,  ]  status = Up  description =   >>, <<  type = ApiServer  name = ApiServer  server_addrs = [ 10.0.0.48,:8082,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[] connection down  >>,  ]  >>  
08/04/2025 01:55:19.981 7f06352c4a20 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-233-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ 10.0.0.48:8086,  ]  status = Initializing  description = Idle to Connect on EvIdleHoldTimerExpired  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.48, ,  ]  status = Up  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.48:2181,  ]  status = Up  description =   >>, <<  type = ApiServer  name = ApiServer  server_addrs = [ 10.0.0.48,:8082,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[] connection down  >>,  ]  >>  
08/04/2025 01:55:21.032 7f063a014ec8 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 
08/04/2025 01:55:21.033 7f063a014ec8 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed:   
08/04/2025 01:55:24.131 7f063a014ec8 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 
08/04/2025 01:55:24.131 7f063a014ec8 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed:   
08/04/2025 01:55:27.162 7f063a014ec8 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 
08/04/2025 01:55:27.163 7f063a014ec8 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed:   
08/04/2025 01:55:30.259 7f063a014ec8 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 
08/04/2025 01:55:30.262 7f063a014ec8 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed:   
08/04/2025 01:55:33.291 7f063a014ec8 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 
08/04/2025 01:55:33.291 7f063a014ec8 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed:   
08/04/2025 01:55:36.336 7f063a014ec8 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 
08/04/2025 01:55:36.336 7f063a014ec8 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed:   
08/04/2025 01:55:39.492 7f063a014ec8 [contrail-kube-manager] [WARNING]: VNCCassandra started with driver cql
08/04/2025 01:55:39.492 7f063a014ec8 [contrail-kube-manager] [WARNING]: initializing cassandra cluster
08/04/2025 01:55:39.493 7f063a014ec8 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create
08/04/2025 01:55:39.499 7f063a014ec8 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created
08/04/2025 01:55:39.544 7f063a014ec8 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected
08/04/2025 01:55:39.556 7f063a014ec8 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS
08/04/2025 01:55:39.566 7f063a014ec8 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready
08/04/2025 01:55:39.599 7f063a014ec8 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready
08/04/2025 01:55:39.605 7f063a014ec8 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.48:5673//', 'pyamqp://guest:guest@:5673//']
08/04/2025 01:55:39.605 7f063a014ec8 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down
08/04/2025 01:55:39.612 7f063a014ec8 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.48:5673// at 0x7f0635eacb00>
08/04/2025 01:55:39.646 7f063a014ec8 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down
08/04/2025 01:55:39.664 7f063a014ec8 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.48:5673// at 0x7f0633051ba8>
08/04/2025 01:55:39.670 7f06331cc2e8 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down
08/04/2025 01:55:39.676 7f0630550e10 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn'
08/04/2025 01:55:39.676 7f0630550e10 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down
08/04/2025 01:55:39.698 7f06331cc2e8 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.48:5673// at 0x7f0633051ba8>
08/04/2025 01:55:39.700 7f0630550e10 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.48:5673// at 0x7f0635eacb00>
08/04/2025 01:55:39.714 7f0630550e10 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn'
08/04/2025 01:55:39.714 7f0630550e10 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down
08/04/2025 01:55:39.744 7f0630550e10 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.48:5673// at 0x7f0635eacb00>
08/04/2025 01:55:39.771 7f0630550e10 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn'
08/04/2025 01:55:39.772 7f0630550e10 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down
08/04/2025 01:55:39.785 7f0630550e10 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.48:5673// at 0x7f0635eacb00>
08/04/2025 01:55:39.865 7f0630550e10 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn'
08/04/2025 01:55:39.865 7f0630550e10 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down
08/04/2025 01:55:39.872 7f0630550e10 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.48:5673// at 0x7f0635eacb00>
08/04/2025 01:55:39.900 7f0630550e10 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn'
08/04/2025 01:55:39.900 7f0630550e10 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down
08/04/2025 01:55:39.918 7f0630550e10 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.48:5673// at 0x7f0635eacb00>
08/04/2025 01:57:48.510 7f06351f4630 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out.
08/04/2025 01:57:48.511 7f06351f4630 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out.  
08/04/2025 01:57:48.534 7f06338e3a20 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out.
08/04/2025 01:57:48.534 7f06338e3a20 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out.  
08/04/2025 01:57:48.544 7f06358ef8d0 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out.
08/04/2025 01:57:48.544 7f06358ef8d0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out.  
08/04/2025 01:59:48.543 7f06338e3a20 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out.
08/04/2025 01:59:48.543 7f06338e3a20 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out.  
08/04/2025 02:01:48.554 7f06338e3a20 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out.
08/04/2025 02:01:48.554 7f06338e3a20 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out.  
08/04/2025 02:03:01.006 7f06331cc0f0 [contrail-kube-manager] [WARNING]: Delete endpoints event received while service ctest-nginx-svc-68922114 does not exist
08/04/2025 02:03:01.006 7f06331cc0f0 [contrail-kube-manager] [WARNING]: __default__ [SYS_WARN]: KubeManagerWarningLog: Delete endpoints event received while service ctest-nginx-svc-68922114 does not exist  
08/04/2025 02:03:48.568 7f06338e3a20 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out.
08/04/2025 02:03:48.568 7f06338e3a20 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out.