02/17/2025 01:54:04.836 7f0e9a079438 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-136-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
02/17/2025 01:54:04.836 7f0e9a079438 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-136-1.cluster.local:KubernetesMgr:contrail-kube-manager:0  client_info = <<  status = Idle  successful_connections = 0  pid = 1  start_time = 1739757244835999  collector_name =   collector_ip =   collector_list = [ 10.0.0.23:8086,  ]  >>  sm_queue_count = 1  max_sm_queue_count = 1  >>  
02/17/2025 01:54:04.915 7f0e9de3e788 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//', 'pyamqp://guest:guest@:5673//']
02/17/2025 01:54:04.923 7f0e9de3e788 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down
02/17/2025 01:54:04.924 7f0e99bd44e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-136-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.23, ,  ]  status = Initializing  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down  >>,  ]  >>  
02/17/2025 01:54:04.924 7f0e99bd44e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-136-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.23, ,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down  >>,  ]  >>  
02/17/2025 01:54:04.950 7f0e9de3e788 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.23:5673// at 0x7f0e973bf668>
02/17/2025 01:54:04.951 7f0e99be4c18 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-136-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.23, ,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
02/17/2025 01:54:04.977 7f0e9de3e788 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down
02/17/2025 01:54:04.977 7f0e99bd4e10 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-136-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.23, ,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down  >>,  ]  >>  
02/17/2025 01:54:04.984 7f0e9de3e788 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.23:5673// at 0x7f0e99bc3e80>
02/17/2025 01:54:04.984 7f0ea2dc7048 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-136-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.23, ,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
02/17/2025 01:54:05.056 7f0e99be9828 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-136-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.23, ,  ]  status = Up  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.23:2181,  ]  status = Initializing  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down  >>,  ]  >>  
02/17/2025 01:54:05.106 7f0e9de3e788 [contrail-kube-manager] [WARNING]: Waiting to be elected as master...
02/17/2025 01:54:05.108 7f0e9de3e788 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Waiting to be elected as master...  
02/17/2025 01:54:05.108 7f0e99b634e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-136-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.23, ,  ]  status = Up  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.23:2181,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
02/17/2025 01:54:05.255 7f0e9de3e788 [contrail-kube-manager] [WARNING]: Elected master kube-manager node. Initializing...
02/17/2025 01:54:05.255 7f0e9de3e788 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Elected master kube-manager node. Initializing...  
02/17/2025 01:54:05.278 7f0e99ac8eb8 [contrail-kube-manager] [ERROR]: Cannot write http_port 8108 to /tmp/contrail-kube-manager.0.http_port
02/17/2025 01:54:05.278 7f0e99ac8eb8 [contrail-kube-manager] [ERROR]: Starting Introspect on HTTP Port 8108
02/17/2025 01:54:05.398 7f0e99b2d978 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-136-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.23, ,  ]  status = Up  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.23:2181,  ]  status = Up  description =   >>, <<  type = ApiServer  name = ApiServer  server_addrs = [ 10.0.0.23,:8082,  ]  status = Initializing  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[] connection down  >>,  ]  >>  
02/17/2025 01:54:05.411 7f0e9de3e788 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 
02/17/2025 01:54:05.411 7f0e9de3e788 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed:   
02/17/2025 01:54:05.411 7f0e99b0aa20 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-136-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.23, ,  ]  status = Up  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.23:2181,  ]  status = Up  description =   >>, <<  type = ApiServer  name = ApiServer  server_addrs = [ 10.0.0.23,:8082,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[] connection down  >>,  ]  >>  
02/17/2025 01:54:08.457 7f0e9de3e788 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 
02/17/2025 01:54:08.458 7f0e9de3e788 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed:   
02/17/2025 01:54:08.840 7f0e97335630 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-136-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ 10.0.0.23:8086,  ]  status = Initializing  description = Idle to Connect on EvIdleHoldTimerExpired  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.23, ,  ]  status = Up  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.23:2181,  ]  status = Up  description =   >>, <<  type = ApiServer  name = ApiServer  server_addrs = [ 10.0.0.23,:8082,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[] connection down  >>,  ]  >>  
02/17/2025 01:54:11.512 7f0e9de3e788 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 
02/17/2025 01:54:11.521 7f0e9de3e788 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed:   
02/17/2025 01:54:14.543 7f0e9de3e788 [contrail-kube-manager] [WARNING]: VNCCassandra started with driver cql
02/17/2025 01:54:14.543 7f0e9de3e788 [contrail-kube-manager] [WARNING]: initializing cassandra cluster
02/17/2025 01:54:14.543 7f0e9de3e788 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create
02/17/2025 01:54:14.545 7f0e9de3e788 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created
02/17/2025 01:54:14.602 7f0e9de3e788 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected
02/17/2025 01:54:14.622 7f0e9de3e788 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS
02/17/2025 01:54:14.640 7f0e9de3e788 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready
02/17/2025 01:54:14.705 7f0e9de3e788 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready
02/17/2025 01:54:14.718 7f0e9de3e788 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//', 'pyamqp://guest:guest@:5673//']
02/17/2025 01:54:14.718 7f0e9de3e788 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down
02/17/2025 01:54:14.736 7f0e9de3e788 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.23:5673// at 0x7f0e9972a438>
02/17/2025 01:54:14.752 7f0e9de3e788 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down
02/17/2025 01:54:14.762 7f0e9de3e788 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.23:5673// at 0x7f0e99728b38>
02/17/2025 01:54:14.765 7f0e99680d68 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down
02/17/2025 01:54:14.780 7f0e99680d68 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.23:5673// at 0x7f0e99728b38>
02/17/2025 01:54:15.170 7f0e996c48d0 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn'
02/17/2025 01:54:15.170 7f0e996c48d0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down
02/17/2025 01:54:15.285 7f0e996c48d0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.23:5673// at 0x7f0e9972a438>
02/17/2025 01:56:26.921 7f0e990724e0 [contrail-kube-manager] [ERROR]: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.
02/17/2025 01:56:26.921 7f0e990724e0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.  
02/17/2025 01:56:26.931 7f0e99078d68 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.
02/17/2025 01:56:26.931 7f0e99078d68 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.  
02/17/2025 01:56:26.964 7f0e990712e8 [contrail-kube-manager] [ERROR]: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.
02/17/2025 01:56:26.964 7f0e990712e8 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.  
02/17/2025 01:56:26.989 7f0e99065588 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.
02/17/2025 01:56:26.990 7f0e99065588 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.  
02/17/2025 01:58:26.932 7f0e990724e0 [contrail-kube-manager] [ERROR]: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.
02/17/2025 01:58:26.933 7f0e990724e0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.  
02/17/2025 01:58:26.940 7f0e99078d68 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.
02/17/2025 01:58:26.940 7f0e99078d68 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.  
02/17/2025 01:58:26.973 7f0e990712e8 [contrail-kube-manager] [ERROR]: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.
02/17/2025 01:58:26.974 7f0e990712e8 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.  
02/17/2025 01:58:26.999 7f0e99065588 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.
02/17/2025 01:58:26.999 7f0e99065588 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.  
02/17/2025 02:00:26.954 7f0e990724e0 [contrail-kube-manager] [ERROR]: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.
02/17/2025 02:00:26.954 7f0e990724e0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.  
02/17/2025 02:00:26.954 7f0e99078d68 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.
02/17/2025 02:00:26.954 7f0e99078d68 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.  
02/17/2025 02:00:26.982 7f0e990712e8 [contrail-kube-manager] [ERROR]: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.
02/17/2025 02:00:26.982 7f0e990712e8 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.  
02/17/2025 02:00:27.011 7f0e99065588 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.
02/17/2025 02:00:27.011 7f0e99065588 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.  
02/17/2025 02:02:26.968 7f0e990724e0 [contrail-kube-manager] [ERROR]: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.
02/17/2025 02:02:26.968 7f0e990724e0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.  
02/17/2025 02:02:26.971 7f0e99078d68 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.
02/17/2025 02:02:26.971 7f0e99078d68 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.  
02/17/2025 02:02:27.020 7f0e99065588 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.
02/17/2025 02:02:27.020 7f0e99065588 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.  
02/17/2025 02:04:26.988 7f0e990724e0 [contrail-kube-manager] [ERROR]: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.
02/17/2025 02:04:26.988 7f0e990724e0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.  
02/17/2025 02:04:27.030 7f0e99065588 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.
02/17/2025 02:04:27.031 7f0e99065588 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.23', port=6443): Read timed out.