11/20/2025 01:28:38.351 7efd00923f40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-272-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
11/20/2025 01:28:38.351 7efd00923f40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-272-1.cluster.local:KubernetesMgr:contrail-kube-manager:0  client_info = <<  status = Idle  successful_connections = 0  pid = 1  start_time = 1763602118351019  collector_name =   collector_ip =   collector_list = [ 10.0.0.242:8086,  ]  >>  sm_queue_count = 1  max_sm_queue_count = 1  >>  
11/20/2025 01:28:38.478 7efd0450ee40 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.242:5673//', 'pyamqp://guest:guest@:5673//']
11/20/2025 01:28:38.525 7efd0450ee40 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down
11/20/2025 01:28:38.526 7efd008d10e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-272-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.242, ,  ]  status = Initializing  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down  >>,  ]  >>  
11/20/2025 01:28:38.526 7efd008d10e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-272-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.242, ,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down  >>,  ]  >>  
11/20/2025 01:28:38.540 7efd0450ee40 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.242:5673// at 0x7efd0188cdf0>
11/20/2025 01:28:38.541 7efd008d1360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-272-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.242, ,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
11/20/2025 01:28:38.573 7efd0450ee40 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down
11/20/2025 01:28:38.573 7efd00882220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-272-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.242, ,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down  >>,  ]  >>  
11/20/2025 01:28:38.595 7efd0450ee40 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.242:5673// at 0x7efcfffeb8b0>
11/20/2025 01:28:38.596 7efd00882220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-272-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.242, ,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
11/20/2025 01:28:38.708 7efd00882220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-272-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.242, ,  ]  status = Up  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.242:2181,  ]  status = Initializing  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down  >>,  ]  >>  
11/20/2025 01:28:38.722 7efd0450ee40 [contrail-kube-manager] [WARNING]: Waiting to be elected as master...
11/20/2025 01:28:38.722 7efd0450ee40 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Waiting to be elected as master...  
11/20/2025 01:28:38.722 7efd00882220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-272-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.242, ,  ]  status = Up  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.242:2181,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
11/20/2025 01:28:39.317 7efd0450ee40 [contrail-kube-manager] [WARNING]: Elected master kube-manager node. Initializing...
11/20/2025 01:28:39.317 7efd0450ee40 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Elected master kube-manager node. Initializing...  
11/20/2025 01:28:39.413 7efd007ef0e0 [contrail-kube-manager] [ERROR]: Cannot write http_port 8108 to /tmp/contrail-kube-manager.0.http_port
11/20/2025 01:28:39.413 7efd007ef0e0 [contrail-kube-manager] [ERROR]: Starting Introspect on HTTP Port 8108
11/20/2025 01:28:39.774 7efd02028c20 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-272-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.242, ,  ]  status = Up  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.242:2181,  ]  status = Up  description =   >>, <<  type = ApiServer  name = ApiServer  server_addrs = [ 10.0.0.242,:8082,  ]  status = Initializing  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[] connection down  >>,  ]  >>  
11/20/2025 01:28:39.784 7efd0450ee40 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 
11/20/2025 01:28:39.784 7efd0450ee40 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed:   
11/20/2025 01:28:39.784 7efd02028c20 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-272-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.242, ,  ]  status = Up  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.242:2181,  ]  status = Up  description =   >>, <<  type = ApiServer  name = ApiServer  server_addrs = [ 10.0.0.242,:8082,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[] connection down  >>,  ]  >>  
11/20/2025 01:28:42.356 7efd02028c20 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-272-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ 10.0.0.242:8086,  ]  status = Initializing  description = Idle to Connect on EvIdleHoldTimerExpired  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.242, ,  ]  status = Up  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.242:2181,  ]  status = Up  description =   >>, <<  type = ApiServer  name = ApiServer  server_addrs = [ 10.0.0.242,:8082,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[] connection down  >>,  ]  >>  
11/20/2025 01:28:42.792 7efd0450ee40 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 
11/20/2025 01:28:42.793 7efd0450ee40 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed:   
11/20/2025 01:28:42.988 7efd02028a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connection Closed
11/20/2025 01:28:42.989 7efd009380e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-272-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ 10.0.0.242:8086,  ]  status = Down  description = ClientInit to Idle on EvTcpClose  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.242, ,  ]  status = Up  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.242:2181,  ]  status = Up  description =   >>, <<  type = ApiServer  name = ApiServer  server_addrs = [ 10.0.0.242,:8082,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[ClientInit to Idle on EvTcpClose], ApiServer:ApiServer[] connection down  >>,  ]  >>  
11/20/2025 01:28:42.989 7efd009380e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-272-1.cluster.local:KubernetesMgr:contrail-kube-manager:0  client_info = <<  status = Idle  successful_connections = 1  pid = 1  http_port = 8108  start_time = 1763602118351019  collector_name =   collector_ip = 10.0.0.242:8086  collector_list = [ 10.0.0.242:8086,  ]  >>  sm_queue_count = 1  max_sm_queue_count = 2  >>  
11/20/2025 01:28:45.810 7efd0450ee40 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 
11/20/2025 01:28:45.810 7efd0450ee40 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed:   
11/20/2025 01:28:46.991 7efd009380e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-272-1.cluster.local  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ 10.0.0.242:8086,  ]  status = Initializing  description = Idle to Connect on EvIdleHoldTimerExpired  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.242, ,  ]  status = Up  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.242:2181,  ]  status = Up  description =   >>, <<  type = ApiServer  name = ApiServer  server_addrs = [ 10.0.0.242,:8082,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[] connection down  >>,  ]  >>  
11/20/2025 01:28:48.855 7efd0450ee40 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 
11/20/2025 01:28:48.858 7efd0450ee40 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed:   
11/20/2025 01:28:52.885 7efd0450ee40 [contrail-kube-manager] [WARNING]: VNCCassandra started with driver cql
11/20/2025 01:28:52.885 7efd0450ee40 [contrail-kube-manager] [WARNING]: initializing cassandra cluster
11/20/2025 01:28:52.885 7efd0450ee40 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create
11/20/2025 01:28:52.888 7efd0450ee40 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created
11/20/2025 01:28:52.996 7efd0450ee40 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected
11/20/2025 01:28:53.017 7efd0450ee40 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS
11/20/2025 01:28:53.023 7efd0450ee40 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready
11/20/2025 01:28:53.073 7efd0450ee40 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready
11/20/2025 01:28:53.081 7efd0450ee40 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.242:5673//', 'pyamqp://guest:guest@:5673//']
11/20/2025 01:28:53.082 7efd0450ee40 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down
11/20/2025 01:28:53.097 7efd0450ee40 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.242:5673// at 0x7efd017ad580>
11/20/2025 01:28:53.119 7efd0450ee40 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down
11/20/2025 01:28:53.144 7efd0450ee40 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.242:5673// at 0x7efd017ad310>
11/20/2025 01:28:53.149 7efd02028680 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down
11/20/2025 01:28:53.161 7efd0070dea0 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn'
11/20/2025 01:28:53.164 7efd0070dea0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down
11/20/2025 01:28:53.179 7efd02028680 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.242:5673// at 0x7efd017ad310>
11/20/2025 01:28:53.180 7efd0070dea0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.242:5673// at 0x7efd017ad580>
11/20/2025 01:28:53.468 7efd0070dea0 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn'
11/20/2025 01:28:53.468 7efd0070dea0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down
11/20/2025 01:28:53.490 7efd0070dea0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.242:5673// at 0x7efd017ad580>
11/20/2025 01:31:03.914 7efd006b8540 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.242', port=6443): Read timed out.
11/20/2025 01:31:03.914 7efd006b8540 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.242', port=6443): Read timed out.  
11/20/2025 01:31:03.922 7efd0070d7c0 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.242', port=6443): Read timed out.
11/20/2025 01:31:03.922 7efd0070d7c0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.242', port=6443): Read timed out.  
11/20/2025 01:33:03.925 7efd006b8540 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.242', port=6443): Read timed out.
11/20/2025 01:33:03.925 7efd006b8540 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.242', port=6443): Read timed out.  
11/20/2025 01:35:03.934 7efd006b8540 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.242', port=6443): Read timed out.
11/20/2025 01:35:03.935 7efd006b8540 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.242', port=6443): Read timed out.  
11/20/2025 01:35:59.726 7efd0070dea0 [contrail-kube-manager] [WARNING]: Object virtual_machine uuid e7e3ff29-da03-4064-a35e-b177a76c80db was not found for operation DELETE
11/20/2025 01:35:59.727 7efd0070dea0 [contrail-kube-manager] [WARNING]: __default__ [SYS_WARN]: KubeManagerWarningLog: Object virtual_machine uuid e7e3ff29-da03-4064-a35e-b177a76c80db was not found for operation DELETE  
11/20/2025 01:36:12.974 7efd0070d360 [contrail-kube-manager] [WARNING]: Pods listed in the received event differ from actual pods attached to service ctest-nginx-svc-97088555
11/20/2025 01:36:12.974 7efd0070d360 [contrail-kube-manager] [WARNING]: __default__ [SYS_WARN]: KubeManagerWarningLog: Pods listed in the received event differ from actual pods attached to service ctest-nginx-svc-97088555  
11/20/2025 01:36:14.386 7efd0070dea0 [contrail-kube-manager] [WARNING]: Object loadbalancer_listener uuid 3b6fa59b-97c3-4745-b1fd-4ff7cb66aa10 was not found for operation DELETE
11/20/2025 01:36:14.387 7efd0070dea0 [contrail-kube-manager] [WARNING]: __default__ [SYS_WARN]: KubeManagerWarningLog: Object loadbalancer_listener uuid 3b6fa59b-97c3-4745-b1fd-4ff7cb66aa10 was not found for operation DELETE  
11/20/2025 01:37:03.948 7efd006b8540 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.242', port=6443): Read timed out.
11/20/2025 01:37:03.948 7efd006b8540 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.242', port=6443): Read timed out.