02/02/2026 01:35:34.989 7f69046a1b80 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-293-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 02/02/2026 01:35:34.990 7f69046a1b80 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-293-1.cluster.local:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 start_time = 1769996134989488 collector_name = collector_ip = collector_list = [ 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 02/02/2026 01:35:35.086 7f69082af540 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.249:5673//', 'pyamqp://guest:guest@:5673//'] 02/02/2026 01:35:35.102 7f69082af540 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 02/02/2026 01:35:35.103 7f69045ba7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-293-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249, , ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 02/02/2026 01:35:35.103 7f69045ba7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-293-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249, , ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 02/02/2026 01:35:35.119 7f69082af540 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 02/02/2026 01:35:35.120 7f69045bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-293-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249, , ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 02/02/2026 01:35:35.142 7f69082af540 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 02/02/2026 01:35:35.142 7f6904564a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-293-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249, , ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 02/02/2026 01:35:35.153 7f69082af540 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 02/02/2026 01:35:35.153 7f6904564a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-293-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249, , ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 02/02/2026 01:35:35.169 7f6904580ae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-293-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 02/02/2026 01:35:35.180 7f69082af540 [contrail-kube-manager] [WARNING]: Waiting to be elected as master... 02/02/2026 01:35:35.181 7f69082af540 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Waiting to be elected as master... 02/02/2026 01:35:35.181 7f6904580ae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-293-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 02/02/2026 01:35:35.336 7f69082af540 [contrail-kube-manager] [WARNING]: Elected master kube-manager node. Initializing... 02/02/2026 01:35:35.336 7f69082af540 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Elected master kube-manager node. Initializing... 02/02/2026 01:35:35.376 7f69044ee040 [contrail-kube-manager] [ERROR]: Cannot write http_port 8108 to /tmp/contrail-kube-manager.0.http_port 02/02/2026 01:35:35.376 7f69044ee040 [contrail-kube-manager] [ERROR]: Starting Introspect on HTTP Port 8108 02/02/2026 01:35:35.735 7f6905ae6cc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-293-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.249,:8082, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[] connection down >>, ] >> 02/02/2026 01:35:35.738 7f69082af540 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 02/02/2026 01:35:35.738 7f69082af540 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 02/02/2026 01:35:35.738 7f6905ae6cc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-293-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.249,:8082, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[] connection down >>, ] >> 02/02/2026 01:35:38.754 7f69082af540 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 02/02/2026 01:35:38.754 7f69082af540 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 02/02/2026 01:35:38.991 7f6905ae6c20 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-293-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.249,:8082, ] status = Down description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[] connection down >>, ] >> 02/02/2026 01:35:40.283 7f6905ae6ae0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connection Closed 02/02/2026 01:35:40.284 7f69046a1cc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-293-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.249,:8082, ] status = Down description = >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose], ApiServer:ApiServer[] connection down >>, ] >> 02/02/2026 01:35:40.285 7f69046a1cc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-293-1.cluster.local:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8108 start_time = 1769996134989488 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 02/02/2026 01:35:41.848 7f69082af540 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 02/02/2026 01:35:41.848 7f69082af540 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 02/02/2026 01:35:44.289 7f69046a1cc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-293-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.249,:8082, ] status = Down description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[] connection down >>, ] >> 02/02/2026 01:35:44.975 7f69082af540 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 02/02/2026 01:35:44.975 7f69082af540 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 02/02/2026 01:35:48.060 7f69082af540 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 02/02/2026 01:35:48.060 7f69082af540 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 02/02/2026 01:35:51.071 7f69082af540 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 02/02/2026 01:35:51.072 7f69082af540 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 02/02/2026 01:35:54.122 7f69082af540 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 02/02/2026 01:35:54.123 7f69082af540 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 02/02/2026 01:35:57.138 7f69082af540 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 02/02/2026 01:35:57.138 7f69082af540 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 02/02/2026 01:36:00.153 7f69082af540 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 02/02/2026 01:36:00.153 7f69082af540 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 02/02/2026 01:36:03.174 7f69082af540 [contrail-kube-manager] [WARNING]: VNCCassandra started with driver cql 02/02/2026 01:36:03.174 7f69082af540 [contrail-kube-manager] [WARNING]: initializing cassandra cluster 02/02/2026 01:36:03.174 7f69082af540 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 02/02/2026 01:36:03.178 7f69082af540 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 02/02/2026 01:36:03.254 7f69082af540 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 02/02/2026 01:36:03.274 7f69082af540 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 02/02/2026 01:36:03.308 7f69082af540 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 02/02/2026 01:36:03.330 7f69082af540 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 02/02/2026 01:36:03.340 7f69082af540 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.249:5673//', 'pyamqp://guest:guest@:5673//'] 02/02/2026 01:36:03.341 7f69082af540 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 02/02/2026 01:36:03.348 7f69082af540 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 02/02/2026 01:36:03.362 7f69082af540 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 02/02/2026 01:36:03.383 7f69082af540 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 02/02/2026 01:36:03.392 7f6905ae6720 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 02/02/2026 01:36:03.395 7f69043e99a0 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn' 02/02/2026 01:36:03.396 7f69043e99a0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 02/02/2026 01:36:03.413 7f6905ae6720 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 02/02/2026 01:36:03.414 7f69043e99a0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 02/02/2026 01:36:03.483 7f69043e99a0 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn' 02/02/2026 01:36:03.483 7f69043e99a0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 02/02/2026 01:36:03.512 7f69043e99a0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 02/02/2026 01:36:03.573 7f69043e99a0 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn' 02/02/2026 01:36:03.573 7f69043e99a0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 02/02/2026 01:36:03.599 7f69043e99a0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 02/02/2026 01:36:03.718 7f69043e99a0 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn' 02/02/2026 01:36:03.718 7f69043e99a0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 02/02/2026 01:36:03.742 7f69043e99a0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 02/02/2026 01:36:15.224 7f69044c7b80 [contrail-kube-manager] [ERROR]: HTTPError: NetworkMonitor - invalidate resourceVersion None (response 410) 02/02/2026 01:36:15.224 7f69044c7b80 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: HTTPError: NetworkMonitor - invalidate resourceVersion None (response 410) 02/02/2026 01:38:15.202 7f69045164a0 [contrail-kube-manager] [ERROR]: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.249', port=6443): Read timed out. 02/02/2026 01:38:15.202 7f69045164a0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.249', port=6443): Read timed out. 02/02/2026 01:38:15.240 7f69044c7f40 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.249', port=6443): Read timed out. 02/02/2026 01:38:15.240 7f69044c7f40 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.249', port=6443): Read timed out. 02/02/2026 01:38:15.266 7f69044c7b80 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.249', port=6443): Read timed out. 02/02/2026 01:38:15.266 7f69044c7b80 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.249', port=6443): Read timed out. 02/02/2026 01:40:15.282 7f69044c7b80 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.249', port=6443): Read timed out. 02/02/2026 01:40:15.282 7f69044c7b80 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.249', port=6443): Read timed out. 02/02/2026 01:42:15.294 7f69044c7b80 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.249', port=6443): Read timed out. 02/02/2026 01:42:15.295 7f69044c7b80 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.249', port=6443): Read timed out. 02/02/2026 01:44:15.308 7f69044c7b80 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.249', port=6443): Read timed out. 02/02/2026 01:44:15.308 7f69044c7b80 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.249', port=6443): Read timed out. 02/02/2026 01:44:20.131 7f69044672c0 [contrail-kube-manager] [WARNING]: Delete endpoints event received while service ctest-nginx-svc-04246394 does not exist 02/02/2026 01:44:20.132 7f69044672c0 [contrail-kube-manager] [WARNING]: __default__ [SYS_WARN]: KubeManagerWarningLog: Delete endpoints event received while service ctest-nginx-svc-04246394 does not exist