03/12/2026 01:34:14.486 7f9250872860 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/12/2026 01:34:14.487 7f9250872860 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 start_time = 1773279254486417 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 03/12/2026 01:34:15.058 7f92548f17c0 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.50:5673//', 'pyamqp://guest:guest@:5673//'] 03/12/2026 01:34:15.070 7f92548f17c0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 03/12/2026 01:34:15.071 7f925080a4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, , ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 03/12/2026 01:34:15.071 7f925080a4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, , ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 03/12/2026 01:34:15.108 7f92548f17c0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 03/12/2026 01:34:15.110 7f925080aae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, , ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/12/2026 01:34:15.188 7f92548f17c0 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 03/12/2026 01:34:15.195 7f92507b4680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, , ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 03/12/2026 01:34:15.217 7f92548f17c0 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 03/12/2026 01:34:15.218 7f92507b4680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, , ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/12/2026 01:34:15.231 7f92507d27c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 03/12/2026 01:34:15.364 7f92548f17c0 [contrail-kube-manager] [WARNING]: Waiting to be elected as master... 03/12/2026 01:34:15.365 7f92548f17c0 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Waiting to be elected as master... 03/12/2026 01:34:15.365 7f92507d27c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/12/2026 01:34:15.618 7f92548f17c0 [contrail-kube-manager] [WARNING]: Elected master kube-manager node. Initializing... 03/12/2026 01:34:15.618 7f92548f17c0 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Elected master kube-manager node. Initializing... 03/12/2026 01:34:15.675 7f92507a4ae0 [contrail-kube-manager] [ERROR]: Cannot write http_port 8108 to /tmp/contrail-kube-manager.0.http_port 03/12/2026 01:34:15.678 7f92507a4ae0 [contrail-kube-manager] [ERROR]: Starting Introspect on HTTP Port 8108 03/12/2026 01:34:15.936 7f92505c5cc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50,:8082, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[] connection down >>, ] >> 03/12/2026 01:34:15.947 7f92548f17c0 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 03/12/2026 01:34:15.947 7f92548f17c0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 03/12/2026 01:34:15.947 7f92505c5cc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50,:8082, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[] connection down >>, ] >> 03/12/2026 01:34:18.490 7f92505c5cc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50,:8082, ] status = Down description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[] connection down >>, ] >> 03/12/2026 01:34:18.971 7f92548f17c0 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 03/12/2026 01:34:18.971 7f92548f17c0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 03/12/2026 01:34:21.992 7f92548f17c0 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 03/12/2026 01:34:21.993 7f92548f17c0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 03/12/2026 01:34:25.014 7f92548f17c0 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 03/12/2026 01:34:25.015 7f92548f17c0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 03/12/2026 01:34:28.045 7f92548f17c0 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 03/12/2026 01:34:28.046 7f92548f17c0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 03/12/2026 01:34:31.062 7f92548f17c0 [contrail-kube-manager] [WARNING]: VNCCassandra started with driver cql 03/12/2026 01:34:31.062 7f92548f17c0 [contrail-kube-manager] [WARNING]: initializing cassandra cluster 03/12/2026 01:34:31.062 7f92548f17c0 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 03/12/2026 01:34:31.067 7f92548f17c0 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 03/12/2026 01:34:31.102 7f92548f17c0 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 03/12/2026 01:34:31.110 7f92548f17c0 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 03/12/2026 01:34:31.118 7f92548f17c0 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 03/12/2026 01:34:31.144 7f92548f17c0 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 03/12/2026 01:34:31.159 7f92548f17c0 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.50:5673//', 'pyamqp://guest:guest@:5673//'] 03/12/2026 01:34:31.159 7f92548f17c0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 03/12/2026 01:34:31.210 7f92548f17c0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 03/12/2026 01:34:31.227 7f92548f17c0 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 03/12/2026 01:34:31.240 7f92548f17c0 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 03/12/2026 01:34:31.243 7f92505c57c0 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 03/12/2026 01:34:31.261 7f92505c57c0 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 03/12/2026 01:36:41.737 7f92506b5180 [contrail-kube-manager] [ERROR]: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 03/12/2026 01:36:41.738 7f92506b5180 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 03/12/2026 01:36:41.789 7f92506b5540 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 03/12/2026 01:36:41.789 7f92506b5540 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 03/12/2026 01:36:41.806 7f92506b5680 [contrail-kube-manager] [ERROR]: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 03/12/2026 01:36:41.806 7f92506b5680 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 03/12/2026 01:36:41.828 7f92506b5c20 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 03/12/2026 01:36:41.828 7f92506b5c20 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 03/12/2026 01:38:10.187 7f9250641a40 [contrail-kube-manager] [WARNING]: Object loadbalancer_listener uuid 40a12dc6-4cdb-4926-a525-fe008a4c2cd1 was not found for operation DELETE 03/12/2026 01:38:10.187 7f9250641a40 [contrail-kube-manager] [WARNING]: __default__ [SYS_WARN]: KubeManagerWarningLog: Object loadbalancer_listener uuid 40a12dc6-4cdb-4926-a525-fe008a4c2cd1 was not found for operation DELETE 03/12/2026 01:38:18.771 7f9250641a40 [contrail-kube-manager] [WARNING]: Object virtual_machine_interface uuid febef54c-1db3-11f1-befe-0242e66778c6 was not found for operation DELETE 03/12/2026 01:38:18.771 7f9250641a40 [contrail-kube-manager] [WARNING]: __default__ [SYS_WARN]: KubeManagerWarningLog: Object virtual_machine_interface uuid febef54c-1db3-11f1-befe-0242e66778c6 was not found for operation DELETE 03/12/2026 01:38:41.799 7f92506b5540 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 03/12/2026 01:38:41.799 7f92506b5540 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 03/12/2026 01:38:41.837 7f92506b5c20 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 03/12/2026 01:38:41.837 7f92506b5c20 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 03/12/2026 01:40:41.812 7f92506b5540 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 03/12/2026 01:40:41.812 7f92506b5540 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 03/12/2026 01:40:41.855 7f92506b5c20 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 03/12/2026 01:40:41.856 7f92506b5c20 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 03/12/2026 01:42:41.869 7f92506b5c20 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 03/12/2026 01:42:41.869 7f92506b5c20 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 03/12/2026 01:44:41.880 7f92506b5c20 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 03/12/2026 01:44:41.880 7f92506b5c20 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 03/12/2026 01:53:47.162 7f9250609860 [contrail-kube-manager] [WARNING]: Pods listed in the received event differ from actual pods attached to service ctest-nginx-svc-88344302 03/12/2026 01:53:47.164 7f9250609860 [contrail-kube-manager] [WARNING]: __default__ [SYS_WARN]: KubeManagerWarningLog: Pods listed in the received event differ from actual pods attached to service ctest-nginx-svc-88344302