04/02/2026 01:39:44.550 7f366fc85360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-388-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 04/02/2026 01:39:44.551 7f366fc85360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-388-1.cluster.local:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 start_time = 1775093984550411 collector_name = collector_ip = collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 04/02/2026 01:39:44.951 7f3673b34980 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.22:5673//', 'pyamqp://guest:guest@:5673//'] 04/02/2026 01:39:45.014 7f3673b34980 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 04/02/2026 01:39:45.035 7f366fc110e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-388-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22, , ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 04/02/2026 01:39:45.035 7f366fc110e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-388-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22, , ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 04/02/2026 01:39:45.057 7f3673b34980 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 04/02/2026 01:39:45.070 7f366fc115e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-388-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22, , ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 04/02/2026 01:39:45.134 7f3673b34980 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 04/02/2026 01:39:45.135 7f366fbc6180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-388-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22, , ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 04/02/2026 01:39:45.143 7f3673b34980 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 04/02/2026 01:39:45.144 7f366fbc6180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-388-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22, , ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 04/02/2026 01:39:45.260 7f366fbe72c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-388-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 04/02/2026 01:39:45.274 7f3673b34980 [contrail-kube-manager] [WARNING]: Waiting to be elected as master... 04/02/2026 01:39:45.274 7f3673b34980 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Waiting to be elected as master... 04/02/2026 01:39:45.274 7f366fbe72c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-388-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 04/02/2026 01:39:45.482 7f3673b34980 [contrail-kube-manager] [WARNING]: Elected master kube-manager node. Initializing... 04/02/2026 01:39:45.482 7f3673b34980 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Elected master kube-manager node. Initializing... 04/02/2026 01:39:45.610 7f366fbae400 [contrail-kube-manager] [ERROR]: Cannot write http_port 8108 to /tmp/contrail-kube-manager.0.http_port 04/02/2026 01:39:45.612 7f366fbae400 [contrail-kube-manager] [ERROR]: Starting Introspect on HTTP Port 8108 04/02/2026 01:39:45.872 7f3671688cc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-388-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22,:8082, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[] connection down >>, ] >> 04/02/2026 01:39:45.874 7f3673b34980 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 04/02/2026 01:39:45.874 7f3673b34980 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 04/02/2026 01:39:45.875 7f3671688cc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-388-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22,:8082, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[] connection down >>, ] >> 04/02/2026 01:39:48.555 7f3671688cc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-388-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22,:8082, ] status = Down description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[] connection down >>, ] >> 04/02/2026 01:39:48.883 7f3673b34980 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 04/02/2026 01:39:48.883 7f3673b34980 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 04/02/2026 01:39:51.896 7f3673b34980 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 04/02/2026 01:39:51.897 7f3673b34980 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 04/02/2026 01:39:54.929 7f3673b34980 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 04/02/2026 01:39:54.929 7f3673b34980 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 04/02/2026 01:39:57.961 7f3673b34980 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 04/02/2026 01:39:57.962 7f3673b34980 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 04/02/2026 01:40:00.989 7f3673b34980 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 04/02/2026 01:40:00.989 7f3673b34980 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 04/02/2026 01:40:04.000 7f3673b34980 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 04/02/2026 01:40:04.000 7f3673b34980 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 04/02/2026 01:40:07.230 7f3673b34980 [contrail-kube-manager] [WARNING]: VNCCassandra started with driver cql 04/02/2026 01:40:07.230 7f3673b34980 [contrail-kube-manager] [WARNING]: initializing cassandra cluster 04/02/2026 01:40:07.231 7f3673b34980 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 04/02/2026 01:40:07.255 7f3673b34980 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 04/02/2026 01:40:07.515 7f3673b34980 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 04/02/2026 01:40:07.543 7f3673b34980 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 04/02/2026 01:40:07.558 7f3673b34980 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 04/02/2026 01:40:07.835 7f3673b34980 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 04/02/2026 01:40:07.849 7f3673b34980 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.22:5673//', 'pyamqp://guest:guest@:5673//'] 04/02/2026 01:40:07.850 7f3673b34980 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 04/02/2026 01:40:07.870 7f3673b34980 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 04/02/2026 01:40:07.886 7f3673b34980 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 04/02/2026 01:40:07.907 7f3673b34980 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 04/02/2026 01:40:07.911 7f36716887c0 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 04/02/2026 01:40:07.933 7f36716887c0 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 04/02/2026 01:40:08.147 7f366fa4acc0 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn' 04/02/2026 01:40:08.147 7f366fa4acc0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 04/02/2026 01:40:08.162 7f366fa4acc0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 04/02/2026 01:40:08.277 7f366fa4acc0 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn' 04/02/2026 01:40:08.277 7f366fa4acc0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 04/02/2026 01:40:08.298 7f366fa4acc0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 04/02/2026 01:40:08.522 7f366fa4acc0 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn' 04/02/2026 01:40:08.522 7f366fa4acc0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 04/02/2026 01:40:08.540 7f366fa4acc0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 04/02/2026 01:40:08.687 7f366fa4acc0 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn' 04/02/2026 01:40:08.687 7f366fa4acc0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 04/02/2026 01:40:08.740 7f366fa4acc0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 04/02/2026 01:40:08.959 7f366fa4acc0 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn' 04/02/2026 01:40:08.959 7f366fa4acc0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 04/02/2026 01:40:08.975 7f366fa4acc0 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 04/02/2026 01:42:21.187 7f366fb28900 [contrail-kube-manager] [ERROR]: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.22', port=6443): Read timed out. 04/02/2026 01:42:21.187 7f366fb28900 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.22', port=6443): Read timed out. 04/02/2026 01:42:21.188 7f366fb28d60 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.22', port=6443): Read timed out. 04/02/2026 01:42:21.188 7f366fb28d60 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.22', port=6443): Read timed out. 04/02/2026 01:42:21.221 7f366fabf180 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.22', port=6443): Read timed out. 04/02/2026 01:42:21.222 7f366fabf180 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.22', port=6443): Read timed out. 04/02/2026 01:42:21.222 7f366fb28f40 [contrail-kube-manager] [ERROR]: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.22', port=6443): Read timed out. 04/02/2026 01:42:21.222 7f366fb28f40 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.22', port=6443): Read timed out. 04/02/2026 01:42:21.246 7f366fabf860 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.22', port=6443): Read timed out. 04/02/2026 01:42:21.247 7f366fabf860 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.22', port=6443): Read timed out. 04/02/2026 01:44:21.206 7f366fb28d60 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.22', port=6443): Read timed out. 04/02/2026 01:44:21.206 7f366fb28d60 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.22', port=6443): Read timed out. 04/02/2026 01:44:21.237 7f366fabf180 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.22', port=6443): Read timed out. 04/02/2026 01:44:21.237 7f366fabf180 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.22', port=6443): Read timed out. 04/02/2026 01:46:21.248 7f366fabf180 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.22', port=6443): Read timed out. 04/02/2026 01:46:21.248 7f366fabf180 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.22', port=6443): Read timed out. 04/02/2026 01:48:17.042 7f366fabf400 [contrail-kube-manager] [WARNING]: Pods listed in the received event differ from actual pods attached to service ctest-nginx-svc-18744296 04/02/2026 01:48:17.042 7f366fabf400 [contrail-kube-manager] [WARNING]: __default__ [SYS_WARN]: KubeManagerWarningLog: Pods listed in the received event differ from actual pods attached to service ctest-nginx-svc-18744296 04/02/2026 01:48:18.695 7f366fa4acc0 [contrail-kube-manager] [WARNING]: Object loadbalancer_listener uuid 21aad817-7289-4aaf-9ac1-ca0e2c52dff2 was not found for operation DELETE 04/02/2026 01:48:18.696 7f366fa4acc0 [contrail-kube-manager] [WARNING]: __default__ [SYS_WARN]: KubeManagerWarningLog: Object loadbalancer_listener uuid 21aad817-7289-4aaf-9ac1-ca0e2c52dff2 was not found for operation DELETE 04/02/2026 01:48:21.266 7f366fabf180 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.22', port=6443): Read timed out. 04/02/2026 01:48:21.266 7f366fabf180 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.22', port=6443): Read timed out.