02/10/2025 01:56:21.422 7f8e04b21438 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-134-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 02/10/2025 01:56:21.422 7f8e04b21438 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-134-1.cluster.local:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 start_time = 1739152581422117 collector_name = collector_ip = collector_list = [ 10.0.0.29:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 02/10/2025 01:56:21.576 7f8e08635f48 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.29:5673//', 'pyamqp://guest:guest@:5673//'] 02/10/2025 01:56:21.585 7f8e08635f48 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 02/10/2025 01:56:21.585 7f8e044a04e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-134-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29, , ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 02/10/2025 01:56:21.586 7f8e044a04e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-134-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29, , ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 02/10/2025 01:56:21.595 7f8e08635f48 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 02/10/2025 01:56:21.596 7f8e044b3c18 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-134-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29, , ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 02/10/2025 01:56:21.628 7f8e08635f48 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 02/10/2025 01:56:21.629 7f8e044a0e10 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-134-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29, , ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 02/10/2025 01:56:21.651 7f8e08635f48 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 02/10/2025 01:56:21.655 7f8e0d1c8048 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-134-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29, , ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 02/10/2025 01:56:21.721 7f8e044b9828 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-134-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.29:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 02/10/2025 01:56:21.761 7f8e08635f48 [contrail-kube-manager] [WARNING]: Waiting to be elected as master... 02/10/2025 01:56:21.761 7f8e08635f48 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Waiting to be elected as master... 02/10/2025 01:56:21.763 7f8e044714e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-134-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.29:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 02/10/2025 01:56:22.259 7f8e08635f48 [contrail-kube-manager] [WARNING]: Elected master kube-manager node. Initializing... 02/10/2025 01:56:22.259 7f8e08635f48 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Elected master kube-manager node. Initializing... 02/10/2025 01:56:22.390 7f8e04399eb8 [contrail-kube-manager] [ERROR]: Cannot write http_port 8108 to /tmp/contrail-kube-manager.0.http_port 02/10/2025 01:56:22.390 7f8e04399eb8 [contrail-kube-manager] [ERROR]: Starting Introspect on HTTP Port 8108 02/10/2025 01:56:22.998 7f8e043fb978 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-134-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.29:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.29,:8082, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[] connection down >>, ] >> 02/10/2025 01:56:23.020 7f8e08635f48 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 02/10/2025 01:56:23.020 7f8e08635f48 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 02/10/2025 01:56:23.020 7f8e043daa20 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-134-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.29:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.29,:8082, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[] connection down >>, ] >> 02/10/2025 01:56:25.453 7f8e043daa20 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-134-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.29:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.29:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.29,:8082, ] status = Down description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[] connection down >>, ] >> 02/10/2025 01:56:26.036 7f8e08635f48 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 02/10/2025 01:56:26.036 7f8e08635f48 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 02/10/2025 01:56:29.055 7f8e08635f48 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 02/10/2025 01:56:29.056 7f8e08635f48 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 02/10/2025 01:56:32.070 7f8e08635f48 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 02/10/2025 01:56:32.070 7f8e08635f48 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 02/10/2025 01:56:35.110 7f8e08635f48 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 02/10/2025 01:56:35.111 7f8e08635f48 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 02/10/2025 01:56:38.127 7f8e08635f48 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 02/10/2025 01:56:38.127 7f8e08635f48 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 02/10/2025 01:56:41.164 7f8e08635f48 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 02/10/2025 01:56:41.164 7f8e08635f48 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 02/10/2025 01:56:44.260 7f8e08635f48 [contrail-kube-manager] [WARNING]: VNCCassandra started with driver cql 02/10/2025 01:56:44.260 7f8e08635f48 [contrail-kube-manager] [WARNING]: initializing cassandra cluster 02/10/2025 01:56:44.260 7f8e08635f48 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 02/10/2025 01:56:44.262 7f8e08635f48 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 02/10/2025 01:56:44.383 7f8e08635f48 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 02/10/2025 01:56:44.429 7f8e08635f48 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 02/10/2025 01:56:44.470 7f8e08635f48 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 02/10/2025 01:56:44.492 7f8e08635f48 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 02/10/2025 01:56:44.505 7f8e08635f48 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.29:5673//', 'pyamqp://guest:guest@:5673//'] 02/10/2025 01:56:44.506 7f8e08635f48 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 02/10/2025 01:56:44.522 7f8e08635f48 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 02/10/2025 01:56:44.540 7f8e08635f48 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 02/10/2025 01:56:44.561 7f8e08635f48 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 02/10/2025 01:56:44.569 7f8e04a694e0 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 02/10/2025 01:56:44.599 7f8e04a694e0 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 02/10/2025 01:58:54.188 7f8e01fb0c18 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 01:58:54.189 7f8e01fb0c18 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 01:58:54.201 7f8e01f8dcc0 [contrail-kube-manager] [ERROR]: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 01:58:54.201 7f8e01f8dcc0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 01:58:54.218 7f8e01fb0240 [contrail-kube-manager] [ERROR]: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 01:58:54.218 7f8e01fb0240 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 01:58:54.228 7f8e01fce6d8 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 01:58:54.228 7f8e01fce6d8 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 01:58:54.234 7f8e03729048 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 01:58:54.234 7f8e03729048 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:00:54.202 7f8e01fb0c18 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:00:54.202 7f8e01fb0c18 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:00:54.226 7f8e01fb0240 [contrail-kube-manager] [ERROR]: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:00:54.227 7f8e01fb0240 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:00:54.238 7f8e01fce6d8 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:00:54.238 7f8e01fce6d8 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:00:54.243 7f8e03729048 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:00:54.244 7f8e03729048 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:02:54.236 7f8e01fb0240 [contrail-kube-manager] [ERROR]: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:02:54.237 7f8e01fb0240 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:02:54.249 7f8e01fce6d8 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:02:54.249 7f8e01fce6d8 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:02:54.253 7f8e03729048 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:02:54.254 7f8e03729048 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:04:54.248 7f8e01fb0240 [contrail-kube-manager] [ERROR]: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:04:54.248 7f8e01fb0240 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:04:54.262 7f8e01fce6d8 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:04:54.262 7f8e01fce6d8 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:06:54.271 7f8e01fce6d8 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:06:54.272 7f8e01fce6d8 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:30:56.513 7f8e043a64e0 [contrail-kube-manager] [ERROR]: IOError PodMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:30:56.515 7f8e043a64e0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError PodMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:39:28.865 7f8e01fce6d8 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out. 02/10/2025 02:39:28.866 7f8e01fce6d8 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.29', port=6443): Read timed out.