02/05/2026 01:36:38.866 7fc3f1202a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-294-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 02/05/2026 01:36:38.866 7fc3f1202a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-294-1.cluster.local:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 start_time = 1770255398865983 collector_name = collector_ip = collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 02/05/2026 01:36:39.063 7fc3f4e03c00 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.33:5673//', 'pyamqp://guest:guest@:5673//'] 02/05/2026 01:36:39.083 7fc3f4e03c00 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 02/05/2026 01:36:39.084 7fc3f111b680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-294-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33, , ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 02/05/2026 01:36:39.084 7fc3f111b680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-294-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33, , ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 02/05/2026 01:36:39.327 7fc3f4e03c00 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 02/05/2026 01:36:39.329 7fc3f111bcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-294-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33, , ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 02/05/2026 01:36:39.362 7fc3f4e03c00 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 02/05/2026 01:36:39.363 7fc3f10c6860 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-294-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33, , ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 02/05/2026 01:36:39.531 7fc3f4e03c00 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 02/05/2026 01:36:39.531 7fc3f10c6860 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-294-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33, , ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 02/05/2026 01:36:39.699 7fc3f10e19a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-294-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 02/05/2026 01:36:39.786 7fc3f4e03c00 [contrail-kube-manager] [WARNING]: Waiting to be elected as master... 02/05/2026 01:36:39.786 7fc3f4e03c00 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Waiting to be elected as master... 02/05/2026 01:36:39.786 7fc3f10e19a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-294-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 02/05/2026 01:36:40.415 7fc3f4e03c00 [contrail-kube-manager] [WARNING]: Elected master kube-manager node. Initializing... 02/05/2026 01:36:40.415 7fc3f4e03c00 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Elected master kube-manager node. Initializing... 02/05/2026 01:36:40.497 7fc3f10b0cc0 [contrail-kube-manager] [ERROR]: Cannot write http_port 8108 to /tmp/contrail-kube-manager.0.http_port 02/05/2026 01:36:40.497 7fc3f10b0cc0 [contrail-kube-manager] [ERROR]: Starting Introspect on HTTP Port 8108 02/05/2026 01:36:40.648 7fc3f27d0cc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-294-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.33,:8082, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[] connection down >>, ] >> 02/05/2026 01:36:40.657 7fc3f4e03c00 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 02/05/2026 01:36:40.657 7fc3f4e03c00 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 02/05/2026 01:36:40.657 7fc3f27d0cc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-294-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.33,:8082, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[] connection down >>, ] >> 02/05/2026 01:36:42.870 7fc3f27d0cc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-294-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.33,:8082, ] status = Down description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[] connection down >>, ] >> 02/05/2026 01:36:43.664 7fc3f4e03c00 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 02/05/2026 01:36:43.664 7fc3f4e03c00 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 02/05/2026 01:36:46.673 7fc3f4e03c00 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 02/05/2026 01:36:46.673 7fc3f4e03c00 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 02/05/2026 01:36:49.685 7fc3f4e03c00 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 02/05/2026 01:36:49.686 7fc3f4e03c00 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 02/05/2026 01:36:52.717 7fc3f4e03c00 [contrail-kube-manager] [WARNING]: VNCCassandra started with driver cql 02/05/2026 01:36:52.717 7fc3f4e03c00 [contrail-kube-manager] [WARNING]: initializing cassandra cluster 02/05/2026 01:36:52.717 7fc3f4e03c00 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 02/05/2026 01:36:52.721 7fc3f4e03c00 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 02/05/2026 01:36:52.812 7fc3f4e03c00 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 02/05/2026 01:36:52.825 7fc3f4e03c00 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 02/05/2026 01:36:52.837 7fc3f4e03c00 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 02/05/2026 01:36:52.860 7fc3f4e03c00 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 02/05/2026 01:36:52.878 7fc3f4e03c00 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.33:5673//', 'pyamqp://guest:guest@:5673//'] 02/05/2026 01:36:52.879 7fc3f4e03c00 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 02/05/2026 01:36:52.896 7fc3f4e03c00 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 02/05/2026 01:36:52.911 7fc3f4e03c00 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 02/05/2026 01:36:52.931 7fc3f4e03c00 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 02/05/2026 01:36:52.934 7fc3f27d0860 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 02/05/2026 01:36:52.954 7fc3f27d0860 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 02/05/2026 01:36:53.172 7fc3f0fb8c20 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn' 02/05/2026 01:36:53.172 7fc3f0fb8c20 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 02/05/2026 01:36:53.207 7fc3f0fb8c20 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 02/05/2026 01:39:05.907 7fc3f1075360 [contrail-kube-manager] [ERROR]: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out. 02/05/2026 01:39:05.908 7fc3f1075360 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out. 02/05/2026 01:39:05.910 7fc3f1075ae0 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out. 02/05/2026 01:39:05.911 7fc3f1075ae0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out. 02/05/2026 01:39:05.964 7fc3f1028180 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out. 02/05/2026 01:39:05.964 7fc3f1028180 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out. 02/05/2026 01:39:05.971 7fc3f1028f40 [contrail-kube-manager] [ERROR]: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out. 02/05/2026 01:39:05.971 7fc3f1028f40 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out. 02/05/2026 01:39:05.972 7fc3f1028a40 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out. 02/05/2026 01:39:05.973 7fc3f1028a40 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out. 02/05/2026 01:41:05.927 7fc3f1075ae0 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out. 02/05/2026 01:41:05.928 7fc3f1075ae0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out. 02/05/2026 01:41:05.997 7fc3f1028180 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out. 02/05/2026 01:41:05.998 7fc3f1028180 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out. 02/05/2026 01:43:06.007 7fc3f1028180 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out. 02/05/2026 01:43:06.007 7fc3f1028180 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out. 02/05/2026 01:44:35.263 7fc3f0fb8c20 [contrail-kube-manager] [WARNING]: Object virtual_machine uuid eb3ebbab-3440-4eaa-97e2-3cf0266691aa was not found for operation DELETE 02/05/2026 01:44:35.263 7fc3f0fb8c20 [contrail-kube-manager] [WARNING]: __default__ [SYS_WARN]: KubeManagerWarningLog: Object virtual_machine uuid eb3ebbab-3440-4eaa-97e2-3cf0266691aa was not found for operation DELETE 02/05/2026 01:45:06.020 7fc3f1028180 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out. 02/05/2026 01:45:06.020 7fc3f1028180 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.33', port=6443): Read timed out. 02/05/2026 01:45:18.661 7fc3f111b680 [contrail-kube-manager] [WARNING]: Delete endpoints event received while service ctest-nginx-svc-58232969 does not exist 02/05/2026 01:45:18.662 7fc3f111b680 [contrail-kube-manager] [WARNING]: __default__ [SYS_WARN]: KubeManagerWarningLog: Delete endpoints event received while service ctest-nginx-svc-58232969 does not exist