08/27/2024 08:03:35.265 7f0e17d9fc48 [contrail-topology] [ERROR]: Cannot write http_port 5921 to /tmp/contrail-topology.0.http_port 08/27/2024 08:03:35.265 7f0e17d9fc48 [contrail-topology] [ERROR]: Starting Introspect on HTTP Port 5921 08/27/2024 08:03:35.266 7f0e1530a748 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-82-1.cluster.local process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.12:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Zookeeper:Zookeeper[Connection to Zookeeper initialized] connection down >>, ] >> 08/27/2024 08:03:35.266 7f0e1530a748 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-82-1.cluster.local process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.12:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Zookeeper:Zookeeper[Connection to Zookeeper initialized], Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 08/27/2024 08:03:35.267 7f0e1530a748 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-82-1.cluster.local:AnalyticsSNMP:contrail-topology:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5921 start_time = 1724745815266371 collector_name = collector_ip = collector_list = [ 10.0.0.12:8086, ] >> sm_queue_count = 2 max_sm_queue_count = 2 >> 08/27/2024 08:03:35.687 7f0e1530a748 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-82-1.cluster.local process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.12:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 08/27/2024 08:03:35.847 7f0e152ca948 [contrail-topology] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.12:5673//'] 08/27/2024 08:03:35.863 7f0e152ca948 [contrail-topology] [WARNING]: RabbitMQ drainer connection down 08/27/2024 08:03:35.867 7f0e152cad48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-82-1.cluster.local process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.12:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.12:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 08/27/2024 08:03:35.867 7f0e152cad48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-82-1.cluster.local process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.12:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.12:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 08/27/2024 08:03:35.907 7f0e152ca948 [contrail-topology] [WARNING]: RabbitMQ drainer connection ESTABLISHED 08/27/2024 08:03:35.907 7f0e152cad48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-82-1.cluster.local process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.12:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.12:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 08/27/2024 08:03:35.949 7f0e152ca948 [contrail-topology] [WARNING]: RabbitMQ publish connection down 08/27/2024 08:03:35.950 7f0e152cad48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-82-1.cluster.local process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.12:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.12:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 08/27/2024 08:03:35.976 7f0e152ca948 [contrail-topology] [WARNING]: RabbitMQ publish connection ESTABLISHED 08/27/2024 08:03:35.977 7f0e152cad48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-82-1.cluster.local process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.12:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.12:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 08/27/2024 08:03:35.981 7f0e152ca948 [contrail-topology] [WARNING]: VNCCassandra started with driver cql 08/27/2024 08:03:35.981 7f0e152ca948 [contrail-topology] [WARNING]: initializing cassandra cluster 08/27/2024 08:03:35.981 7f0e152ca948 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster before create 08/27/2024 08:03:35.982 7f0e152cad48 [contrail-topology] [WARNING]: RabbitMQ publish connection down 08/27/2024 08:03:35.983 7f0e14275a48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-82-1.cluster.local process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.12:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.12:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.12:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 08/27/2024 08:03:35.983 7f0e14275a48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-82-1.cluster.local process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.12:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.12:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.12:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 08/27/2024 08:03:35.984 7f0e152ca948 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster created 08/27/2024 08:03:36.025 7f0e152cad48 [contrail-topology] [WARNING]: RabbitMQ publish connection ESTABLISHED 08/27/2024 08:03:36.026 7f0e14275e48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-82-1.cluster.local process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.12:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.12:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.12:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 08/27/2024 08:03:36.290 7f0e152ca948 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 08/27/2024 08:03:36.292 7f0e14090648 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-82-1.cluster.local process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.12:2181, ] status = Initializing description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.12:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.12:9041, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[], Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 08/27/2024 08:03:36.322 7f0e152ca948 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 08/27/2024 08:03:36.323 7f0e14090648 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-82-1.cluster.local process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.12:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.12:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.12:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 08/27/2024 08:03:36.461 7f0e152ca948 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 08/27/2024 08:03:36.866 7f0e152ca948 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 08/27/2024 08:03:37.616 7f0e152ca948 [contrail-topology] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/27/2024 08:03:37.661 7f0e152ca948 [contrail-topology] [WARNING]: table 'obj_shared_table', created 08/27/2024 08:03:37.687 7f0e152ca948 [contrail-topology] [WARNING]: Zookeeper lock released. 08/27/2024 08:03:39.271 7f0e14275d48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-82-1.cluster.local process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.12:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.12:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.12:9041, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 08/27/2024 08:03:39.355 7f0e152ca948 [contrail-topology] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/27/2024 08:03:39.615 7f0e152ca948 [contrail-topology] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 08/27/2024 08:03:39.630 7f0e152ca948 [contrail-topology] [WARNING]: Zookeeper lock released. 08/27/2024 08:03:39.647 7f0e152ca948 [contrail-topology] [ERROR]: Error in db walk read 'type'