05/26/2025 01:54:12.999 7f8b85892048 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 05/26/2025 01:54:12.999 7f8b85892048 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 05/26/2025 01:54:13.004 7f8b85894978 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-191-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 05/26/2025 01:54:13.005 7f8b85894978 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-191-1.cluster.local:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1748224453004132 collector_name = collector_ip = collector_list = [ 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 05/26/2025 01:54:13.007 7f8b88279ac8 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-191-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.57:6379 server_addrs = [ 10.0.0.57:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.57:6379[Redis Instance initialized] connection down >>, ] >> 05/26/2025 01:54:13.010 7f8b88279ac8 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-191-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.57:6379 server_addrs = [ 10.0.0.57:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.57:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.57:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 05/26/2025 01:54:13.015 7f8b88f47390 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-191-1.cluster.local counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-k8s-manifests-191-1.cluster.local:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 05/26/2025 01:54:13.016 7f8b88f47390 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-191-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.57:6379 server_addrs = [ 10.0.0.57:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.57:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.57:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 05/26/2025 01:54:13.176 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.57:5673//'] 05/26/2025 01:54:13.196 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 05/26/2025 01:54:13.203 7f8b857ee828 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-191-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.57:6379 server_addrs = [ 10.0.0.57:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.57:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.57:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.57:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 05/26/2025 01:54:13.203 7f8b857ee828 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-191-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.57:6379 server_addrs = [ 10.0.0.57:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.57:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.57:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.57:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 05/26/2025 01:54:13.225 7f8b85809c18 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-191-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.57:6379 server_addrs = [ 10.0.0.57:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.57:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.57:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 05/26/2025 01:54:13.233 7f8b858096d8 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-191-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.57:6379 server_addrs = [ 10.0.0.57:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.57:2181, ] status = Up description = Connection to Zookeeper re-established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.57:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 05/26/2025 01:54:13.234 7f8b858096d8 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-191-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.57:6379 server_addrs = [ 10.0.0.57:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.57:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.57:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 05/26/2025 01:54:13.242 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 05/26/2025 01:54:13.242 7f8b85809c18 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-191-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.57:6379 server_addrs = [ 10.0.0.57:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.57:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.57:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 05/26/2025 01:54:13.310 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 05/26/2025 01:54:13.322 7f8b85809a20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-191-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.57:6379 server_addrs = [ 10.0.0.57:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.57:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.57:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 05/26/2025 01:54:13.355 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 05/26/2025 01:54:13.356 7f8b83a65ac8 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-191-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.57:6379 server_addrs = [ 10.0.0.57:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.57:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.57:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 05/26/2025 01:54:13.363 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 05/26/2025 01:54:13.363 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 05/26/2025 01:54:13.363 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 05/26/2025 01:54:13.365 7f8b857ee2e8 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 05/26/2025 01:54:13.366 7f8b856b0198 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-191-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.57:6379 server_addrs = [ 10.0.0.57:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.57:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.57:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 05/26/2025 01:54:13.366 7f8b856b0198 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-191-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.57:6379 server_addrs = [ 10.0.0.57:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.57:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.57:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.57:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 05/26/2025 01:54:13.366 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 05/26/2025 01:54:13.401 7f8b857ee2e8 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 05/26/2025 01:54:13.401 7f8b83a88e10 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-191-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.57:6379 server_addrs = [ 10.0.0.57:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.57:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.57:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 05/26/2025 01:54:13.744 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 05/26/2025 01:54:13.745 7f8b85770cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-191-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.57:6379 server_addrs = [ 10.0.0.57:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.57:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.57:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.57:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 05/26/2025 01:54:13.773 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 05/26/2025 01:54:13.775 7f8b85847ac8 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-191-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.57:6379 server_addrs = [ 10.0.0.57:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.57:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.57:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.57:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 05/26/2025 01:54:13.793 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 05/26/2025 01:54:13.985 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 05/26/2025 01:54:14.016 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/26/2025 01:54:14.668 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: table 'obj_uuid_table', created 05/26/2025 01:54:14.682 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: Zookeeper lock released. 05/26/2025 01:54:14.718 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/26/2025 01:54:15.344 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: table 'obj_uuid_table' fixed with properties {'gc_grace_seconds': 864000} 05/26/2025 01:54:15.405 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: Zookeeper lock released. 05/26/2025 01:54:16.042 7f8b8577f390 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-191-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.57:6379 server_addrs = [ 10.0.0.57:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.57:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.57:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.57:2181, ] status = Up description = >>, << type = Redis-UVE name = AggregateRedis server_addrs = [ 10.0.0.57:6379, ] status = Up description = Connected to Aggregate Redis >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 05/26/2025 01:54:16.508 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/26/2025 01:54:17.004 7f8b83a88eb8 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-191-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.57:6379 server_addrs = [ 10.0.0.57:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.57:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.57:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.57:2181, ] status = Up description = >>, << type = Redis-UVE name = AggregateRedis server_addrs = [ 10.0.0.57:6379, ] status = Up description = Connected to Aggregate Redis >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 05/26/2025 01:54:17.219 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: table 'obj_fq_name_table', created 05/26/2025 01:54:17.241 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: Zookeeper lock released. 05/26/2025 01:54:17.822 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/26/2025 01:54:20.552 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: table 'obj_fq_name_table' fixed with properties {'gc_grace_seconds': 864000} 05/26/2025 01:54:20.832 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: Zookeeper lock released. 05/26/2025 01:54:22.281 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/26/2025 01:54:22.312 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: table 'obj_shared_table', created 05/26/2025 01:54:22.345 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: Zookeeper lock released. 05/26/2025 01:54:23.439 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/26/2025 01:54:24.672 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 05/26/2025 01:54:24.685 7f8b858a6f60 [contrail-alarm-gen] [WARNING]: Zookeeper lock released. 05/26/2025 01:54:33.586 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:0 05/26/2025 01:54:33.586 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:1 05/26/2025 01:54:33.586 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:2 05/26/2025 01:54:33.586 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:3 05/26/2025 01:54:33.586 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:4 05/26/2025 01:54:33.587 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:5 05/26/2025 01:54:33.587 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:6 05/26/2025 01:54:33.587 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:7 05/26/2025 01:54:33.587 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:8 05/26/2025 01:54:33.587 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:9 05/26/2025 01:54:33.587 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:10 05/26/2025 01:54:33.587 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:11 05/26/2025 01:54:33.587 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:12 05/26/2025 01:54:33.587 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:13 05/26/2025 01:54:33.587 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:14 05/26/2025 01:54:33.588 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:15 05/26/2025 01:54:33.588 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:16 05/26/2025 01:54:33.588 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:17 05/26/2025 01:54:33.588 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:18 05/26/2025 01:54:33.588 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:19 05/26/2025 01:54:33.588 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:20 05/26/2025 01:54:33.588 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:21 05/26/2025 01:54:33.588 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:22 05/26/2025 01:54:33.588 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:23 05/26/2025 01:54:33.589 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:24 05/26/2025 01:54:33.591 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:25 05/26/2025 01:54:33.591 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:26 05/26/2025 01:54:33.591 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:27 05/26/2025 01:54:33.591 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:28 05/26/2025 01:54:33.591 7f8b858a68d0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:29