07/24/2025 06:37:26.888 7f89d886b898 [contrail-topology] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 07/24/2025 06:37:26.888 7f89d886b898 [contrail-topology] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 07/24/2025 06:37:26.888 7f89d886b898 [contrail-topology] [INFO]: SANDESH: Flow Logging: [None] -> [False] 07/24/2025 06:37:26.889 7f89d886b898 [contrail-topology] [DEBUG]: Add uve in the [ObjectAnalyticsSNMPInfo:NodeStatusUVE] map 07/24/2025 06:37:26.889 7f89d886b898 [contrail-topology] [INFO]: Consistent scheduler zk start 07/24/2025 06:37:26.889 7f89d2596d48 [contrail-topology] [ERROR]: Cannot write http_port 5921 to /tmp/contrail-topology.0.http_port 07/24/2025 06:37:26.889 7f89d2596d48 [contrail-topology] [ERROR]: Starting Introspect on HTTP Port 5921 07/24/2025 06:37:26.890 7f89cfac3848 [contrail-topology] [INFO]: Processing event[EvStart] in state[none] 07/24/2025 06:37:26.890 7f89cfac3848 [contrail-topology] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 07/24/2025 06:37:26.890 7f89cfac3848 [contrail-topology] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 07/24/2025 06:37:26.891 7f89cfac3848 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:26.891 7f89cfac3848 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Zookeeper:Zookeeper[Connection to Zookeeper initialized] connection down >>, ] >> 07/24/2025 06:37:26.891 7f89cfac3848 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:26.891 7f89cfac3848 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:26.892 7f89cfac3848 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Zookeeper:Zookeeper[Connection to Zookeeper initialized], Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/24/2025 06:37:26.892 7f89cfac3848 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:26.892 7f89cfac3848 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:26.892 7f89cfac3848 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3.:AnalyticsSNMP:contrail-topology:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5921 start_time = 1753339046890744 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, 10.0.0.38:8086, 10.0.0.254:8086, ] >> sm_queue_count = 2 max_sm_queue_count = 2 >> 07/24/2025 06:37:26.892 7f89cfac3848 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:26.909 7f89cfa3a848 [contrail-topology] [INFO]: Consistent scheduler listen CONNECTED 07/24/2025 06:37:26.909 7f89cfac3848 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:26.909 7f89cfac3848 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/24/2025 06:37:26.910 7f89cfac3848 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:27.051 7f89cfa3ab48 [contrail-topology] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.254:5673//', 'pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 07/24/2025 06:37:27.055 7f89cfa3ab48 [contrail-topology] [WARNING]: RabbitMQ drainer connection down 07/24/2025 06:37:27.056 7f89cfa3ae48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:27.056 7f89cfa3ae48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 07/24/2025 06:37:27.056 7f89cfa3ae48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:27.056 7f89cfa3ae48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:27.057 7f89cfa3ae48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 07/24/2025 06:37:27.057 7f89cfa3ae48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:27.075 7f89cfa3ab48 [contrail-topology] [WARNING]: RabbitMQ drainer connection ESTABLISHED 07/24/2025 06:37:27.075 7f89cfa3ae48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:27.076 7f89cfa3ae48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/24/2025 06:37:27.076 7f89cfa3ae48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:27.088 7f89cfa3aa48 [contrail-topology] [DEBUG]: Waiting for allocation... 07/24/2025 06:37:27.136 7f89cfa3ab48 [contrail-topology] [WARNING]: RabbitMQ publish connection down 07/24/2025 06:37:27.137 7f89cfa3ad48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:27.137 7f89cfa3ad48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 07/24/2025 06:37:27.137 7f89cfa3ad48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:27.147 7f89cfa3ab48 [contrail-topology] [WARNING]: RabbitMQ publish connection ESTABLISHED 07/24/2025 06:37:27.147 7f89cfa3ad48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:27.147 7f89cfa3ad48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/24/2025 06:37:27.147 7f89cfa3ad48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:27.149 7f89cfa3ab48 [contrail-topology] [WARNING]: VNCCassandra started with driver cql 07/24/2025 06:37:27.149 7f89cfa3ab48 [contrail-topology] [WARNING]: initializing cassandra cluster 07/24/2025 06:37:27.149 7f89cfa3ab48 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster before create 07/24/2025 06:37:27.149 7f89cfa3ad48 [contrail-topology] [WARNING]: RabbitMQ publish connection down 07/24/2025 06:37:27.150 7f89cfa3ae48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:27.150 7f89cfa3ae48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 07/24/2025 06:37:27.150 7f89cfa3ae48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:27.150 7f89cfa3ae48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:27.150 7f89cfa3ae48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 07/24/2025 06:37:27.150 7f89cfa3ae48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:27.151 7f89cfa3ab48 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster created 07/24/2025 06:37:27.159 7f89cfa3ad48 [contrail-topology] [WARNING]: RabbitMQ publish connection ESTABLISHED 07/24/2025 06:37:27.163 7f89cd67c248 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:27.163 7f89cd67c248 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 07/24/2025 06:37:27.163 7f89cd67c248 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:27.197 7f89cfa3ab48 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 07/24/2025 06:37:27.204 7f89cd67cb48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:27.205 7f89cd67cb48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Initializing description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[], Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 07/24/2025 06:37:27.205 7f89cd67cb48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:27.212 7f89cfa3ab48 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 07/24/2025 06:37:27.212 7f89cd67c848 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:27.213 7f89cd67c848 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 07/24/2025 06:37:27.213 7f89cd67c848 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:37:27.240 7f89cfa3ab48 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 07/24/2025 06:37:30.891 7f89cd684848 [contrail-topology] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/24/2025 06:37:30.892 7f89cd684848 [contrail-topology] [INFO]: Session Event: TCP Connected 07/24/2025 06:37:30.892 7f89cd684848 [contrail-topology] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/24/2025 06:37:30.892 7f89cd684848 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/24/2025 06:37:30.892 7f89cd684848 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 07/24/2025 06:37:30.892 7f89cd684848 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/24/2025 06:37:30.892 7f89cd684848 [contrail-topology] [INFO]: Processing event[EvTcpConnected] in state[Connect] 07/24/2025 06:37:30.892 7f89cd684848 [contrail-topology] [DEBUG]: Send sandesh control message. uve type count # 5 07/24/2025 06:37:30.893 7f89cd684848 [contrail-topology] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 07/24/2025 06:37:30.893 7f89cd684848 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 07/24/2025 06:37:30.893 7f89cd684848 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 07/24/2025 06:37:30.893 7f89cd684a48 [contrail-topology] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 07/24/2025 06:37:30.894 7f89cd684a48 [contrail-topology] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 07/24/2025 06:37:30.897 7f89cd684948 [contrail-topology] [DEBUG]: Extracted sandesh length: 0000001017 07/24/2025 06:37:30.898 7f89cd684948 [contrail-topology] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 07/24/2025 06:37:30.899 7f89cd684a48 [contrail-topology] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 07/24/2025 06:37:30.899 7f89cd684a48 [contrail-topology] [DEBUG]: Number of uve types in sandesh control message is 0 07/24/2025 06:37:30.899 7f89cd684a48 [contrail-topology] [DEBUG]: send sync_uve in the [ObjectAnalyticsSNMPInfo:NodeStatusUVE] map 07/24/2025 06:37:30.899 7f89cd684a48 [contrail-topology] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 07/24/2025 06:37:30.899 7f89cd684a48 [contrail-topology] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 07/24/2025 06:37:30.899 7f89cfa3a948 [contrail-topology] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 07/24/2025 06:37:30.900 7f89cfa3a948 [contrail-topology] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 07/24/2025 06:37:58.103 7f89cfa3aa48 [contrail-topology] [DEBUG]: Waiting for allocation... 07/24/2025 06:38:27.074 7f89cfac3848 [contrail-topology] [INFO]: members: ['cn-jenkins-deploy-platform-ansible-os-3714-1.-1', 'cn-jenkins-deploy-platform-ansible-os-3714-2.-1', 'cn-jenkins-deploy-platform-ansible-os-3714-3.-1'] 07/24/2025 06:38:27.075 7f89cfac3848 [contrail-topology] [INFO]: partitions: ['1', '2', '7', '9', '16', '19', '21', '30', '32', '33', '35', '36', '37', '38', '39', '41', '43', '45'] 07/24/2025 06:38:28.197 7f89cfa3aa48 [contrail-topology] [DEBUG]: got work: ['1', '2', '7', '9', '16', '19', '21', '30', '32', '33', '35', '36', '37', '38', '39', '41', '43', '45'] 07/24/2025 06:38:28.197 7f89cfa3aa48 [contrail-topology] [DEBUG]: @populate_work_items(): done! 07/24/2025 06:38:28.198 7f89cfa3aa48 [contrail-topology] [DEBUG]: work items: [] from the list [] 07/24/2025 06:38:28.198 7f89cfa3aa48 [contrail-topology] [DEBUG]: Add uve in the [ObjectAnalyticsSNMPInfo:TopologyUVE] map 07/24/2025 06:39:27.246 7f89cfa3ab48 [contrail-topology] [ERROR]: Exception OperationTimedOut connecting to Config DB. Arguments: ("errors={'Connection defunct by heartbeat': 'Client request timeout. See Session.execute[_async](timeout)'}, last_host=10.0.0.38:9041",): Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/tf_topology/config_handler.py", line 67, in start zk_servers=self.zk_servers) File "/usr/lib/python3.6/site-packages/cfgm_common/vnc_object_db.py", line 47, in __init__ zk_servers=zk_servers) File "/usr/lib/python3.6/site-packages/cfgm_common/vnc_cassandra.py", line 121, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 197, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/lib/python3.6/site-packages/cfgm_common/datastore/api.py", line 169, in __init__ self._init_cluster() File "/usr/lib/python3.6/site-packages/cfgm_common/datastore/api.py", line 193, in _init_cluster self._Init_Cluster() File "/usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 461, in _Init_Cluster self._get_keyspace_tables(ks, existing_tables) File "/usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 602, in _get_keyspace_tables rows = ses.execute(cql) File "cassandra/cluster.py", line 2618, in cassandra.cluster.Session.execute File "cassandra/cluster.py", line 4901, in cassandra.cluster.ResponseFuture.result cassandra.OperationTimedOut: errors={'Connection defunct by heartbeat': 'Client request timeout. See Session.execute[_async](timeout)'}, last_host=10.0.0.38:9041 07/24/2025 06:39:27.247 7f89d886b898 [contrail-topology] [ERROR]: stopping everything! 07/24/2025 06:39:27.247 7f89d886b898 [contrail-topology] [ERROR]: Stopped http server 07/24/2025 06:39:27.250 7f89d886b898 [contrail-topology] [ERROR]: killing 1 of 2 07/24/2025 06:39:27.250 7f89d886b898 [contrail-topology] [ERROR]: joining 1 of 2 07/24/2025 06:39:27.250 7f89d886b898 [contrail-topology] [ERROR]: stopped 1 of 2 07/24/2025 06:39:27.250 7f89d886b898 [contrail-topology] [ERROR]: killing 2 of 2 07/24/2025 06:39:27.250 7f89d886b898 [contrail-topology] [ERROR]: joining 2 of 2 07/24/2025 06:39:27.250 7f89d886b898 [contrail-topology] [ERROR]: stopped 2 of 2 07/24/2025 06:39:28.296 7f89d886b898 [contrail-topology] [INFO]: Stopping kazooclient successful 07/24/2025 06:39:28.297 7f89d886b898 [contrail-topology] [INFO]: Closing kazooclient successful 07/24/2025 06:39:34.802 7f5be4768898 [contrail-topology] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 07/24/2025 06:39:34.802 7f5be4768898 [contrail-topology] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 07/24/2025 06:39:34.802 7f5be4768898 [contrail-topology] [INFO]: SANDESH: Flow Logging: [None] -> [False] 07/24/2025 06:39:34.802 7f5be4768898 [contrail-topology] [DEBUG]: Add uve in the [ObjectAnalyticsSNMPInfo:NodeStatusUVE] map 07/24/2025 06:39:34.802 7f5be4768898 [contrail-topology] [INFO]: Consistent scheduler zk start 07/24/2025 06:39:34.803 7f5bde493d48 [contrail-topology] [ERROR]: Cannot write http_port 5921 to /tmp/contrail-topology.0.http_port 07/24/2025 06:39:34.803 7f5bde493d48 [contrail-topology] [ERROR]: Starting Introspect on HTTP Port 5921 07/24/2025 06:39:34.803 7f5bdb9bf848 [contrail-topology] [INFO]: Processing event[EvStart] in state[none] 07/24/2025 06:39:34.804 7f5bdb9bf848 [contrail-topology] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 07/24/2025 06:39:34.804 7f5bdb9bf848 [contrail-topology] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 07/24/2025 06:39:34.804 7f5bdb9bf848 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.804 7f5bdb9bf848 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Zookeeper:Zookeeper[Connection to Zookeeper initialized] connection down >>, ] >> 07/24/2025 06:39:34.804 7f5bdb9bf848 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.804 7f5bdb9bf848 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.804 7f5bdb9bf848 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Zookeeper:Zookeeper[Connection to Zookeeper initialized], Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/24/2025 06:39:34.805 7f5bdb9bf848 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.805 7f5bdb9bf848 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.805 7f5bdb9bf848 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3.:AnalyticsSNMP:contrail-topology:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5921 start_time = 1753339174804369 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 2 max_sm_queue_count = 2 >> 07/24/2025 06:39:34.805 7f5bdb9bf848 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.814 7f5bdb939848 [contrail-topology] [INFO]: Consistent scheduler listen CONNECTED 07/24/2025 06:39:34.814 7f5bdb9bf848 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.815 7f5bdb9bf848 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/24/2025 06:39:34.815 7f5bdb9bf848 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.862 7f5bdb939b48 [contrail-topology] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.254:5673//', 'pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 07/24/2025 06:39:34.867 7f5bdb939b48 [contrail-topology] [WARNING]: RabbitMQ drainer connection down 07/24/2025 06:39:34.868 7f5bdb939e48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.869 7f5bdb939e48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 07/24/2025 06:39:34.869 7f5bdb939e48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.869 7f5bdb939e48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.869 7f5bdb939e48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 07/24/2025 06:39:34.869 7f5bdb939e48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.882 7f5bdb939b48 [contrail-topology] [WARNING]: RabbitMQ drainer connection ESTABLISHED 07/24/2025 06:39:34.883 7f5bdb939e48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.883 7f5bdb939e48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/24/2025 06:39:34.883 7f5bdb939e48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.887 7f5bdb939a48 [contrail-topology] [DEBUG]: Waiting for allocation... 07/24/2025 06:39:34.945 7f5bdb939b48 [contrail-topology] [WARNING]: RabbitMQ publish connection down 07/24/2025 06:39:34.947 7f5bdb939d48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.947 7f5bdb939d48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 07/24/2025 06:39:34.947 7f5bdb939d48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.950 7f5bdb939b48 [contrail-topology] [WARNING]: RabbitMQ publish connection ESTABLISHED 07/24/2025 06:39:34.951 7f5bdb939d48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.951 7f5bdb939d48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/24/2025 06:39:34.951 7f5bdb939d48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.952 7f5bdb939b48 [contrail-topology] [WARNING]: VNCCassandra started with driver cql 07/24/2025 06:39:34.952 7f5bdb939b48 [contrail-topology] [WARNING]: initializing cassandra cluster 07/24/2025 06:39:34.952 7f5bdb939b48 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster before create 07/24/2025 06:39:34.952 7f5bdb939d48 [contrail-topology] [WARNING]: RabbitMQ publish connection down 07/24/2025 06:39:34.953 7f5bd9945948 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.953 7f5bd9945948 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 07/24/2025 06:39:34.953 7f5bd9945948 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.953 7f5bd9945948 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.953 7f5bd9945948 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 07/24/2025 06:39:34.953 7f5bd9945948 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.954 7f5bdb939b48 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster created 07/24/2025 06:39:34.959 7f5bdb939d48 [contrail-topology] [WARNING]: RabbitMQ publish connection ESTABLISHED 07/24/2025 06:39:34.960 7f5bd9578048 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.960 7f5bd9578048 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 07/24/2025 06:39:34.960 7f5bd9578048 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.984 7f5bdb939b48 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 07/24/2025 06:39:34.986 7f5bd9578b48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.986 7f5bd9578b48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Initializing description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[], Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 07/24/2025 06:39:34.986 7f5bd9578b48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.992 7f5bdb939b48 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 07/24/2025 06:39:34.993 7f5bd9578b48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:34.993 7f5bd9578b48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 07/24/2025 06:39:34.993 7f5bd9578b48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:35.001 7f5bdb939b48 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 07/24/2025 06:39:35.031 7f5bdb939b48 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 07/24/2025 06:39:35.032 7f5bd94da548 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:35.032 7f5bd94da548 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/24/2025 06:39:35.032 7f5bd94da548 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/24/2025 06:39:38.806 7f5bd94da748 [contrail-topology] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/24/2025 06:39:38.808 7f5bd94da748 [contrail-topology] [INFO]: Session Event: TCP Connected 07/24/2025 06:39:38.808 7f5bd94da748 [contrail-topology] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/24/2025 06:39:38.808 7f5bd94da748 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/24/2025 06:39:38.808 7f5bd94da748 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3714-3. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/24/2025 06:39:38.808 7f5bd94da748 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/24/2025 06:39:38.809 7f5bd94da748 [contrail-topology] [INFO]: Processing event[EvTcpConnected] in state[Connect] 07/24/2025 06:39:38.809 7f5bd94da748 [contrail-topology] [DEBUG]: Send sandesh control message. uve type count # 5 07/24/2025 06:39:38.809 7f5bd94da748 [contrail-topology] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 07/24/2025 06:39:38.810 7f5bd94da748 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 07/24/2025 06:39:38.810 7f5bd94da748 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 07/24/2025 06:39:38.810 7f5bd94da948 [contrail-topology] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 07/24/2025 06:39:38.812 7f5bd94da948 [contrail-topology] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 07/24/2025 06:39:38.817 7f5bd94da848 [contrail-topology] [DEBUG]: Extracted sandesh length: 0000001017 07/24/2025 06:39:38.819 7f5bd94da848 [contrail-topology] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 07/24/2025 06:39:38.819 7f5bd94da948 [contrail-topology] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 07/24/2025 06:39:38.819 7f5bd94da948 [contrail-topology] [DEBUG]: Number of uve types in sandesh control message is 0 07/24/2025 06:39:38.820 7f5bd94da948 [contrail-topology] [DEBUG]: send sync_uve in the [ObjectAnalyticsSNMPInfo:NodeStatusUVE] map 07/24/2025 06:39:38.820 7f5bd94da948 [contrail-topology] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 07/24/2025 06:39:38.820 7f5bd94da948 [contrail-topology] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 07/24/2025 06:39:38.821 7f5bdb939948 [contrail-topology] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 07/24/2025 06:39:38.823 7f5bdb939948 [contrail-topology] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 07/24/2025 06:39:58.922 7f5bdb939e48 [contrail-topology] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3714-2'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-3714-2', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3714-2'], 'id_perms': {'created': '2025-07-24T06:39:58.916383', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-07-24T06:39:58.916383', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10309367934322498758, 'uuid_mslong': 11563772748602754383}}, 'parent_type': 'global-system-config', 'parent_uuid': 'cca7bcae-f3dd-4ef2-bd5f-2708a1f6d994', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a07ac60c-1fcf-454f-8f12-3b83d0185cc6', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.193'}, 'oper': 'CREATE', 'request-id': 'req-fcd1b71a-106c-4179-801e-962bea1f4c51', 'type': 'virtual_router', 'uuid': 'a07ac60c-1fcf-454f-8f12-3b83d0185cc6'} 07/24/2025 06:39:59.360 7f5bdb939e48 [contrail-topology] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3714-2', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3714-2', 'vhost0'], 'id_perms': {'created': '2025-07-24T06:39:59.335889', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-07-24T06:39:59.335889', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12846124812883995019, 'uuid_mslong': 9243824517073423121}}, 'parent_type': 'virtual-router', 'parent_uuid': 'a07ac60c-1fcf-454f-8f12-3b83d0185cc6', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '8048a96c-2235-4b11-b246-9a74a448298b', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:80:48:a9:6c:22']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': '849f7cc7-7836-4f32-a7bb-d62b962f9490'}]}, 'oper': 'CREATE', 'request-id': 'req-c60da313-d4db-43a2-9f82-6c091840e08e', 'type': 'virtual_machine_interface', 'uuid': '8048a96c-2235-4b11-b246-9a74a448298b'} 07/24/2025 06:39:59.430 7f5bdb939e48 [contrail-topology] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3714-2', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-c60da313-d4db-43a2-9f82-6c091840e08e', 'type': 'virtual_machine_interface', 'uuid': '8048a96c-2235-4b11-b246-9a74a448298b'} 07/24/2025 06:40:04.875 7f5bdb9bf848 [contrail-topology] [INFO]: members: ['cn-jenkins-deploy-platform-ansible-os-3714-1.-1', 'cn-jenkins-deploy-platform-ansible-os-3714-2.-1', 'cn-jenkins-deploy-platform-ansible-os-3714-3.-1'] 07/24/2025 06:40:04.876 7f5bdb9bf848 [contrail-topology] [INFO]: partitions: ['1', '2', '7', '9', '16', '19', '21', '30', '32', '33', '35', '36', '37', '38', '39', '41', '43', '45'] 07/24/2025 06:40:05.901 7f5bdb939a48 [contrail-topology] [DEBUG]: got work: ['1', '2', '7', '9', '16', '19', '21', '30', '32', '33', '35', '36', '37', '38', '39', '41', '43', '45'] 07/24/2025 06:40:05.901 7f5bdb939a48 [contrail-topology] [DEBUG]: @populate_work_items(): done! 07/24/2025 06:40:05.902 7f5bdb939a48 [contrail-topology] [DEBUG]: work items: [] from the list [] 07/24/2025 06:40:05.902 7f5bdb939a48 [contrail-topology] [DEBUG]: Add uve in the [ObjectAnalyticsSNMPInfo:TopologyUVE] map 07/24/2025 06:40:51.659 7f5bdb9bf848 [contrail-topology] [DEBUG]: send sync_uve in the [ObjectAnalyticsSNMPInfo:NodeStatusUVE] map 07/24/2025 06:41:02.864 7f5bdb9bf848 [contrail-topology] [DEBUG]: send sync_uve in the [ObjectAnalyticsSNMPInfo:NodeStatusUVE] map 07/24/2025 06:43:09.261 7f5bdb9bf848 [contrail-topology] [DEBUG]: send sync_uve in the [ObjectAnalyticsSNMPInfo:NodeStatusUVE] map