06/05/2025 00:45:19.640 7f0670f3b898 [contrail-topology] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0]
06/05/2025 00:45:19.641 7f0670f3b898 [contrail-topology] [INFO]: SANDESH: Trace: PRINT: [None] -> [False]
06/05/2025 00:45:19.641 7f0670f3b898 [contrail-topology] [INFO]: SANDESH: Flow Logging: [None] -> [False]
06/05/2025 00:45:19.641 7f0670f3b898 [contrail-topology] [DEBUG]: Add uve <cn-jenkins-deploy-platform-ansible-os-3401-2., 1> in the [ObjectAnalyticsSNMPInfo:NodeStatusUVE] map
06/05/2025 00:45:19.641 7f0670f3b898 [contrail-topology] [INFO]: Consistent scheduler zk start
06/05/2025 00:45:19.642 7f066ac65d48 [contrail-topology] [ERROR]: Cannot write http_port 5921 to /tmp/contrail-topology.0.http_port
06/05/2025 00:45:19.642 7f066ac65d48 [contrail-topology] [ERROR]: Starting Introspect on HTTP Port 5921
06/05/2025 00:45:19.642 7f06681d2848 [contrail-topology] [INFO]: Processing event[EvStart] in state[none]
06/05/2025 00:45:19.643 7f06681d2848 [contrail-topology] [DEBUG]: Add uve <cn-jenkins-deploy-platform-ansible-os-3401-2.:AnalyticsSNMP:contrail-topology:0, 1> in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map
06/05/2025 00:45:19.643 7f06681d2848 [contrail-topology] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle]
06/05/2025 00:45:19.643 7f06681d2848 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.643 7f06681d2848 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-ansible-os-3401-2.  process_status = [ <<  module_id = contrail-topology  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.249:2181, 10.0.0.27:2181, 10.0.0.131:2181,  ]  status = Initializing  description = Connection to Zookeeper initialized  >>,  ]  description = Zookeeper:Zookeeper[Connection to Zookeeper initialized] connection down  >>,  ]  >>  
06/05/2025 00:45:19.643 7f06681d2848 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.643 7f06681d2848 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.643 7f06681d2848 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-ansible-os-3401-2.  process_status = [ <<  module_id = contrail-topology  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.249:2181, 10.0.0.27:2181, 10.0.0.131:2181,  ]  status = Initializing  description = Connection to Zookeeper initialized  >>, <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>,  ]  description = Zookeeper:Zookeeper[Connection to Zookeeper initialized], Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
06/05/2025 00:45:19.643 7f06681d2848 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.643 7f06681d2848 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.643 7f06681d2848 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = <<  name = cn-jenkins-deploy-platform-ansible-os-3401-2.:AnalyticsSNMP:contrail-topology:0  client_info = <<  status = Idle  successful_connections = 0  pid = 1  http_port = 5921  start_time = 1749084319643079  collector_name =   collector_ip =   collector_list = [ 10.0.0.249:8086, 10.0.0.131:8086, 10.0.0.27:8086,  ]  >>  sm_queue_count = 2  max_sm_queue_count = 2  >>  
06/05/2025 00:45:19.643 7f06681d2848 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.649 7f066810a848 [contrail-topology] [INFO]: Consistent scheduler listen CONNECTED
06/05/2025 00:45:19.649 7f06681d2848 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.650 7f06681d2848 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-ansible-os-3401-2.  process_status = [ <<  module_id = contrail-topology  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.249:2181, 10.0.0.27:2181, 10.0.0.131:2181,  ]  status = Up  description = Connection to Zookeeper established  >>, <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
06/05/2025 00:45:19.650 7f06681d2848 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.735 7f066810ab48 [contrail-topology] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.249:5673//', 'pyamqp://guest:guest@10.0.0.27:5673//', 'pyamqp://guest:guest@10.0.0.131:5673//']
06/05/2025 00:45:19.740 7f066810ab48 [contrail-topology] [WARNING]: RabbitMQ drainer connection down
06/05/2025 00:45:19.741 7f066810ae48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.741 7f066810ae48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-ansible-os-3401-2.  process_status = [ <<  module_id = contrail-topology  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.249:2181, 10.0.0.27:2181, 10.0.0.131:2181,  ]  status = Up  description = Connection to Zookeeper established  >>, <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.249:5673, 10.0.0.27:5673, 10.0.0.131:5673,  ]  status = Initializing  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down  >>,  ]  >>  
06/05/2025 00:45:19.741 7f066810ae48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.741 7f066810ae48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.741 7f066810ae48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-ansible-os-3401-2.  process_status = [ <<  module_id = contrail-topology  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.249:2181, 10.0.0.27:2181, 10.0.0.131:2181,  ]  status = Up  description = Connection to Zookeeper established  >>, <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.249:5673, 10.0.0.27:5673, 10.0.0.131:5673,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down  >>,  ]  >>  
06/05/2025 00:45:19.741 7f066810ae48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.750 7f066810ab48 [contrail-topology] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.249:5673// at 0x7f0665dd2eb8>
06/05/2025 00:45:19.751 7f066810ae48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.751 7f066810ae48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-ansible-os-3401-2.  process_status = [ <<  module_id = contrail-topology  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.249:2181, 10.0.0.27:2181, 10.0.0.131:2181,  ]  status = Up  description = Connection to Zookeeper established  >>, <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.249:5673, 10.0.0.27:5673, 10.0.0.131:5673,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
06/05/2025 00:45:19.751 7f066810ae48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.764 7f066810aa48 [contrail-topology] [DEBUG]: Waiting for allocation...
06/05/2025 00:45:19.794 7f066810ab48 [contrail-topology] [WARNING]: RabbitMQ publish connection down
06/05/2025 00:45:19.795 7f066810ac48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.796 7f066810ac48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-ansible-os-3401-2.  process_status = [ <<  module_id = contrail-topology  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.249:2181, 10.0.0.27:2181, 10.0.0.131:2181,  ]  status = Up  description = Connection to Zookeeper established  >>, <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.249:5673, 10.0.0.27:5673, 10.0.0.131:5673,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down  >>,  ]  >>  
06/05/2025 00:45:19.796 7f066810ac48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.800 7f066810ab48 [contrail-topology] [WARNING]: RabbitMQ publish connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.249:5673// at 0x7f0665dd55f8>
06/05/2025 00:45:19.800 7f066810ac48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.800 7f066810ac48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-ansible-os-3401-2.  process_status = [ <<  module_id = contrail-topology  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.249:2181, 10.0.0.27:2181, 10.0.0.131:2181,  ]  status = Up  description = Connection to Zookeeper established  >>, <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.249:5673, 10.0.0.27:5673, 10.0.0.131:5673,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
06/05/2025 00:45:19.800 7f066810ac48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.801 7f066810ab48 [contrail-topology] [WARNING]: VNCCassandra started with driver cql
06/05/2025 00:45:19.801 7f066810ab48 [contrail-topology] [WARNING]: initializing cassandra cluster
06/05/2025 00:45:19.801 7f066810ab48 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster before create
06/05/2025 00:45:19.801 7f066810ac48 [contrail-topology] [WARNING]: RabbitMQ publish connection down
06/05/2025 00:45:19.802 7f066810ae48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.802 7f066810ae48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-ansible-os-3401-2.  process_status = [ <<  module_id = contrail-topology  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.249:2181, 10.0.0.27:2181, 10.0.0.131:2181,  ]  status = Up  description = Connection to Zookeeper established  >>, <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.249:5673, 10.0.0.27:5673, 10.0.0.131:5673,  ]  status = Up  description =   >>, <<  type = Database  name = Cassandra  server_addrs = [ 10.0.0.249:9041, 10.0.0.27:9041, 10.0.0.131:9041,  ]  status = Initializing  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down  >>,  ]  >>  
06/05/2025 00:45:19.802 7f066810ae48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.802 7f066810ae48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.803 7f066810ae48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-ansible-os-3401-2.  process_status = [ <<  module_id = contrail-topology  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.249:2181, 10.0.0.27:2181, 10.0.0.131:2181,  ]  status = Up  description = Connection to Zookeeper established  >>, <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.249:5673, 10.0.0.27:5673, 10.0.0.131:5673,  ]  status = Down  description =   >>, <<  type = Database  name = Cassandra  server_addrs = [ 10.0.0.249:9041, 10.0.0.27:9041, 10.0.0.131:9041,  ]  status = Initializing  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[], Database:Cassandra[] connection down  >>,  ]  >>  
06/05/2025 00:45:19.803 7f066810ae48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.805 7f066810ab48 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster created
06/05/2025 00:45:19.809 7f066810ac48 [contrail-topology] [WARNING]: RabbitMQ publish connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.249:5673// at 0x7f0665dd55f8>
06/05/2025 00:45:19.809 7f0665d4a048 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.810 7f0665d4a048 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-ansible-os-3401-2.  process_status = [ <<  module_id = contrail-topology  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.249:2181, 10.0.0.27:2181, 10.0.0.131:2181,  ]  status = Up  description = Connection to Zookeeper established  >>, <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.249:5673, 10.0.0.27:5673, 10.0.0.131:5673,  ]  status = Up  description =   >>, <<  type = Database  name = Cassandra  server_addrs = [ 10.0.0.249:9041, 10.0.0.27:9041, 10.0.0.131:9041,  ]  status = Initializing  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down  >>,  ]  >>  
06/05/2025 00:45:19.810 7f0665d4a048 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.840 7f066810ab48 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected
06/05/2025 00:45:19.841 7f0665d4c948 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.841 7f0665d4c948 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-ansible-os-3401-2.  process_status = [ <<  module_id = contrail-topology  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.249:2181, 10.0.0.27:2181, 10.0.0.131:2181,  ]  status = Initializing  description =   >>, <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.249:5673, 10.0.0.27:5673, 10.0.0.131:5673,  ]  status = Up  description =   >>, <<  type = Database  name = Cassandra  server_addrs = [ 10.0.0.249:9041, 10.0.0.27:9041, 10.0.0.131:9041,  ]  status = Initializing  description =   >>,  ]  description = Zookeeper:Zookeeper[], Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down  >>,  ]  >>  
06/05/2025 00:45:19.842 7f0665d4c948 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.847 7f066810ab48 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster init KS
06/05/2025 00:45:19.847 7f0665d4c648 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.847 7f0665d4c648 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-ansible-os-3401-2.  process_status = [ <<  module_id = contrail-topology  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.249:2181, 10.0.0.27:2181, 10.0.0.131:2181,  ]  status = Up  description =   >>, <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.249:5673, 10.0.0.27:5673, 10.0.0.131:5673,  ]  status = Up  description =   >>, <<  type = Database  name = Cassandra  server_addrs = [ 10.0.0.249:9041, 10.0.0.27:9041, 10.0.0.131:9041,  ]  status = Initializing  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down  >>,  ]  >>  
06/05/2025 00:45:19.847 7f0665d4c648 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.850 7f066810ab48 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready
06/05/2025 00:45:19.868 7f066810ab48 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready
06/05/2025 00:45:19.868 7f0665caab48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:19.868 7f0665caab48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-ansible-os-3401-2.  process_status = [ <<  module_id = contrail-topology  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.249:2181, 10.0.0.27:2181, 10.0.0.131:2181,  ]  status = Up  description =   >>, <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.249:5673, 10.0.0.27:5673, 10.0.0.131:5673,  ]  status = Up  description =   >>, <<  type = Database  name = Cassandra  server_addrs = [ 10.0.0.249:9041, 10.0.0.27:9041, 10.0.0.131:9041,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
06/05/2025 00:45:19.868 7f0665caab48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle]
06/05/2025 00:45:23.644 7f0665caab48 [contrail-topology] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle]
06/05/2025 00:45:23.645 7f0665caab48 [contrail-topology] [INFO]: Session Event: TCP Connected
06/05/2025 00:45:23.645 7f0665caab48 [contrail-topology] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect]
06/05/2025 00:45:23.645 7f0665caab48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Connect]
06/05/2025 00:45:23.645 7f0665caab48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-ansible-os-3401-2.  process_status = [ <<  module_id = contrail-topology  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.249:2181, 10.0.0.27:2181, 10.0.0.131:2181,  ]  status = Up  description =   >>, <<  type = Collector  name = Collector  server_addrs = [ 10.0.0.249:8086,  ]  status = Initializing  description = Idle to Connect on EvIdleHoldTimerExpired  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.249:5673, 10.0.0.27:5673, 10.0.0.131:5673,  ]  status = Up  description =   >>, <<  type = Database  name = Cassandra  server_addrs = [ 10.0.0.249:9041, 10.0.0.27:9041, 10.0.0.131:9041,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down  >>,  ]  >>  
06/05/2025 00:45:23.645 7f0665caab48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect]
06/05/2025 00:45:23.645 7f0665caab48 [contrail-topology] [INFO]: Processing event[EvTcpConnected] in state[Connect]
06/05/2025 00:45:23.645 7f0665caab48 [contrail-topology] [DEBUG]: Send sandesh control message. uve type count # 5
06/05/2025 00:45:23.646 7f0665caab48 [contrail-topology] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit]
06/05/2025 00:45:23.646 7f0665caab48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit]
06/05/2025 00:45:23.646 7f0665caab48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit]
06/05/2025 00:45:23.646 7f0665caad48 [contrail-topology] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG]
06/05/2025 00:45:23.647 7f0665caad48 [contrail-topology] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID]
06/05/2025 00:45:23.656 7f0665caac48 [contrail-topology] [DEBUG]: Extracted sandesh length: 0000001017
06/05/2025 00:45:23.656 7f0665caac48 [contrail-topology] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient]
06/05/2025 00:45:23.656 7f0665caad48 [contrail-topology] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit]
06/05/2025 00:45:23.656 7f0665caad48 [contrail-topology] [DEBUG]: Number of uve types in sandesh control message is 0
06/05/2025 00:45:23.656 7f0665caad48 [contrail-topology] [DEBUG]: send sync_uve <cn-jenkins-deploy-platform-ansible-os-3401-2.: 16> in the [ObjectAnalyticsSNMPInfo:NodeStatusUVE] map
06/05/2025 00:45:23.657 7f0665caad48 [contrail-topology] [DEBUG]: send sync_uve <cn-jenkins-deploy-platform-ansible-os-3401-2.:AnalyticsSNMP:contrail-topology:0: 2> in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map
06/05/2025 00:45:23.657 7f0665caad48 [contrail-topology] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established]
06/05/2025 00:45:23.657 7f066810a948 [contrail-topology] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG]
06/05/2025 00:45:23.658 7f066810a948 [contrail-topology] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID]
06/05/2025 00:45:50.781 7f066810aa48 [contrail-topology] [DEBUG]: Waiting for allocation...
06/05/2025 00:46:19.750 7f06681d2848 [contrail-topology] [INFO]: members: ['cn-jenkins-deploy-platform-ansible-os-3401-1.-1', 'cn-jenkins-deploy-platform-ansible-os-3401-3.-1', 'cn-jenkins-deploy-platform-ansible-os-3401-2.-1']
06/05/2025 00:46:19.752 7f06681d2848 [contrail-topology] [INFO]: partitions: ['0', '8', '10', '12', '13', '14', '16', '17', '23', '24', '25', '27', '33', '39', '46']
06/05/2025 00:46:20.899 7f066810aa48 [contrail-topology] [DEBUG]: got work:  ['0', '8', '10', '12', '13', '14', '16', '17', '23', '24', '25', '27', '33', '39', '46']
06/05/2025 00:46:20.899 7f066810aa48 [contrail-topology] [DEBUG]: @populate_work_items(): done!
06/05/2025 00:46:20.900 7f066810aa48 [contrail-topology] [DEBUG]: work items:  [] from the list []
06/05/2025 00:46:20.900 7f066810aa48 [contrail-topology] [DEBUG]: Add uve <cn-jenkins-deploy-platform-ansible-os-3401-2., 1> in the [ObjectAnalyticsSNMPInfo:TopologyUVE] map
06/05/2025 00:46:55.513 7f066810ad48 [contrail-topology] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config'],
 'oper': 'UPDATE',
 'request-id': 'req-f6a17767-f471-4b98-ba91-1cfdf4b638e1',
 'type': 'global_system_config',
 'uuid': '7685c044-9f7d-4d52-9039-0babc11f3849'}
06/05/2025 00:46:55.655 7f066810ad48 [contrail-topology] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config',
             'an-jenkins-deploy-platform-ansible-os-3401-2'],
 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-3401-2',
              'fq_name': ['default-global-system-config',
                          'an-jenkins-deploy-platform-ansible-os-3401-2'],
              'id_perms': {'created': '2025-06-05T00:46:55.640517',
                           'creator': None,
                           'description': None,
                           'enable': True,
                           'last_modified': '2025-06-05T00:46:55.640517',
                           'permissions': {'group': 'heat_stack_owner',
                                           'group_access': 7,
                                           'other_access': 7,
                                           'owner': 'admin',
                                           'owner_access': 7},
                           'user_visible': True,
                           'uuid': {'uuid_lslong': 11130148355826326220,
                                    'uuid_mslong': 3599767279899134208}},
              'parent_type': 'global-system-config',
              'parent_uuid': '7685c044-9f7d-4d52-9039-0babc11f3849',
              'perms2': {'global_access': 0,
                         'owner': 'cloud-admin',
                         'owner_access': 7,
                         'share': []},
              'uuid': '31f4f144-c876-4900-9a76-3af5a66d1acc',
              'virtual_router_dpdk_enabled': False,
              'virtual_router_ip_address': '10.20.0.193'},
 'oper': 'CREATE',
 'request-id': 'req-1f3bc652-69e3-4db1-b979-e6df4fc3a3e8',
 'type': 'virtual_router',
 'uuid': '31f4f144-c876-4900-9a76-3af5a66d1acc'}
06/05/2025 00:46:55.994 7f066810ad48 [contrail-topology] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config',
             'an-jenkins-deploy-platform-ansible-os-3401-2',
             'vhost0'],
 'obj_dict': {'display_name': 'vhost0',
              'fq_name': ['default-global-system-config',
                          'an-jenkins-deploy-platform-ansible-os-3401-2',
                          'vhost0'],
              'id_perms': {'created': '2025-06-05T00:46:55.976693',
                           'creator': None,
                           'description': None,
                           'enable': True,
                           'last_modified': '2025-06-05T00:46:55.976693',
                           'permissions': {'group': 'heat_stack_owner',
                                           'group_access': 7,
                                           'other_access': 7,
                                           'owner': 'admin',
                                           'owner_access': 7},
                           'user_visible': True,
                           'uuid': {'uuid_lslong': 13089009622088256425,
                                    'uuid_mslong': 5912294089183284161}},
              'parent_type': 'virtual-router',
              'parent_uuid': '31f4f144-c876-4900-9a76-3af5a66d1acc',
              'perms2': {'global_access': 0,
                         'owner': 'cloud-admin',
                         'owner_access': 7,
                         'share': []},
              'uuid': '520cb026-4169-4bc1-b5a5-80e71a3d8ba9',
              'virtual_machine_interface_disable_policy': True,
              'virtual_machine_interface_mac_addresses': {'mac_address': ['02:52:0c:b0:26:41']},
              'virtual_network_refs': [{'to': ['default-domain',
                                               'default-project',
                                               'ip-fabric'],
                                        'uuid': '98a68765-400c-42d7-849e-e430b8719b6d'}]},
 'oper': 'CREATE',
 'request-id': 'req-66a36709-af65-4e2d-bf1d-31cdbf05ac16',
 'type': 'virtual_machine_interface',
 'uuid': '520cb026-4169-4bc1-b5a5-80e71a3d8ba9'}
06/05/2025 00:46:56.035 7f066810ad48 [contrail-topology] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config',
             'an-jenkins-deploy-platform-ansible-os-3401-2',
             'vhost0'],
 'oper': 'UPDATE',
 'request-id': 'req-66a36709-af65-4e2d-bf1d-31cdbf05ac16',
 'type': 'virtual_machine_interface',
 'uuid': '520cb026-4169-4bc1-b5a5-80e71a3d8ba9'}
06/05/2025 00:46:56.290 7f066810ad48 [contrail-topology] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config',
             'an-jenkins-deploy-platform-ansible-os-3401-1'],
 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-3401-1',
              'fq_name': ['default-global-system-config',
                          'an-jenkins-deploy-platform-ansible-os-3401-1'],
              'id_perms': {'created': '2025-06-05T00:46:56.284111',
                           'creator': None,
                           'description': None,
                           'enable': True,
                           'last_modified': '2025-06-05T00:46:56.284111',
                           'permissions': {'group': 'heat_stack_owner',
                                           'group_access': 7,
                                           'other_access': 7,
                                           'owner': 'admin',
                                           'owner_access': 7},
                           'user_visible': True,
                           'uuid': {'uuid_lslong': 11857002851222459896,
                                    'uuid_mslong': 8416889699372126042}},
              'parent_type': 'global-system-config',
              'parent_uuid': '7685c044-9f7d-4d52-9039-0babc11f3849',
              'perms2': {'global_access': 0,
                         'owner': 'cloud-admin',
                         'owner_access': 7,
                         'share': []},
              'uuid': '74cecc96-44b8-4f5a-a48c-8945a7e895f8',
              'virtual_router_dpdk_enabled': False,
              'virtual_router_ip_address': '10.20.0.25'},
 'oper': 'CREATE',
 'request-id': 'req-9bcac08a-12cb-46dd-a586-79468cd30018',
 'type': 'virtual_router',
 'uuid': '74cecc96-44b8-4f5a-a48c-8945a7e895f8'}
06/05/2025 00:46:56.546 7f066810ad48 [contrail-topology] [DEBUG]: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'],
 'oper': 'UPDATE',
 'request-id': 'req-f38a07f0-f8bf-4172-bfdc-b43bdd388bee',
 'type': 'access_control_list',
 'uuid': '09df5933-56a8-4624-bb39-6f82cee0bcf8'}
06/05/2025 00:46:56.628 7f066810ad48 [contrail-topology] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config',
             'an-jenkins-deploy-platform-ansible-os-3401-1',
             'vhost0'],
 'obj_dict': {'display_name': 'vhost0',
              'fq_name': ['default-global-system-config',
                          'an-jenkins-deploy-platform-ansible-os-3401-1',
                          'vhost0'],
              'id_perms': {'created': '2025-06-05T00:46:56.613495',
                           'creator': None,
                           'description': None,
                           'enable': True,
                           'last_modified': '2025-06-05T00:46:56.613495',
                           'permissions': {'group': 'heat_stack_owner',
                                           'group_access': 7,
                                           'other_access': 7,
                                           'owner': 'admin',
                                           'owner_access': 7},
                           'user_visible': True,
                           'uuid': {'uuid_lslong': 13466647569305989790,
                                    'uuid_mslong': 15917060270280624010}},
              'parent_type': 'virtual-router',
              'parent_uuid': '74cecc96-44b8-4f5a-a48c-8945a7e895f8',
              'perms2': {'global_access': 0,
                         'owner': 'cloud-admin',
                         'owner_access': 7,
                         'share': []},
              'uuid': 'dce4c1fb-ce8f-478a-bae3-249d6bdc729e',
              'virtual_machine_interface_disable_policy': True,
              'virtual_machine_interface_mac_addresses': {'mac_address': ['02:dc:e4:c1:fb:ce']},
              'virtual_network_refs': [{'to': ['default-domain',
                                               'default-project',
                                               'ip-fabric'],
                                        'uuid': '98a68765-400c-42d7-849e-e430b8719b6d'}]},
 'oper': 'CREATE',
 'request-id': 'req-cc9ea0c7-21c7-4552-a588-ccfccdf1ac8b',
 'type': 'virtual_machine_interface',
 'uuid': 'dce4c1fb-ce8f-478a-bae3-249d6bdc729e'}
06/05/2025 00:46:56.671 7f066810ad48 [contrail-topology] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config',
             'an-jenkins-deploy-platform-ansible-os-3401-1',
             'vhost0'],
 'oper': 'UPDATE',
 'request-id': 'req-cc9ea0c7-21c7-4552-a588-ccfccdf1ac8b',
 'type': 'virtual_machine_interface',
 'uuid': 'dce4c1fb-ce8f-478a-bae3-249d6bdc729e'}
06/05/2025 00:47:32.169 7f06681d2848 [contrail-topology] [DEBUG]: send sync_uve <cn-jenkins-deploy-platform-ansible-os-3401-2.: 17> in the [ObjectAnalyticsSNMPInfo:NodeStatusUVE] map
06/05/2025 00:47:54.917 7f06681d2848 [contrail-topology] [DEBUG]: send sync_uve <cn-jenkins-deploy-platform-ansible-os-3401-2.: 17> in the [ObjectAnalyticsSNMPInfo:NodeStatusUVE] map
06/05/2025 00:56:20.913 7f066810aa48 [contrail-topology] [DEBUG]: got work:  ['0', '8', '10', '12', '13', '14', '16', '17', '23', '24', '25', '27', '33', '39', '46']
06/05/2025 00:56:20.914 7f066810aa48 [contrail-topology] [DEBUG]: @populate_work_items(): done!
06/05/2025 00:56:20.914 7f066810aa48 [contrail-topology] [DEBUG]: work items:  [] from the list []
06/05/2025 01:06:20.928 7f066810aa48 [contrail-topology] [DEBUG]: got work:  ['0', '8', '10', '12', '13', '14', '16', '17', '23', '24', '25', '27', '33', '39', '46']
06/05/2025 01:06:20.928 7f066810aa48 [contrail-topology] [DEBUG]: @populate_work_items(): done!
06/05/2025 01:06:20.928 7f066810aa48 [contrail-topology] [DEBUG]: work items:  [] from the list []
06/05/2025 01:16:20.941 7f066810aa48 [contrail-topology] [DEBUG]: got work:  ['0', '8', '10', '12', '13', '14', '16', '17', '23', '24', '25', '27', '33', '39', '46']
06/05/2025 01:16:20.942 7f066810aa48 [contrail-topology] [DEBUG]: @populate_work_items(): done!
06/05/2025 01:16:20.942 7f066810aa48 [contrail-topology] [DEBUG]: work items:  [] from the list []
06/05/2025 01:26:20.954 7f066810aa48 [contrail-topology] [DEBUG]: got work:  ['0', '8', '10', '12', '13', '14', '16', '17', '23', '24', '25', '27', '33', '39', '46']
06/05/2025 01:26:20.955 7f066810aa48 [contrail-topology] [DEBUG]: @populate_work_items(): done!
06/05/2025 01:26:20.955 7f066810aa48 [contrail-topology] [DEBUG]: work items:  [] from the list []
06/05/2025 01:36:20.966 7f066810aa48 [contrail-topology] [DEBUG]: got work:  ['0', '8', '10', '12', '13', '14', '16', '17', '23', '24', '25', '27', '33', '39', '46']
06/05/2025 01:36:20.967 7f066810aa48 [contrail-topology] [DEBUG]: @populate_work_items(): done!
06/05/2025 01:36:20.967 7f066810aa48 [contrail-topology] [DEBUG]: work items:  [] from the list []
06/05/2025 01:44:39.280 7f06681d2848 [contrail-topology] [DEBUG]: send sync_uve <cn-jenkins-deploy-platform-ansible-os-3401-2.: 17> in the [ObjectAnalyticsSNMPInfo:NodeStatusUVE] map