08/21/2025 01:52:12.892 7fab0f97a348 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 08/21/2025 01:52:12.892 7fab0f97a348 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 08/21/2025 01:52:12.892 7fab0f97a348 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 08/21/2025 01:52:12.892 7fab070602e8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 08/21/2025 01:52:12.893 7fab070602e8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 08/21/2025 01:52:12.893 7fab070602e8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 08/21/2025 01:52:12.893 7fab070602e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 08/21/2025 01:52:12.893 7fab070602e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:12.893 7fab070602e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 08/21/2025 01:52:12.893 7fab070602e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:12.893 7fab070602e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:12.894 7fab070602e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 start_time = 1755741132893273 collector_name = collector_ip = collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 08/21/2025 01:52:12.894 7fab070602e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:12.939 7fab0f97a348 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.27:5673//', 'pyamqp://guest:guest@10.0.0.249:5673//'] 08/21/2025 01:52:12.947 7fab0f97a348 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 08/21/2025 01:52:12.947 7fab06fb3588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:12.948 7fab06fb3588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 08/21/2025 01:52:12.948 7fab06fb3588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:12.948 7fab06fb3588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:12.948 7fab06fb3588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 08/21/2025 01:52:12.948 7fab06fb3588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:12.951 7fab0f97a348 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 08/21/2025 01:52:12.960 7fab06fb3240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:12.960 7fab06fb3240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 08/21/2025 01:52:12.960 7fab06fb3240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:12.997 7fab0f97a348 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 08/21/2025 01:52:12.997 7fab06fb30f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:12.997 7fab06fb30f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 08/21/2025 01:52:12.998 7fab06fb30f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:13.000 7fab0f97a348 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 08/21/2025 01:52:13.001 7fab15865978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:13.001 7fab15865978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 08/21/2025 01:52:13.001 7fab15865978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:13.013 7fab0f97a348 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 08/21/2025 01:52:13.014 7fab0f97a348 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 08/21/2025 01:52:13.014 7fab0f97a348 [contrail-device-manager] [WARNING]: initializing cassandra cluster 08/21/2025 01:52:13.014 7fab0f97a348 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 08/21/2025 01:52:13.014 7fab0c8eb6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 08/21/2025 01:52:13.015 7fab0c8eb6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 08/21/2025 01:52:13.015 7fab097fd978 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 08/21/2025 01:52:13.015 7fab097fd978 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 08/21/2025 01:52:13.015 7fab06e4c588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:13.015 7fab06e4c588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 08/21/2025 01:52:13.015 7fab06e4c588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:13.017 7fab0f97a348 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 08/21/2025 01:52:13.018 7fab0c8eb6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 08/21/2025 01:52:13.019 7fab0c8eb6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 08/21/2025 01:52:13.019 7fab097fd978 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 08/21/2025 01:52:13.059 7fab0f97a348 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 08/21/2025 01:52:13.060 7fab0656c048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:13.060 7fab0656c048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 08/21/2025 01:52:13.060 7fab0656c048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:13.071 7fab0f97a348 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 08/21/2025 01:52:13.071 7fab06da62e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:13.071 7fab06da62e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 08/21/2025 01:52:13.071 7fab06da62e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:14.079 7fab0f97a348 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 08/21/2025 01:52:14.087 7fab0f97a348 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 08/21/2025 01:52:14.119 7fab0f97a348 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 08/21/2025 01:52:14.821 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/21/2025 01:52:15.634 7fab0f97a348 [contrail-device-manager] [WARNING]: table 'obj_uuid_table', created 08/21/2025 01:52:15.641 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/21/2025 01:52:16.893 7fab06539c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:52:16.894 7fab06539c18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:52:16.894 7fab06539c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:52:16.894 7fab06539c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:52:16.894 7fab06539c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 08/21/2025 01:52:16.894 7fab06539c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:52:16.895 7fab06539c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:52:16.895 7fab06539c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:52:16.895 7fab06539c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:16.895 7fab06539c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 08/21/2025 01:52:16.895 7fab06539c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:16.895 7fab06539c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:16.895 7fab06539c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:52:16.895 7fab06539c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:17.947 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/21/2025 01:52:18.656 7fab0f97a348 [contrail-device-manager] [WARNING]: table 'obj_uuid_table' fixed with properties {'gc_grace_seconds': 864000} 08/21/2025 01:52:18.663 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/21/2025 01:52:20.500 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/21/2025 01:52:20.894 7fab065340f0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:52:20.896 7fab065340f0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:52:20.896 7fab065340f0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:52:20.896 7fab065340f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:52:20.896 7fab065340f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 08/21/2025 01:52:20.896 7fab065340f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:52:20.896 7fab065340f0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:52:20.896 7fab065340f0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:52:20.896 7fab065340f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:20.897 7fab065340f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 08/21/2025 01:52:20.897 7fab065340f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:20.897 7fab065340f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:20.897 7fab065340f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:52:20.897 7fab065340f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:21.644 7fab0f97a348 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table', created 08/21/2025 01:52:21.649 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/21/2025 01:52:22.889 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/21/2025 01:52:24.603 7fab0f97a348 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table' fixed with properties {'gc_grace_seconds': 864000} 08/21/2025 01:52:24.609 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/21/2025 01:52:24.897 7fab064d1d68 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:52:24.898 7fab064d1d68 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:52:24.898 7fab064d1d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:52:24.898 7fab064d1d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:52:24.898 7fab064d1d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 08/21/2025 01:52:24.898 7fab064d1d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:52:24.898 7fab064d1d68 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:52:24.899 7fab064d1d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:52:24.899 7fab064d1d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:24.899 7fab064d1d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 08/21/2025 01:52:24.899 7fab064d1d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:24.899 7fab064d1d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:24.899 7fab064d1d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:52:24.899 7fab064d1d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:26.651 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/21/2025 01:52:26.656 7fab0f97a348 [contrail-device-manager] [WARNING]: table 'obj_shared_table', created 08/21/2025 01:52:26.662 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/21/2025 01:52:28.900 7fab064c96d8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:52:28.900 7fab064c96d8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:52:28.901 7fab064c96d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:52:28.901 7fab064c96d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:52:28.901 7fab064c96d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 08/21/2025 01:52:28.901 7fab064c96d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:52:28.901 7fab064c96d8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:52:28.901 7fab064c96d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:52:28.901 7fab064c96d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:28.901 7fab064c96d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 08/21/2025 01:52:28.901 7fab064c96d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:28.901 7fab064c96d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:28.902 7fab064c96d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:52:28.902 7fab064c96d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:29.703 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/21/2025 01:52:30.604 7fab0f97a348 [contrail-device-manager] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 08/21/2025 01:52:30.608 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/21/2025 01:52:30.609 7fab06da6e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:30.609 7fab06da6e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:52:30.609 7fab06da6e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:30.637 7fab0f97a348 [contrail-device-manager] [ERROR]: Error in db walk read 'type' 08/21/2025 01:52:30.638 7fab065298d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:30.638 7fab065298d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Zookeeper:Zookeeper[] connection down >>, ] >> 08/21/2025 01:52:30.638 7fab065298d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:30.648 7fab064d9240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:30.648 7fab064d9240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:52:30.648 7fab064d9240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:30.648 7fab0f97a348 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 08/21/2025 01:52:30.648 7fab0f97a348 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 08/21/2025 01:52:30.648 7fab0c8eb6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 08/21/2025 01:52:30.649 7fab0f97a348 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-3902-3. 08/21/2025 01:52:30.652 7fab0c8eb6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 08/21/2025 01:52:30.722 7fab0c8eb6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-3902-3.'] 08/21/2025 01:52:30.731 7fab0c8eb6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 08/21/2025 01:52:30.735 7fab0c8eb6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 08/21/2025 01:52:30.768 7fab0c8eb6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-3902-3.'] 08/21/2025 01:52:31.158 7fab0f97a348 [contrail-device-manager] [DEBUG]: Fetching all containers 08/21/2025 01:52:31.193 7fab0f97a348 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 08/21/2025 01:52:31.344 7fab0f97a348 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3902-3..config_queue 08/21/2025 01:52:31.345 7fab0f97a348 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3902-3..tftp_queue 08/21/2025 01:52:31.785 7fab0c8eb6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 08/21/2025 01:52:31.789 7fab0c8eb6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 08/21/2025 01:52:31.846 7fab0f97a348 [contrail-device-manager] [WARNING]: Process 140 prepared to run in Full mode... 08/21/2025 01:52:31.846 7fab0f97a348 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 08/21/2025 01:52:31.846 7fab0f97a348 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 08/21/2025 01:52:31.849 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 08/21/2025 01:52:31.849 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 08/21/2025 01:52:31.850 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 08/21/2025 01:52:31.850 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 08/21/2025 01:52:31.851 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 08/21/2025 01:52:31.851 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 08/21/2025 01:52:31.854 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 08/21/2025 01:52:31.854 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 08/21/2025 01:52:31.854 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 08/21/2025 01:52:31.854 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 08/21/2025 01:52:31.854 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 08/21/2025 01:52:31.855 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 08/21/2025 01:52:31.855 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 08/21/2025 01:52:31.855 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 08/21/2025 01:52:31.855 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 08/21/2025 01:52:31.855 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 08/21/2025 01:52:31.855 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 08/21/2025 01:52:31.855 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 08/21/2025 01:52:31.855 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 08/21/2025 01:52:31.857 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 08/21/2025 01:52:31.858 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 08/21/2025 01:52:31.858 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 08/21/2025 01:52:31.858 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 08/21/2025 01:52:31.859 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 08/21/2025 01:52:31.859 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 08/21/2025 01:52:31.859 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 08/21/2025 01:52:31.860 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 08/21/2025 01:52:31.860 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 08/21/2025 01:52:31.860 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 08/21/2025 01:52:31.860 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 08/21/2025 01:52:31.860 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 08/21/2025 01:52:31.860 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 08/21/2025 01:52:31.861 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 08/21/2025 01:52:31.862 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 08/21/2025 01:52:31.862 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 08/21/2025 01:52:31.862 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 08/21/2025 01:52:31.862 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 08/21/2025 01:52:31.862 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 08/21/2025 01:52:31.862 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 08/21/2025 01:52:31.862 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 08/21/2025 01:52:31.863 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 08/21/2025 01:52:31.864 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 08/21/2025 01:52:31.864 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 08/21/2025 01:52:31.864 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 08/21/2025 01:52:31.865 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 08/21/2025 01:52:31.866 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 08/21/2025 01:52:31.866 7fab0f97a348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 08/21/2025 01:52:31.870 7fab0f97a348 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 08/21/2025 01:52:31.884 7fab064054e0 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 08/21/2025 01:52:31.884 7fab064054e0 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 08/21/2025 01:52:31.927 7fab05c0b0f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:31.927 7fab05c0b0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 08/21/2025 01:52:31.927 7fab05c0b0f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:31.939 7fab05b49630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:31.940 7fab05b49630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 08/21/2025 01:52:31.940 7fab05b49630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:32.012 7fab0c8eb6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-3902-3.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3902-3..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3902-3..tftp_queue', 'device_manager_ztp.ztp_queue'] 08/21/2025 01:52:32.033 7fab0c8eb6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 08/21/2025 01:52:32.037 7fab0c8eb6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 08/21/2025 01:52:32.078 7fab0c8eb6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-3902-3.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3902-3..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3902-3..tftp_queue', 'device_manager_ztp.ztp_queue'] 08/21/2025 01:52:32.901 7fab05b512e8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:52:32.902 7fab05b512e8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:52:32.903 7fab05b512e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:52:32.903 7fab05b512e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:52:32.903 7fab05b512e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 08/21/2025 01:52:32.903 7fab05b512e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:52:32.903 7fab05b512e8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:52:32.904 7fab05b512e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:52:32.904 7fab05b512e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:32.904 7fab05b512e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 08/21/2025 01:52:32.904 7fab05b512e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:32.904 7fab05b512e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:32.904 7fab05b512e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:52:32.904 7fab05b512e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:36.906 7fab05a818d0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:52:36.909 7fab05a818d0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:52:36.910 7fab05a818d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:52:36.910 7fab05a818d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:52:36.910 7fab05a818d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 08/21/2025 01:52:36.910 7fab05a818d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:52:36.910 7fab05a818d0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:52:36.910 7fab05a818d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:52:36.911 7fab05a818d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:36.911 7fab05a818d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 08/21/2025 01:52:36.911 7fab05a818d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:36.911 7fab05a818d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:36.911 7fab05a818d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:52:36.911 7fab05a818d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:38.055 7fab05a71780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:38.056 7fab05a71780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:52:38.056 7fab05a71780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:40.909 7fab05a69f60 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:52:40.910 7fab05a69f60 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:52:40.910 7fab05a69f60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:52:40.910 7fab05a69f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:52:40.910 7fab05a69f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/21/2025 01:52:40.910 7fab05a69f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:52:40.911 7fab05a69f60 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:52:40.911 7fab05a69f60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:52:40.912 7fab05a69f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:40.912 7fab05a69f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:52:40.912 7fab05a69f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:40.912 7fab05a69f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:40.912 7fab05a69f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:52:40.912 7fab05a69f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:44.911 7fab05a06e10 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:52:44.912 7fab05a06e10 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:52:44.912 7fab05a06e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:52:44.912 7fab05a06e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:52:44.912 7fab05a06e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/21/2025 01:52:44.912 7fab05a06e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:52:44.912 7fab05a06e10 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:52:44.913 7fab05a06e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:52:44.913 7fab05a06e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:44.913 7fab05a06e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:52:44.913 7fab05a06e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:44.913 7fab05a06e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:44.913 7fab05a06e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:52:44.913 7fab05a06e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:48.914 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:52:48.915 7fab0e5954e0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:52:48.915 7fab0e5954e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:52:48.915 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:52:48.915 7fab0e5954e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/21/2025 01:52:48.915 7fab0e5954e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:52:48.915 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:52:48.916 7fab0e5954e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:52:48.916 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:48.916 7fab0e5954e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:52:48.916 7fab0e5954e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:48.916 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:48.916 7fab0e5954e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:52:48.916 7fab0e5954e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:52.916 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:52:52.917 7fab0e5954e0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:52:52.917 7fab0e5954e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:52:52.918 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:52:52.918 7fab0e5954e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/21/2025 01:52:52.918 7fab0e5954e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:52:52.918 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:52:52.918 7fab0e5954e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:52:52.918 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:52.918 7fab0e5954e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:52:52.918 7fab0e5954e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:52.918 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:52.919 7fab0e5954e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:52:52.919 7fab0e5954e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:56.918 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:52:56.919 7fab0e5954e0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:52:56.919 7fab0e5954e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:52:56.920 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:52:56.920 7fab0e5954e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/21/2025 01:52:56.920 7fab0e5954e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:52:56.920 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:52:56.920 7fab0e5954e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:52:56.920 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:56.920 7fab0e5954e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:52:56.920 7fab0e5954e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:56.920 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:52:56.920 7fab0e5954e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:52:56.920 7fab0e5954e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:00.921 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:53:00.922 7fab0e5954e0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:53:00.922 7fab0e5954e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:53:00.922 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:00.922 7fab0e5954e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/21/2025 01:53:00.922 7fab0e5954e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:00.922 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:53:00.923 7fab0e5954e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:53:00.923 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:00.923 7fab0e5954e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:53:00.923 7fab0e5954e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:00.923 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:00.923 7fab0e5954e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:53:00.923 7fab0e5954e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:04.923 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:53:04.924 7fab0e5954e0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:53:04.924 7fab0e5954e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:53:04.925 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:04.925 7fab0e5954e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/21/2025 01:53:04.925 7fab0e5954e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:04.925 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:53:04.925 7fab0e5954e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:53:04.925 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:04.925 7fab0e5954e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:53:04.925 7fab0e5954e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:04.925 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:04.926 7fab0e5954e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:53:04.926 7fab0e5954e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:08.926 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:53:08.927 7fab0e5954e0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:53:08.927 7fab0e5954e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:53:08.927 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:08.927 7fab0e5954e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/21/2025 01:53:08.927 7fab0e5954e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:08.927 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:53:08.928 7fab0e5954e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:53:08.928 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:08.928 7fab0e5954e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:53:08.928 7fab0e5954e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:08.928 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:08.928 7fab0e5954e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:53:08.928 7fab0e5954e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:12.927 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:53:12.929 7fab0e5954e0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:53:12.929 7fab0e5954e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:53:12.929 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:12.929 7fab0e5954e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/21/2025 01:53:12.929 7fab0e5954e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:12.929 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:53:12.929 7fab0e5954e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:53:12.929 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:12.929 7fab0e5954e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:53:12.930 7fab0e5954e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:12.930 7fab0e5954e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:12.930 7fab0e5954e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:53:12.930 7fab0e5954e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:16.929 7fab05a06c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:53:16.931 7fab05a06c18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:53:16.931 7fab05a06c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:53:16.931 7fab05a06c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:16.931 7fab05a06c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/21/2025 01:53:16.931 7fab05a06c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:16.931 7fab05a06c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:53:16.931 7fab05a06c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:53:16.931 7fab05a06c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:16.932 7fab05a06c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:53:16.932 7fab05a06c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:16.932 7fab05a06c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:16.932 7fab05a06c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:53:16.932 7fab05a06c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:20.932 7fab06fb3390 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:53:20.933 7fab06fb3390 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:53:20.933 7fab06fb3390 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:53:20.933 7fab06fb3390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:20.933 7fab06fb3390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/21/2025 01:53:20.933 7fab06fb3390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:20.933 7fab06fb3390 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:53:20.933 7fab06fb3390 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:53:20.934 7fab06fb3390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:20.934 7fab06fb3390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:53:20.934 7fab06fb3390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:20.934 7fab06fb3390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:20.934 7fab06fb3390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:53:20.934 7fab06fb3390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:24.934 7fab06fb3630 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:53:24.935 7fab06fb3630 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:53:24.935 7fab06fb3630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:53:24.935 7fab06fb3630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:24.935 7fab06fb3630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/21/2025 01:53:24.936 7fab06fb3630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:24.936 7fab06fb3630 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:53:24.936 7fab06fb3630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:53:24.936 7fab06fb3630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:24.937 7fab06fb3630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:53:24.937 7fab06fb3630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:24.937 7fab06fb3630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:24.937 7fab06fb3630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:53:24.937 7fab06fb3630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:28.936 7fab0cab2240 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:53:28.937 7fab0cab2240 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:53:28.937 7fab0cab2240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:53:28.937 7fab0cab2240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:28.937 7fab0cab2240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/21/2025 01:53:28.937 7fab0cab2240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:28.937 7fab0cab2240 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:53:28.938 7fab0cab2240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:53:28.938 7fab0cab2240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:28.938 7fab0cab2240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:53:28.938 7fab0cab2240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:28.938 7fab0cab2240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:28.938 7fab0cab2240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:53:28.938 7fab0cab2240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:32.938 7fab05ab12e8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:53:32.939 7fab05ab12e8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:53:32.939 7fab05ab12e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:53:32.939 7fab05ab12e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:32.940 7fab05ab12e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/21/2025 01:53:32.940 7fab05ab12e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:32.940 7fab05ab12e8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:53:32.940 7fab05ab12e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:53:32.940 7fab05ab12e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:32.940 7fab05ab12e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:53:32.940 7fab05ab12e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:32.940 7fab05ab12e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:32.940 7fab05ab12e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:53:32.940 7fab05ab12e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:36.940 7fab0cab2438 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:53:36.941 7fab0cab2438 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:53:36.941 7fab0cab2438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:53:36.942 7fab0cab2438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:36.942 7fab0cab2438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/21/2025 01:53:36.942 7fab0cab2438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:36.942 7fab0cab2438 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:53:36.942 7fab0cab2438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:53:36.942 7fab0cab2438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:36.942 7fab0cab2438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:53:36.942 7fab0cab2438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:36.942 7fab0cab2438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:36.943 7fab0cab2438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:53:36.943 7fab0cab2438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:40.942 7fab0ca93438 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:53:40.944 7fab0ca93438 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:53:40.944 7fab0ca93438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:53:40.945 7fab0ca93438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:40.946 7fab0ca93438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/21/2025 01:53:40.946 7fab0ca93438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:40.946 7fab0ca93438 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:53:40.946 7fab0ca93438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:53:40.946 7fab0ca93438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:40.946 7fab0ca93438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:53:40.947 7fab0ca93438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:40.947 7fab0ca93438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:40.947 7fab0ca93438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:53:40.947 7fab0ca93438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:44.945 7fab05b2aac8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:53:44.946 7fab05b2aac8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:53:44.946 7fab05b2aac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:53:44.946 7fab05b2aac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:44.946 7fab05b2aac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/21/2025 01:53:44.946 7fab05b2aac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:44.946 7fab05b2aac8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:53:44.946 7fab05b2aac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:53:44.947 7fab05b2aac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:44.947 7fab05b2aac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:53:44.947 7fab05b2aac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:44.947 7fab05b2aac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:44.947 7fab05b2aac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:53:44.947 7fab05b2aac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:46.188 7fab0f97a348 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 08/21/2025 01:53:46.188 7fab0f97a348 [contrail-device-manager] [WARNING]: initializing cassandra cluster 08/21/2025 01:53:46.188 7fab0f97a348 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 08/21/2025 01:53:46.188 7fab0dcf2d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:46.189 7fab0dcf2d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 08/21/2025 01:53:46.189 7fab0dcf2d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:46.190 7fab0f97a348 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 08/21/2025 01:53:46.232 7fab0f97a348 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 08/21/2025 01:53:46.234 7fab05b84978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:46.234 7fab05b84978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Initializing description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 08/21/2025 01:53:46.234 7fab05b84978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:46.258 7fab0f97a348 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 08/21/2025 01:53:46.258 7fab05b84780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:46.258 7fab05b84780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 08/21/2025 01:53:46.258 7fab05b84780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:46.290 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/21/2025 01:53:46.713 7fab0f97a348 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 08/21/2025 01:53:46.717 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/21/2025 01:53:46.722 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/21/2025 01:53:47.738 7fab0f97a348 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 08/21/2025 01:53:47.743 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/21/2025 01:53:47.743 7fab0f97a348 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 08/21/2025 01:53:47.810 7fab0f97a348 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 08/21/2025 01:53:47.822 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/21/2025 01:53:48.925 7fab0f97a348 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 08/21/2025 01:53:48.931 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/21/2025 01:53:48.938 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/21/2025 01:53:48.947 7fab05afb588 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:53:48.949 7fab05afb588 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:53:48.949 7fab05afb588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:53:48.949 7fab05afb588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:48.950 7fab05afb588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 08/21/2025 01:53:48.950 7fab05afb588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:48.950 7fab05afb588 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:53:48.951 7fab05afb588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:53:48.956 7fab05afb588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:48.957 7fab05afb588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 08/21/2025 01:53:48.957 7fab05afb588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:48.957 7fab05afb588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:48.957 7fab05afb588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:53:48.957 7fab05afb588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:50.460 7fab0f97a348 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 08/21/2025 01:53:50.469 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/21/2025 01:53:50.475 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/21/2025 01:53:51.636 7fab0f97a348 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 08/21/2025 01:53:51.641 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/21/2025 01:53:51.652 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/21/2025 01:53:52.051 7fab0f97a348 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 08/21/2025 01:53:52.055 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/21/2025 01:53:52.068 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/21/2025 01:53:52.949 7fab05be4a20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:53:52.950 7fab05be4a20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:53:52.950 7fab05be4a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:53:52.951 7fab05be4a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:52.951 7fab05be4a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 08/21/2025 01:53:52.951 7fab05be4a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:52.951 7fab05be4a20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:53:52.951 7fab05be4a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:53:52.951 7fab05be4a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:52.951 7fab05be4a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 08/21/2025 01:53:52.951 7fab05be4a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:52.952 7fab05be4a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:52.952 7fab05be4a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:53:52.952 7fab05be4a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:52.992 7fab0f97a348 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 08/21/2025 01:53:52.999 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/21/2025 01:53:53.004 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/21/2025 01:53:53.735 7fab0f97a348 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 08/21/2025 01:53:53.738 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/21/2025 01:53:53.744 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/21/2025 01:53:54.756 7fab0f97a348 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 08/21/2025 01:53:54.760 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/21/2025 01:53:54.769 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/21/2025 01:53:55.576 7fab0f97a348 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 08/21/2025 01:53:55.580 7fab0f97a348 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/21/2025 01:53:55.580 7fab0640d438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:55.580 7fab0640d438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:53:55.580 7fab0640d438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:55.625 7fab0f97a348 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.27:5673//', 'pyamqp://guest:guest@10.0.0.249:5673//'] 08/21/2025 01:53:55.625 7fab0f97a348 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 08/21/2025 01:53:55.625 7fab059cf240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:55.626 7fab059cf240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Initializing description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 08/21/2025 01:53:55.626 7fab059cf240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:55.626 7fab059cf240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:55.626 7fab059cf240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 08/21/2025 01:53:55.626 7fab059cf240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:55.629 7fab0f97a348 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 08/21/2025 01:53:55.630 7fab059cfcc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:55.630 7fab059cfcc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:53:55.630 7fab059cfcc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:55.675 7fab0f97a348 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 08/21/2025 01:53:55.675 7fab059d3f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:55.676 7fab059d3f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 08/21/2025 01:53:55.676 7fab059d3f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:55.678 7fab0f97a348 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 08/21/2025 01:53:55.679 7fab05bd88d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:55.679 7fab05bd88d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:53:55.679 7fab05bd88d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:55.680 7fab059e0828 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 08/21/2025 01:53:55.680 7fab05a0f828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:55.680 7fab05a0f828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 08/21/2025 01:53:55.681 7fab05a0f828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:55.683 7fab059e0828 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 08/21/2025 01:53:55.683 7fab05b71b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:55.684 7fab05b71b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:53:55.684 7fab05b71b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:56.952 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:53:56.953 7fab059cf828 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:53:56.953 7fab059cf828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:53:56.953 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:56.953 7fab059cf828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/21/2025 01:53:56.953 7fab059cf828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:53:56.953 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:53:56.954 7fab059cf828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:53:56.954 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:56.954 7fab059cf828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:53:56.954 7fab059cf828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:56.954 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:53:56.954 7fab059cf828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:53:56.954 7fab059cf828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:54:00.955 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:54:00.956 7fab059cf828 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:54:00.956 7fab059cf828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:54:00.956 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:54:00.957 7fab059cf828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/21/2025 01:54:00.957 7fab059cf828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:54:00.957 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:54:00.957 7fab059cf828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:54:00.957 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:54:00.957 7fab059cf828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:54:00.958 7fab059cf828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:54:00.958 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:54:00.958 7fab059cf828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:54:00.958 7fab059cf828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:54:04.957 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:54:04.958 7fab059cf828 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:54:04.959 7fab059cf828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:54:04.959 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:54:04.959 7fab059cf828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/21/2025 01:54:04.959 7fab059cf828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:54:04.959 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:54:04.959 7fab059cf828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:54:04.959 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:54:04.959 7fab059cf828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:54:04.959 7fab059cf828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:54:04.959 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:54:04.959 7fab059cf828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:54:04.960 7fab059cf828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:54:08.959 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:54:08.960 7fab059cf828 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:54:08.961 7fab059cf828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:54:08.961 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:54:08.961 7fab059cf828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/21/2025 01:54:08.961 7fab059cf828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:54:08.961 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:54:08.961 7fab059cf828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:54:08.961 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:54:08.961 7fab059cf828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:54:08.961 7fab059cf828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:54:08.961 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:54:08.962 7fab059cf828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:54:08.962 7fab059cf828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:54:12.962 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:54:12.963 7fab059cf828 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/21/2025 01:54:12.963 7fab059cf828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:54:12.963 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:54:12.963 7fab059cf828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/21/2025 01:54:12.963 7fab059cf828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:54:12.963 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/21/2025 01:54:12.963 7fab059cf828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/21/2025 01:54:12.963 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:54:12.963 7fab059cf828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/21/2025 01:54:12.964 7fab059cf828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:54:12.964 7fab059cf828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:54:12.964 7fab059cf828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1755741132893273 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.27:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/21/2025 01:54:12.964 7fab059cf828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/21/2025 01:54:13.955 7fab05bd88d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3902-3.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3902-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3902-3.'], 'id_perms': {'created': '2025-08-21T01:54:13.947803', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-21T01:54:13.947803', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10231971802870624235, 'uuid_mslong': 5059113504535561160}}, 'parent_type': 'global-system-config', 'parent_uuid': '93f37ef3-c46b-43eb-b40a-111476951d25', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '463594ed-9439-43c8-8dff-4424b5eaafeb'}, 'oper': 'CREATE', 'request-id': 'req-fa0fa3d4-068f-4d2e-b61b-e8d8f4f9e8ab', 'type': 'analytics_node', 'uuid': '463594ed-9439-43c8-8dff-4424b5eaafeb'} 08/21/2025 01:54:14.349 7fab05bd88d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3902-2.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.27', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3902-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3902-2.'], 'id_perms': {'created': '2025-08-21T01:54:14.337957', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-21T01:54:14.337957', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12667952515219607346, 'uuid_mslong': 1263940799794660629}}, 'parent_type': 'global-system-config', 'parent_uuid': '93f37ef3-c46b-43eb-b40a-111476951d25', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '118a6b75-f4e2-4515-afcd-9bb1e52e4f32'}, 'oper': 'CREATE', 'request-id': 'req-90a55ea2-a6f8-4d7a-8ba8-2427ef9bb379', 'type': 'analytics_node', 'uuid': '118a6b75-f4e2-4515-afcd-9bb1e52e4f32'} 08/21/2025 01:54:14.426 7fab05bd88d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3902-1.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3902-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3902-1.'], 'id_perms': {'created': '2025-08-21T01:54:14.414856', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-21T01:54:14.414856', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10218053667666227595, 'uuid_mslong': 9967423697140207142}}, 'parent_type': 'global-system-config', 'parent_uuid': '93f37ef3-c46b-43eb-b40a-111476951d25', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '8a53670a-f9da-4e26-8dcd-d1ac551f718b'}, 'oper': 'CREATE', 'request-id': 'req-ba191737-eb97-45b7-a229-0adaae7e16be', 'type': 'analytics_node', 'uuid': '8a53670a-f9da-4e26-8dcd-d1ac551f718b'} 08/21/2025 01:54:16.964 7fab059d6a20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/21/2025 01:54:16.965 7fab059d6a20 [contrail-device-manager] [INFO]: Session Event: TCP Connected 08/21/2025 01:54:16.965 7fab059d6a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/21/2025 01:54:16.965 7fab059d6a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:54:16.965 7fab059d6a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3902-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.27:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.27:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.27:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.27,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/21/2025 01:54:16.965 7fab059d6a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/21/2025 01:54:16.965 7fab059d6a20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 08/21/2025 01:54:16.965 7fab059d6a20 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 08/21/2025 01:54:16.966 7fab059d6a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 08/21/2025 01:54:16.966 7fab059d6a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 08/21/2025 01:54:16.966 7fab059d6a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 08/21/2025 01:54:16.966 7fab058f2978 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 08/21/2025 01:54:16.967 7fab058f2978 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 08/21/2025 01:54:24.703 7fab058f24e0 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001016 08/21/2025 01:54:24.704 7fab058f24e0 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 08/21/2025 01:54:24.704 7fab05b32f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 08/21/2025 01:54:24.705 7fab05b32f60 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 08/21/2025 01:54:24.705 7fab05b32f60 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 08/21/2025 01:54:24.705 7fab05b32f60 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 08/21/2025 01:54:24.705 7fab05b32f60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 08/21/2025 01:54:24.706 7fab059d6a20 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 08/21/2025 01:54:24.707 7fab059d6a20 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 08/21/2025 01:54:28.396 7fab05bd88d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3902-2.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.27', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3902-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3902-2.'], 'id_perms': {'created': '2025-08-21T01:54:28.374521', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-21T01:54:28.374521', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12859218242556057623, 'uuid_mslong': 4103529270180397325}}, 'parent_type': 'global-system-config', 'parent_uuid': '93f37ef3-c46b-43eb-b40a-111476951d25', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '38f2aa21-3a85-410d-b275-1edc4cc6d817'}, 'oper': 'CREATE', 'request-id': 'req-029b1b0c-3095-4d74-9f9c-637c9da70283', 'type': 'analytics_alarm_node', 'uuid': '38f2aa21-3a85-410d-b275-1edc4cc6d817'} 08/21/2025 01:54:28.458 7fab05bd88d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3902-1.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3902-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3902-1.'], 'id_perms': {'created': '2025-08-21T01:54:28.444986', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-21T01:54:28.444986', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12549082701693810739, 'uuid_mslong': 5199152212283835070}}, 'parent_type': 'global-system-config', 'parent_uuid': '93f37ef3-c46b-43eb-b40a-111476951d25', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '48271964-0a5d-42be-ae27-4c3ab3757833'}, 'oper': 'CREATE', 'request-id': 'req-4678cf21-bcaf-4af8-824d-ea1d2ddf8f08', 'type': 'analytics_alarm_node', 'uuid': '48271964-0a5d-42be-ae27-4c3ab3757833'} 08/21/2025 01:54:28.938 7fab05bd88d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3902-3.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3902-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3902-3.'], 'id_perms': {'created': '2025-08-21T01:54:28.920537', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-21T01:54:28.920537', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10631508356567769995, 'uuid_mslong': 350248588980666895}}, 'parent_type': 'global-system-config', 'parent_uuid': '93f37ef3-c46b-43eb-b40a-111476951d25', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '04dc553c-6946-420f-938a-b485abd1a38b'}, 'oper': 'CREATE', 'request-id': 'req-b84c9b09-33e5-4d6b-8ee8-2ea91f9ad54e', 'type': 'analytics_alarm_node', 'uuid': '04dc553c-6946-420f-938a-b485abd1a38b'} 08/21/2025 01:54:34.682 7fab058f24e0 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001296 08/21/2025 01:54:34.683 7fab058f24e0 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 08/21/2025 01:54:34.683 7fab07c78828 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 08/21/2025 01:54:34.683 7fab07c78828 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 2 08/21/2025 01:54:34.683 7fab07c78828 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 08/21/2025 01:54:34.684 7fab07c78828 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 08/21/2025 01:54:37.470 7fab05bd88d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3902-1.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3902-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3902-1.'], 'id_perms': {'created': '2025-08-21T01:54:37.451824', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-21T01:54:37.451824', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13291068846429034944, 'uuid_mslong': 4491876931446787898}}, 'parent_type': 'global-system-config', 'parent_uuid': '93f37ef3-c46b-43eb-b40a-111476951d25', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '3e565a45-29a8-4f3a-b873-5cb25ca9d9c0'}, 'oper': 'CREATE', 'request-id': 'req-70f192c7-0a46-4707-98e4-22a21024d08c', 'type': 'analytics_snmp_node', 'uuid': '3e565a45-29a8-4f3a-b873-5cb25ca9d9c0'} 08/21/2025 01:54:37.837 7fab05bd88d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3902-2.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.27', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3902-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3902-2.'], 'id_perms': {'created': '2025-08-21T01:54:37.816382', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-21T01:54:37.816382', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12484067803875087290, 'uuid_mslong': 9772055124261160789}}, 'parent_type': 'global-system-config', 'parent_uuid': '93f37ef3-c46b-43eb-b40a-111476951d25', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '879d505c-671f-4755-ad40-518631dd93ba'}, 'oper': 'CREATE', 'request-id': 'req-7d156b9f-07bc-467b-9a1d-69b8c0983d91', 'type': 'analytics_snmp_node', 'uuid': '879d505c-671f-4755-ad40-518631dd93ba'} 08/21/2025 01:54:37.938 7fab05bd88d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3902-3.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3902-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3902-3.'], 'id_perms': {'created': '2025-08-21T01:54:37.916393', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-21T01:54:37.916393', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12741621428945289938, 'uuid_mslong': 11378443718813238849}}, 'parent_type': 'global-system-config', 'parent_uuid': '93f37ef3-c46b-43eb-b40a-111476951d25', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '9de85a46-d4a3-4641-b0d3-552e7eca02d2'}, 'oper': 'CREATE', 'request-id': 'req-541d4364-c9c1-4c7e-b0c3-96e2eac21022', 'type': 'analytics_snmp_node', 'uuid': '9de85a46-d4a3-4641-b0d3-552e7eca02d2'} 08/21/2025 01:56:16.339 7fab05bd88d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-b2a27617-47c3-4887-9d41-0af973261d5e', 'type': 'global_system_config', 'uuid': '93f37ef3-c46b-43eb-b40a-111476951d25'} 08/21/2025 01:56:16.476 7fab05bd88d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3902-1'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-3902-1', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3902-1'], 'id_perms': {'created': '2025-08-21T01:56:16.468705', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-21T01:56:16.468705', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12514909158584753316, 'uuid_mslong': 5147515511785016384}}, 'parent_type': 'global-system-config', 'parent_uuid': '93f37ef3-c46b-43eb-b40a-111476951d25', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '476fa615-d3c2-4c40-adad-e392a9ba1ca4', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.25'}, 'oper': 'CREATE', 'request-id': 'req-17d4afb1-b2bc-4377-a7a7-d8dab722b3af', 'type': 'virtual_router', 'uuid': '476fa615-d3c2-4c40-adad-e392a9ba1ca4'} 08/21/2025 01:56:16.826 7fab05bd88d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3902-1', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3902-1', 'vhost0'], 'id_perms': {'created': '2025-08-21T01:56:16.808012', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-21T01:56:16.808012', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10653094768304105279, 'uuid_mslong': 13600300560282240752}}, 'parent_type': 'virtual-router', 'parent_uuid': '476fa615-d3c2-4c40-adad-e392a9ba1ca4', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'bcbdf94d-562b-46f0-93d7-653fcf56a73f', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:bc:bd:f9:4d:56']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': 'c67c8e2d-5c0d-4211-bac1-63f229c128ce'}]}, 'oper': 'CREATE', 'request-id': 'req-502e822f-d869-4461-9cb1-62f3581a79f4', 'type': 'virtual_machine_interface', 'uuid': 'bcbdf94d-562b-46f0-93d7-653fcf56a73f'} 08/21/2025 01:56:16.856 7fab05bd88d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3902-1', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-502e822f-d869-4461-9cb1-62f3581a79f4', 'type': 'virtual_machine_interface', 'uuid': 'bcbdf94d-562b-46f0-93d7-653fcf56a73f'} 08/21/2025 01:56:16.987 7fab05bd88d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3902-2'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-3902-2', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3902-2'], 'id_perms': {'created': '2025-08-21T01:56:16.965488', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-21T01:56:16.965488', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12025594528410509957, 'uuid_mslong': 10933144463807496382}}, 'parent_type': 'global-system-config', 'parent_uuid': '93f37ef3-c46b-43eb-b40a-111476951d25', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '97ba54f6-ab46-40be-a6e3-7e825eca2285', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.193'}, 'oper': 'CREATE', 'request-id': 'req-02fcdc7d-42cc-4879-b15d-d4054f0fadca', 'type': 'virtual_router', 'uuid': '97ba54f6-ab46-40be-a6e3-7e825eca2285'} 08/21/2025 01:56:17.310 7fab05bd88d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3902-2', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3902-2', 'vhost0'], 'id_perms': {'created': '2025-08-21T01:56:17.294961', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-21T01:56:17.294961', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9958130433641007114, 'uuid_mslong': 14132903134082190804}}, 'parent_type': 'virtual-router', 'parent_uuid': '97ba54f6-ab46-40be-a6e3-7e825eca2285', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c4222885-67f9-4dd4-8a32-62de742dc80a', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:c4:22:28:85:67']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': 'c67c8e2d-5c0d-4211-bac1-63f229c128ce'}]}, 'oper': 'CREATE', 'request-id': 'req-9d140286-4fc1-49ee-a44c-9e0c046f74ee', 'type': 'virtual_machine_interface', 'uuid': 'c4222885-67f9-4dd4-8a32-62de742dc80a'} 08/21/2025 01:56:17.399 7fab05bd88d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3902-2', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-9d140286-4fc1-49ee-a44c-9e0c046f74ee', 'type': 'virtual_machine_interface', 'uuid': 'c4222885-67f9-4dd4-8a32-62de742dc80a'} 08/21/2025 01:56:17.404 7fab05bd88d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-31adbad2-9e2a-4fa0-9d06-2988467c4956', 'type': 'access_control_list', 'uuid': '5a0d9657-f37a-47bb-9691-1220d34fe725'} 08/21/2025 01:56:39.633 7fab059d5780 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 08/21/2025 01:57:09.697 7fab059d5780 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 08/21/2025 01:58:23.868 7fab059ce8d0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map